Ouch.
I couldn't drive wooden stakes into my ground here... too many rocks. DW & I made a whole rock garden, from all the stones we dug up the first few years. All I use is metal poles, which won't shatter if I hit something. But even with them, I use either a pole driver (for T-posts) or a 12" length of pipe - with screw on pipe cap - to pound smaller poles. The pipe helps keep the pole straight, and provides a good flat surface to pound on. No chasing a dodging pole with a hammer (which often leads to the ouch).
But the main reason I would recommend placing a capped pipe over a pole to drive it is safety. The pipe will catch any flying splinters; and if properly sized to just fit over the poles, may even reduce damage to the stakes. You still have to watch for debris when pulling the pipe off, and be careful not to bonk yourself if lifted above head height. The piece of pipe doesn't weigh much, and is less dangerous than the hammer which drives it. For wooden stakes, this is the best & safest option.
But if using a much heavier pole driver, you REALLY need to be careful when lifting it off of a tall pole. For my 6.5' trellis poles, I've learned to flip it up & over... and be prepared to dodge quickly if I lose my grip.
If pounding directly on ANY pole with a hammer, be sure to wear eye protection - especially pounding metal-on-metal. If you ever need to get an MRI, that forgotten metal shaving/splinter could give you a very rude awakening. Talk about "ouch".