I have only been smithing for a few months now but already I've had good and bad experiences with wearing gloves. Last week I managed to get a hot coal down my hammer hand glove. It must have been in there 2 seconds before I noticed. Now I have a very tender spot on my wrist!
Other than that I think that wearing the gloves all the time gives you too much confidence in holding hot metal so the chances of picking up a hot piece with bare hands is always a risk if you are used to working with gloves.