sounds like you just need to take a little more time lining up the hole when starting to punch the slot...making sure it is correct to start with, if its off you can correct it . also if you look at a hammer eye it has and hour glass profile so when you put the wedges in the wood will move to the swell to keep the head from flying off.
as for the oversize hole its easier to use a narrow slot punch than a large round punch . you could always drill a large hole . I've also heard of a couple of small holes drilled at each end of the slot to help keep the slot lined up