When forming the head it leans to one side. I might mention that the 1/4" rod is a little loose in the tool that holds the blank.

Roy, you have described your problem exactly.  If the male/female meeting of the die and the shank is loose the shank will bend to one side or another when the head is upset.  You need a new and smaller hole or a larger diameter stock to eliminate wiggle.  The stock will upset a bit and grip the side of the hole which may require a tap with a punch to remove it.  Some rivet making dies are split and hinged (like a lead bullet mold) to make removal easier.

"By hammer and hand all arts do stand."

Good Morning Roy,

Make a Mandrel out of a good material, larger stock. Make it so it fits in your Post Vice. Drill the depth (or more) of the length of rivet you want. Don't drill the size of your stock, drill a size or 2 larger. Drill a smaller hole just off-center from the bottom (for a punch to remove a stuck one when it cools). Don't heat the whole rivet blank, just the top. Drop your blank in the hole and set the head, in the first 2 or 3 blows, don't baby it. If you have a Monkey Tool be quick with it, not gentle. Put a slight chamfer in the top of the drilled hole, or a slight radius (don't let it cut the material). This would be good for a Fly Press or a Hydraulic Press (if it is quick enough).

Neil

 

Using a set hammer to ensure your blows are square with the piece is an excellent way to keep the material upsetting square. 

Alternatively, using a top dome die can be a corrective measure when it starts to tilt a little. Both of those can be used with great success, depending on how you want to head it. With practice, you can even learn to redirect your upsets by angling your hammer blows to either match or counteract the "s" tendency in the metal being upset. 

As a last word, there is a way to figure out how much material you need for what size head. In general I think it is 1 + 1/2 x the diameter of the bar. If you need bigger rivet heads, then butcher and draw out the rivet shank from larger stock - basically a tenon, then finish the head in the riveting tool.