It depends on the amount of rust, what you are forging, and whether the rust is uneven.
My humble observation is that the more refined the finished surface you are aiming for, the more concerned you will likely be about rust on your iron, scale on the anvil, as well as blemishes on the anvil or hammer surfaces. If you are hammering a tent stake such concerns are minimal. If you are aiming to have a knife with a mirror or etched finish, then such concerns increase dramatically.