Moving an off center hole like that back to center and keeping the lobe properly timed is more involved that it would first appear. Everything tries to take the path of least resistance, which is not where you want things to end up. You will likely end up with a nice round hole,, in the wrong place, Good luck if you try it by hand or on a drill press because even on a mill it is a challenge. Your best bet would be to start over with a new shaft. You need to make friends with a Bridgeport owner and have him drill you a new shaft and then buy a reamer from McMaster Carr plus the pins,,,, ya, it was an expensive ooops fixed correctly. I have the tooling and pins but all is tucked away as I move my shop operations or I would help you out,