If you use screw machine drill bits they have less flex and will not be as likely to wander. They are shorter compared to jobber bits. As someone said if the bit has been sharpened wrong it will walk sideways. "Starter" drills are available from places that sell metal working supplies. They are short and stiff but then you need to change to a regular bit to finish the cut. Regular twist drills also come with either 118 degree points or 135 degree. 118 most common. There are wood working bits but most are made for drilling into the flat of a board not the end grain as in a dowel. If you are looking for close fits you can buy a large set of twist drills that include numbered and letter bits in addition to fractional. Very handy to have. Gripping a twist drill in the "twist" area is not a good idea. It can either damage the chuck or the bit. You might want to check the runout on your chuck/drill press.