Measure the forster bit and twist bit with a caliper. It's possible it's oversized. Or... the dowel is undersized. On top of that wobble in the drill press spindle might make the hole larger, but sounds unlikely because of the way the forstner bit is designed.
A forum community dedicated to professional woodworkers and enthusiasts. Come join the discussion about shop safety, wood, carpentry, lumber, finishing, tools, machinery, woodworking related topics, styles, scales, reviews, accessories, classifieds, and more!