when you put the dowel in... is there room to adjust it at all? What if you try 2 different dowels in the hole, will they come out to be in the same place?
wood is not like metal... it will give with the right leverage, so even inserting the dowel, even if it feels snug after its inserted, when you hammer it in, if you hammer it a degree off, it will find a home in that position.
My suggestion would be to create a guide, some over engineered rigid box/tower that you clamp to the upright, loose fitting hole right down the middle. Put the dowel in, hammer it home, the box should make sure it seats straight, then its just hope that the dowel doesnt warp after the fact.