possibly it's ...
If it ain't the bits and it ain't the collet, it would have to be the bottom bearing in the router itself. Assuming you have diagnosed the collet correctly and it tightens down properly, then you can grab the bit and see if you can determine any play in it, unplugged of course! There should be no play in or out or sideways.
Let us know what you find.
The answer to your question will only be as detailed and specific as the question is detailed and specific. Good questions also include a sketch or a photo that illustrates your issue. (:< D)