I would also make sure you're using high quality sandpaper. Cheapo sandpaper can have inconsistent grit which will leave rogue swirl marks.
For what it's worth, I start at 100, then 120, then 150, then 180, then 220. It's a lot of paper changes, but I don't need to spend too much time on each grit.
Also, don't push too hard on the sander. As they say, "let the sander do the work" the more the pad can spin, the more efficiently it works and the less likely that an irregular grit will "dig in".
Lastly, my favorite tip. If your sander has a vacuum port, hook it up to a vacuum. Having the vacuum suck up the dust not only makes for less dust up your nose, but getting that dust out of the way saves the sander from re-sanding the dust it just made and it works a zillion times better.