Skipping grits during the sanding sequence will definitely cause the swirls to be noticeable. Corning as shown above will also cause the problem, although usually to a worse degree. I keep a small wire brush and a sand paper eraser block handy to clear the paper if it starts to clog. Cheap sandpaper that doesn't have uniform grit size can also cause the swirls. With a 1/4 sheet sander, the pad or platen, must be flat and you have to keep it flat on the work piece. I recently purchased a new Makita 1/4 sheet sander after my 43 year old B&D died and the platen was not flat across the entire surface. This caused swirls in all grits. Warranty replacement at the local tool store corrected the issue.
I've gone out to find myself. If I return before I get back, have me wait for me.
"Sawdust is Man Glitter"