Just a guess, but that's the dust falling out of the air (although it seems like your first coat would have it as well). If true, it's almost impossible to eliminate completely, but it can be reduced substantially. Let the shop sit for several hours before applying the finish, a lot will drop out and you can spray. You could also get an air filtering system of some kind, and run it before finishing. That could be as simple as a box fan with a furnace filter taped to the intake side, an overhead shop made ambient air cleaner, or a purchased AAC. You (maybe) could also move the finishing to a completely separate area if you set up to do so. Regardless, you will still have some dust nibs, let the varnish cure (several days), then rub it with a wadded up piece of kraft paper (brown grocery bags or such). That smooths out the smaller bumps.
The problem with oil based finishes is that they stay wet a long time. How are you applying it? Spray can or gun/compressor?
Like Fred said, it's nearly impossible to eradicate all the airborne debris, especially if you finish in your work area. What you might try is using a waterbase polyurethane. It dries very fast to touch, stays clear, and cleans up with water.
A clean environment is good when working with oil based finishes but most of the time when dust and debris gets into the finish it comes off the painter. It's especially important to make sure you have yourself and clothing free of dust and dirt before finishing.
A forum community dedicated to professional woodworkers and enthusiasts. Come join the discussion about shop safety, wood, carpentry, lumber, finishing, tools, machinery, woodworking related topics, styles, scales, reviews, accessories, classifieds, and more!