I'm guessing by "dirt" you are referring to dust nibs? If so, after the varnish is mostly cured, try using some wadded up kraft paper (brown paper, like the old paper grocery bags). Polyurethane varnishes should be scuff sanded between coats for adhesion, doing that would remove the nibs...so I'm suggesting you would be doing that with each coat. with other formulas (non-poly varnishes) the scuff sanding isn't needed. So I usually apply 3 coats, let cure a little, then remove the nibs. Sometimes I'll then put on a last coat of wiping varnish (highly thinned, with a cloth). BTW, if the kraft paper doesn't get all the nibs out, then you'll have to go to a fibril wool, or wet sand with a very fine grit paper. I shun steel wool, but you could use that as well.