In my opinion the 4-jaw chuck will be easiest as you can turn your jam chuck and leave it in the chuck you complete your peppermill.
If you want to use the jam chuck on multiple pepper mills, mark it so you can put it back in the same jaws as it was created in. Depending on your 4-jaw chuck it may run several thousandths more true.
You can take the time to make one that works with a drive center, and if you do multiple pepper mills this could be beneficial, and may save you a little bit of time. However wooden jamb chucks in my experience can change over time, so for the most accuracy I would to the jamb chuck within the 4-jaw chuck.