From experience doing 3 CD systems/remodels and some research I've concluded 6" is the minimum for any blower over 1.5 HP/1100CFM. 4" will be too restrictive I know this was the case with my original system. Going to 6" ducts was a serious upgrade in performance even with the small 1.5HP blower I have. In fact, I was preparing to go to a 3HP blower because I am using a cyclone but gave it a try and was very surprised. All I can say is it does the job and I'm still using it.
Your 3HP blower could easily handle 7 or even 8" ducts. Keep ducts big as far as you can before reducing, long sweeping 90's or 2x45's.
Venting outside is another option sometimes available that will enhance performance. Considerations are your location & heat loss. If this is not possible I recommend getting rid of the bags they are extremely inefficient and will prove to be a hassle emptying and cleaning. I would recommend either a cyclone or canisters type filters.
Forums can be a good or bad source of info, so I would suggest you do a little research and learn about CFM, static pressure, air velocity, filter bags, etc and how these are affected by duct/blower sizes.