Something similar happened on 460 fords used on jet boats. They used a standard 460 engine and added a Cobra Jet style intake manifold. The CJ intake ports being much larger than the small port cylinder heads. No one that took the engine apart could figure why there were not issues. However, the motor ran fine, in fact it made more power with the CJ setup than the standard four bbl manifold.
I'd do it. Probably be fine. The only problem I can see is that every time I've switched from 2-4's to a single 4 bbl on a good intake, I've made nearly the same power and it always ran better.
Lorenr