"He’s [threatened to withdraw] 100 times. It would totally [screw] us as a country," according to one source