Small motor on 3S: 3400kV x 11.1V = 37,740RPM.
Big motor on 4S: 1800kV x 14.8V = 26,640RPM.
Big motor on 6S: 1800kV x 22.2V = 39,960RPM.
That's why it felt quicker with the smaller motor. You'd need to adjust your gearing to get the most out of the bigger motor. The larger rotor and magnets should result in more torque and therefore a faster spin-up, despite being heavier.
One can lose things and one can loosen things. "Loose" is very rarely a verb; it is an adjective, and it describes the grasp some have on this concept.