Any reason not to get a bigger tent.

I'm looking to get into camping with my wife and dog. We will almost always be car camping. As I'm looking at tents online, the price difference between different sizes of the same model seems pretty small across the board. I'm thinking a 4-6 person tent compared to something like 10-12 person. Is there any reason I wouldn't just buy a larger tent to have more space?

Edit* Also, we have a queen blow-up mattress and live in the desert. So besides the occasional Flagstaff/Grand Canyon trips staying cool will be more important than staying warm if that's relevant.

Edit* Thank you everybody for your responses we ended up ordering a Gazelle T4 Overland and a Gazelle G5 Gazebo. It made sense to have the single smaller tent just in case space was limited. But having a hangout area option when space allows.