climate plays a huge role. The very warm, CO2 rich climate of the jurrassic was conducive to massive reptiles and massive spiders. I believe modern spiders are essentially no different--were we in the same climate, they would be growing up to near human size.
It was more an O2 rich atmosphere than CO2 rich atmosphere, and that is more relevant to invertebrates that do not have circulatory systems than vertebrates that have lungs, hearts, etc. Essentially, insect size is limited by how far molecular O2 can diffuse into tissues from the exterior environment, whereas this is not the case for an animal that can transfer O2 via hemoglobin or whatnot throughout its body.
In "Guns, Germs, and Steel" Jared Diamond theorizes that when earlier humans developed sufficient hunting techniques they were able take down mega fauna fairly easily, and quickly decimated the populations. The animals moved to slowly and couldn't adapt quick enough, and were simply killed off by humans a few tens of thousands of years ago. That might be why the many species of large land animals, from Australia to North America, died out fairly recently.
Diamond is referring to Paul Martin's Pleistocene Overkill Hypothesis. It has good evidence to support it, but is not without its flaws/holes.
There is also speculation that dinosaurs simply had a more efficient metabolism, but whatever the case modern birds do not get that large and mammals haven't been around nearly as long leaving the exact reasons up for debate.
Yep. I've researched bird mitochondrial physiology in comparison to bat mitochondrial physiology and birds have more efficient metabolisms at least in the context of mitochondria than most mammals (bats are exceptionally efficient, like birds). Whether birds evolved this after the deaths of their dinosaur ancestors or inherited it from them, I don't think anyone knows for sure.
I know this is your area, so I'll ask you instead of googling - isn't there a general trend of predators/prey to both get larger over time? If that's true, given enough time and a large enough populations, might we hypothetically expect mammals to find this or maybe some other solution, and then reach dino-like sizes on land?
Yes, there is a well-recognized phenomenon in evolution called 'phyletic size increase' or Cope's rule. It's by no means a biological law, but it happens often. There's lots of research into why animal size increases on average over millions of years. One of the leading general ideas is that larger animals face lessened predation pressure. Interestingly, humans are a great example of phyletic size increase - our earliest ancestors were barely 4' tall on average.