No, they usually don't say that. They usually say they don't recommend it, and then less than 20, they say don't do it because they may crack. Not all of them crack, but not good to be taking chances at those temps anyway. The thing is, your tires may not be at 40 degrees due to several reasons, the air temp may be less, but the pavement will generally be warmer if it's been warmed by the sun for a few hours, or you drove from your garage where the temp was warmer, etc. Then with some driving, they tend to heat up a little too. It's not an immediate death sentence with temps at 40 degrees or anything. You do need to be watching for freezing temps though, as if you are getting down to 20 degrees, you are getting into the possibility of black ice (frost deposited on the road) and of course any other precip will be freezing somehow, so that's where you wouldn't want to be driving for various reasons.
Life is kinda short to not have a car with summer tires during summer...