It's actually complicated.
I spoke with a PhD physical therapist about this (his undergrad work was in exercise physiology), and at about 40, all other things being equal, you're going to start losing muscle mass. By "all other things being equal", I mean that even if your diet is identical, you exercise at the same intensity, and through some previously unknown magic (e.g., drugs) you have identical hormonal levels to your 18yo self, you're still going to end up losing muscle mass and strength when you get somewhere in your 40s. He explained the basic outlines of the mechanism, but I simply don't have the science background to understand it, or to explain it.
That loss of muscle mass means that you're simply not going to be burning through calories. Muscle burns more calories per kilogram than fat does.
So, that's part of it; there's just a certain level of decline that's going to happen with age, and there's nothing you can do to prevent it.
But the other part is that activity levels do tend to sharply decrease for men (and women) outside of their 20s, while what they consume does not. Once you start having kids and/or other responsibilities, it's hard to find time to maintain the same level of physical activity that you might have previously had.