It's the irreverence with which they are used. If the average European medieval peasant affirm their sincerity and honesty by saying "May God damn me to hell if I lie", that's an (implicit) oath. They're putting their salvation on the line. That's how serious the matter at hand is.

If I casually say "damn, that ass", I'm using a boiled down version of that (when the oath formula becomes so widespread, people start omitting words because everyone knows what you mean anyway, even if you just say "God Damn me" and eventually just "damn"). But I'm not doing it out of a devout belief that the thing I'm saying warrants reinforcement by invoking divine wrath. I'm abusing the sanctity of good for an entirely profane matter. I'm reducing God's power and wrarh to a colloquial tool.