Exactly. In most countries, you reason that you never need to count floors unless you are going up or down. If you are walking up stairs, each floor you go past, you count it: F1, F2, F3, etc. If you are walking down stairs, you count each floor you go past: B1, B2, B3, etc.

Americans think about it more like a cake. Each "story" or "floor" is a ~3m or 4m, floor-to-ceiling, architectural layer. You don't look at a 3-layer cake and say "that cake has a ground layer, then a first layer and a second layer" you say "that cake has three layers".