it's not like people were fleeing to West Germany (where many former Nazis were still in power) in droves

They were though, when it was announced that the Berlin wall would shut down everybody was pretty much camping outside and the guards had to remind people that it would only happen at midnight through force