As @[email protected] correctly highlighted, it wasn't even Latin that displaced so much of the local language. It was the Germanic tribals invading the islands later on. So I'll focus specifically on the Roman role.
The Roman process of Latinisation was rather slow. For reference: Gallia was conquered in 58-50 BCE, but odds are that Gaulish survived until the ~sixth? century of the common era, 600 years later. That's because the Romans didn't really give much of a fuck about what rural local folks spoke - if they rebel you kill them and done, problem solved.
Instead they were actively placing colonies in the conquered regions (to give land to Roman citizens) and converting the local elites to Roman habits and customs, because unlike the farmers the elites could be actually dangerous if rebellious.
The same applies to Britannia. Except that it was conquered ~a century after Gallia, it's a fucking island in the middle of nowhere with harder access, it doesn't grow grapes or olives, grain production in the Empire was mostly in Africa and Egypt so odds are that they couldn't reliably grow their wheat variety there either... really, the island was mostly a tin mining outpost.
Another factor to consider is the distribution of the Roman settlements in the area:
Are you noticing a pattern? Most settlements were in the Southeast, specially the larger ones (in yellow, full of Roman citizens). Perhaps not surprisingly the extant Brittonic languages are spoken further West, when you couple this with the tribal invasions. (That's simply because of the Fretum Oceani aka Strait of Dover. It was easier to reach the island by there.)