Bearing in mind I'm a total amateur...
One thing that shouldn't be forgotten is dialects. In some medieval periods, the inhabitants of one valley village would barely be able to understand those of a valley in the neighbouring county, despite both speaking 'English.' Just think of the modern variations between British, American, and Australian English, let alone comparing Scouse, Manc and Geordie! As languages were written down, they became more codified - spellings were determined, even if pronunciations shifted (which is why some modern words are not spelt as they sound, and some of Shakespeare's rhymes seem dodgy - in his day, they did rhyme). (The fact that Italian and Spanish speakers can usually understand the other language is another unusual linguistic fact, given that France and Germany are in the way between them, but not the point.)
It might be logical, then, that the Brythonic languages - because it was unlikely to be merely one language - gradually got swamped or infiltrated with loan-words until Germanic languages replaced them. In addition, parts of northern England, especially in the east, gained a strong Norse influence, with the Danelaw and such (parts of Scotland too - anyone looking at Scots will sometimes notice Norse influence, and Manx surnames are fascinating in this respect).
There's also the current belief that the Celts were not a people as much as a group of peoples with similar languages and cultural elements, which may or may not have anything to do with it.
I'd argue that the biggest difference is probably in the actual people doing the 'invading' - most of the Germanic people were probably of a similar rank to the Celts, merging into family lines and interacting at a similar level. If my understanding of the Norman invasion is correct, most of what they did was replacing the local nobility - French became the language of the court, but the common people weren't expected to know that. So while there was a great deal of influence on the language, it was more of a merger with English remaining dominant than outright replacement. Which is probably why English has the most words of any given language, and the common Internet memes about English being three languages stacked on top of each other wearing a trenchcoat and that English didn't develop as a language - it lurked around dark corners and attacked other languages and routed through their pockets for loose vocabulary came from!