Besides the fact that the very attempt would change the basic efforts of the Western nations from democracy to fascism light?
It would have required all the WW II western allies to turn their collective backs on their former colonies, likely making it far easier for communism to spread as the "non-white" countries economies utterly collapse (followed in fairly short order by the Westrn nations as the lose many of their overseas markets). That alone would fundementally alter the Cold War.
Might be a better than average way to have the Pact survive into the 21st Century.
Hmm, not as dire as that. Democracies like the United States have restricted non-white immigration without becoming fascist, and preventing their citizens from migrating to Europe does not mean no investment in the former European colonies. In any case, even if all the colonial economies collapse, remember that the primary trading partners of all advanced nations are other advanced nations. Trade between Britain and Nigeria are nothing compared to trade between Britain and the U.S. or France.
The problem is the forms of nationalism, racism, and nativism most conducive to restricting nonwhite immigration were thoroughly discredited by the Nazis, so the Second World War has to be eliminated or changed so that does not happen. The philosophical opposite of Nazism, a sort of naive idealism that gripped the Western World after WWII certainly rules out the sort of cold blooded pragmatism that is the only non-racist reason for such restrictions. Europe also needs to avoid getting the spirit and will beaten out of it to keep traditional values.(and large families)