I think it was more the fact that oranges don't need to be transported in refrigerated ships, whereas most other fruits do. The more modern reefers were particularly valuable shipping not so much for the refrigeration but because they were quick, so didn't need to go in convoy. Prior to the ban on imports the 'banana boats' were sailing independently and getting away with it due to their speed.
After the banana ban those ships could either go onto importing something more important like meat or other products, or be converted to other purposes like RFA stores ship, ocean boarding vessel or whatever. The first RN escort carrier, HMS Audacity, was originally a banana boat, albeit a German one that had been captured.
Jamaica had also been pounded badly in the '39 hurricane season, so the banana plantations were in a bad way even before the war.
Pre-war there wasn't really a Dollar area and I'm not sure I see the driver for one post-war. Certainly I'd be surprised to see anything like Bretton Woods emerge, so I look forward to seeing what does happen as something has to change because the pre-war setup cannot be returned to.