Bermuda would likely remains special case. The Bahamas are not likely to be let go, they're just to valuable, but Bermuda itself is not improbable as American territory. Likely it gets attached to Virginia as a county and eventually becomes its now territory or state.
Bahamas was a shoot-off of Carolina in the vein Bermuda was of Virginia, though - most of its population tripled (!) with the Loyalist exodus as was a massive amount of slaves being brought in to farm sea cotton. They never had sugar as a major crop (although I could be wrong in this). They aren't even geologically part of the Caribbean plate, but rather North America!
This sudden influx won't happen in TTL but the same fate will happen to Bahamas - island appendage of the Deep South, sea cotton, hell of a convenient port - just more gradually.
Look at the big picture of the Bahamas in a cultural-historical sense and it's influenced as much by America as it was Britain. Arguably the same for Bermuda. It's no worse than how the Maritimes share a ton of culture even to this day with New England, or how the Pacific Coast of both America and Canada are startlingly similar in day-to-day-life and culture.
The Bahamas would become a state. Bermuda would be administered from whichever of the Eastern Seaboard states appeased the federal government enough.
One could argue why Delaware isn't part of Pennsylvania still, then. If Bermuda is a separate government - and it was - then it shall remain so.
Surely there had to be some legislation prior to 1860/1861 that came down to only just a few votes in the Senate that the North won? I'd think that 4 new Pro-South Senators could possibly have some big butterflies. If they prevent the Civil War even longer, perhaps even to its latest plausible date (late 1860's/early 1870's, I would assume) then...well...yeah, the butterflies are endless.
I don't disagree, but I believe (off the top of my head) by 1861 the Northern states had slightly more senators anyways thanks to California coming in entirely as a free state, which was why Southland seceded. Factor in one could swing a senator here or there for political purposes and I suspect that history would remain quite similar in a political sense.
EDIT: 17 slave states in 1861 to 19 free states, adding in Bahamas and Bermuda.