For heaven's sake man, the book is about the reversal of white-rule in the future - how blacks are equally capable of being the ruling race, dominating the world, and showing a paternal view towards the "savages".
Hugh Farnham, his daughter, and his new wife - the favored characters in the story - all have no problem with race. Farnham's son and old wife - portrayed as pathetic idiots - are the main racist characters (Farnham has to reprimand his son for calling Joseph a nigger). Joseph is a sympathetic character who eventually collaborates with the black ruling establishment, partly out of spite, partly to seek opportunity for himself, and partly because he has little other choice.
Perhaps some quotes would help you?
Some real conversations:
[1]
The Farnham men have a fistfight shortly after arriving in the future:
Duke clouted his father. "That's for bullying Mother!" he clouted him from the other side and harder, knocking his father off his feet. "And that's for having that nigger pull a gun on me!"
Hugh Farnham lay where he had fallen. "Not 'nigger,' Duke. Negro."
"He's a Negro as long as he behaves himself. Pulling a gun on me makes him a goddam nigger. You can get up. I won't hit you again."
[2]
Duke Farnham: "Nuts! I simply want things run democratically."
Hugh Farnham: "You do? Shall we vote on whether Grace is to work like the rest of us? Whether she shall hog the liquor? Shall we use Robert's Rules of Order? Should she withdraw while we debate it? Or should she stay and defend herself against charges of indolence and drunkeness? Do you wish to submit your mother to such ignominy?"
"Don't be silly!"
"I am trying to find out what you mean by 'democratically.' If you mean putting every decision to a vote, I am willing - if you will bind yourself to abide by every majority decision. You're welcome to run for chairman. I'm sick of the responsibility and I know that Joe does not like being my deputy."
"That's another thing. Why should Joe have any voice in these matters?"
"I thought you wanted to do it 'democratically'?"
"Yes, but he is -"
"What, Duke? A 'nigger'? Or a servant?"
"You've got a nasty way of putting things."
"You've got nasty ideas...."
[3]
Barbara discussing the idea of marriage with Joseph (yes, that's the black character)
Joe? My admiration for him is unqualified -- and he doesn't have a mother problem.
Joe is the first Negro I've had a chance to know well -- and I think most well of him. He plays better contract [bridge] than I do; I suppose he's smarter than I am. He is fastidious and never comes indoors without bathing. Oh, get downwind after he has spent a day digging and he's pretty whiff. But so is Duke and Hugh is worse. I don't believe this story about a distinctive "nigger musk."
Have you ever been in a dirty powder room? Women stink worse than men.
The trouble with Joe is the same as with Duke: No spark jumps. Since he is so shy that he is most unlikely to court me -- Well, it won't happen.
[4]
Karen Farnham is also considering the problem of husbands when there are only two available men - one of them her brother.
Hugh: "And I so take it. Do I understand that you have eliminated Joseph? Or have you considered him?"
"Certainly I have."
"Well?"
"How could I avoid it, Daddy? Joe is nice. But he's just a boy, even though he's older than I am. If I said, 'Boo!' he would jump out of his skin. No."
"Does his skin have something to do with your choice?"
"Daddy, you tempt me to spit in your face. I'm not Mother!"
"I wanted to be sure. Karen, you know that color does not matter to me. I want to know other things about a man. Is his word good? Does he meet his obligations? Does he do honest work? Is he brave? Will he stand up and be counted? Joe is very much a man by all standards that interest me. I think you are being hasty."
He sighed. "If we were in Mountain Springs I would not urge you to marry any Negro. The pressures are too great; such a marriage is almost a tragedy. But those barbaric factors do not obtain here. I urge that you give Joe serious thought."
[5]
Duke and Hugh arguing:
Duke: "Crap. There never was a nigger bastard who wouldn't rape a white woman if he had the chance."
Hugh: "Duke! That's poisonous, insane nonsense. You almost persuade me that you ARE crazy."
"I-"
"Shut up! You know that Joseph, to give one example, had endless opportunity to rape any of three white women, for nine long months. You also know that his behavior was above reproach."
"Well...he didn't have a chance to."
"I told you to shut up this poison. He had endless chance. While you were hunting, any day. He was alone with each of them, many times. Drop it! Slandering Joseph, I mean, even by innuendo. I'm ashamed of you."
[6]
Joe, now an up-and-coming member of the black ruling class, has an argument with Hugh. Hugh speaks first:
"Joe, do you know what you sound like? Like some white-supremacy apologist telling how well off the darkies used to be, a-sittin' outside their cabins, a-strummin' their banjoes, singin' spirituals."
Joe blinked. "I could resent that."
Hugh Farnham was angry and feeling reckless. "Go ahead and resent it! I can't stop you. You're a Chosen, I'm a servant. Can I fetch your white sheet for you, Massah? What time does the Klan meet?"
"Shut up!"
Hugh Farnham shut up. Joe went on quietly. "I won't bandy words with you. I suppose it does look that way to you. If so, do you expect me to weep? The shoe is on the other foot, that's all - and high time. I used to be a servant, now I'm a respected businessman - with a good chance of becoming a newhew by marriage of some noble family. Do you think I would swap back, even if I could? For Duke? Not for anybody, I'm no hypocrite. I was a servant, now you are one. What are you beefing about?"
"Joe, you were a decently treated employee. You were not a slave."
The younger man's eyes suddenly became opaque and his features took on an ebony hardness Hugh had never seen in him before. "Hugh," he said softly, "have you ever made a bus trip through Alabama? As a 'nigger'?"
"No."
"Then shut up. You don't know what you are talking about."
[7]
Hugh Farnham reflects on racial differences:
This matter of racial differences - or the nonsense of "racial equality" - had never been examined scientifically; there was too much emotion on both sides. Nobody wanted honest data.
Hugh recalled an area of Pernambuco he had seen while in the Navy, a place where rich plantation owners, dignified, polished, educated in France, were black, while their servants and field hands - giggling, shuffling, shiftless knuckleheads "obviously" incapable of better things - were mostly white men. He had stopped telling this anecdote in the States; it was never really believed and it was almost always resented - even by whites who made a big thing of how anxious they were to "help the American Negro improve himself." Hugh had formed the opinion that almost all of those bleeding hearts wanted the Negro's lot improved until it was ALMOST as high as their own - and no longer on their consciences - but the idea that the tables could ever be turned was one they rejected emphatically.
Hugh knew that the tables could indeed be turned. He had seen it once, now he was experiencing it.
But High knew that the situation was still more confused. Many Roman citizens had been "black as the ace of spades" and many slaves of Romans had been as blond as Hitler wanted to be -- so any "white man" of European ancestry was certain to have a dash of Negro blood. Sometimes more than a dash. That southern Senator, what was his name? -- the one who had built his career on "white supremacy." Hugh had come across two sardonic facts: This old boy had died from cancer and had had many transfusions -- and his blood type was such that the chances were two hundred to one that its owner had not just a touch of the tarbrush but practically the whole tar barrel. A navy surgeon had gleefully pointed this out to Hugh and had proved both points in the medical literature.
Nevertheless, this confused matter of races would never be straightened out - because almost nobody wanted the truth.
[8]
Finally, Barbara and Hugh, right at the end of the book:
Barbara said stubbornly, "Hugh, how many white men of today could be trusted with the power Ponse had and use it with as much gentleness as he did use it?"
"Huh? None. Not even yours truly. And that was a low blow about 'white men.' Color doesn't enter into it."
"I withdraw the word 'white.' And I' sure that you are one who could be trusted with it. But I don't know any others."
"Not even me ...."
I honestly do not understand what the problem is here. All this was blindingly obvious to me. It seems that if Heinlein had one real problem, it was that he had too high an opinion of his readers. By your standards I imagine Huckleberry Finn would be White Supremacist literature as well.
If I were you I'd reread Heinlein. If you've gotten one book that far wrong, well, odds are you may have missed something else.