I don't disagree that Casimir had a stronger position than later Polish kings and did much to reverse the issues from previous reigns, but you can't deny that the trend towards favoring the nobility began in his reign. He had to make significant concessions to ensure their military support throughout his reign, and later concessions were made to secure Louis of Hungary's succession to the crown: he lowered their tax burden and ensured they wouldn't have to pay the costs of military expeditions outside of Poland. Whatever good he did was absolutely squandered in pursuing the succession of Louis of Hungary; which even at the end, he attempted to undo by making Casimir IV of Pomerania his heir instead.
The number of that concessions was actually zero, there wasn't a single privilege issued by him (privilege of Buda of 1355 was issued by Louis and was promise only fulfilled in the event of Casimir's death and until Casimir actually died, it wasn't law, Casimir collected taxes as much as he pleased). Trend actually began even earlier, during so-called feudal collapse (in Polish: "rozbicie dzielnicowe") of Poland, but unification reversed most of it and it resurfaced with Casimir's own succesion problem. Casimir IV thing was illustration what was wrong with Poland of that time, mid-tier power wasn't enough to make sovereign policy when surrounded by much stronger neighbours.
While Louis was probably keen to maintain a union between Hungary and Poland even after his death, I don't see how this would be possible barring him having a son. Certainly the Polish had little interest in having Sigismund as their king, and Jagwiga's succession meant that her original betrothal with William of Austria was broken. Hungary dealt with their own issues in the aftermath of Louis' death, there was a lot of unrest following Mary's succession, and her eventual death meant that Sigismund had issues maintaining his power there, not to mention his focus elsewhere, especially after he became Holy Roman Emperor, the Hussites, ect. Sigismund did favor an anti-Polish policy, but the Battle of Grunwald basically meant that it ended in failure.
If Jadwiga had children post union with Lithuania these children would be legal heirs of Hungary and as you said, Hungarians had little problem with declaring Polish princes as Kings, but that isn't necessarily good for Poland as state (only real incorporation of Lithuania and utter conquest of TO could help Poland avoid being Hungarian province in any union with Hungary), and that's the problem, only solution in late XIVth century which might stop the tendency of the system to autodestruct could be also way to negate Polish independence way before it was lost IOTL.
Hungary didn't have ideal situation, but it had vast financial advantage over Poland due to abundant gold resources in modern-day Slovakia, it had better soil and was overall richer.
The factor which allowed Austria to dominate Hungary was that it was destroyed by civil war, maybe that could be replicated if Naples, Sigismund and Hedwig's son all warred for that crown, it could be replicated, but if takeover was smooth, it'd strenghten Hungary.
