Honestly, I’m not sure how much potential there is to expand. Hokkaido’s an obvious one that could probably be done, and the Ryukyus north of Okinawa, but after that, choices are limited. Both Korea and Okinawa are going to be in the Chinese sphere, so an attack on them is going to wake the dragon and cause problems. If they strike during a period of fragmentation in China, they could stand a decent chance of holding something, but once a new dynasty rises, they’ll want to rebuild their sphere of influence. What better way to ensure Korea’s loyalty to China than by helping them expel these island invaders? That said though, China might be willing to let Okinawa slide since its small and not that important, but it will raise some alarms in China about Japan muscling in on their turf.
Japan and China are natural rivals, as evidenced since the 7th century when Japan supported Baekje against Silla (and their Tang allies), the Yuan invasion, or the Imjin War. The analogy is England vs France, with the weaker England and their naval superiority (which premodern Japan never had, but likely could have developed) vs the stronger France and their land superiority. The likely main theater is Korea, where at some point we'd have a parallel to today's DPRK vs ROK with a pro-Chinese government (maybe Goryeo or Joseon) vs an ATL pro-Japanese ally (maybe called Baekje after the historic Japanese ally). I believe it's possible for Japan to do better in an attempt to conquer Korea than Hideyoshi's attempt--had they simply propped up a puppet state instead of Hideyoshi's insanity, they would've done much better. Ryukyus wouldn't be too complex to sort out with China, and even Taiwan (Takasago) is feasible though it would likely become a Chinese target in the long run (as it did at the end of the Ming era).
A big one is Hokkaido/Ezo being opened to Japanese settlers, which means Sakhalin/Karafuto (and probably the Kurils/Chishima) becomes the "new Hokkaido" in the sense of its Japanese merchants operating under one clan purchasing goods from the natives and thus enriching them and irretrievably altering their lifestyle. The Chinese claimed Sakhalin until the late Qing, so this would lead to a conflict, and the Yuan OTL did spur conflict in that area with their Nivkh allies.
Some Japanese historians have suspected this was a plot to help the Yuan attack Japan from the north, and although this never panned out (the end result was the non-Ainu groups in Hokkaido vanished and the Ainu themselves migrated to Karafuto), it does seem feasible that a Yuan-supported war like this combined with a successful Yuan invasion in Kyushu would lead to a seriously differently Japanese mindset toward their north. In the 13th century, there were still Ainu tribes in the northernmost parts of Hokkaido, so theoretically Mongol related warfare could result in disturbances there that would eventually reach the ears of the Kamakura Shogunate. I believe this could result in the perception of a Japan under siege that would inevitably result in an expansion into Hokkaido.
TTL's Hokkaido settlement would be a series of wars with Yuan vassals like Nivkh tribes in Hokkaido (Okhotsk culture people) and various Ainu tribes and inevitably result in Japanese victory through sheer numbers. The Japanese demand for furs would be transferred to the Kurils/Chishima and Sakhalin/Karafuto which have less to offer. I believe this will naturally result in Japanese expansion toward Kamchatka in the long run, especially once they expel the Chinese/Mongols, since IMO they can take Kyoto but most anti-Mongol lords in Japan will retreat toward eastern Japan, probably depose the Kamakura Shoguns for their incompetence, and defeat the overextended Mongols and their native allies near Sekigahara (the natural point between east and west Japan). This will combine with the pre-existing struggles with the Imperial Court and result in a far more violent Nanboku-cho period which will probably be Japan's equivalent of the Hundred Years War and War of the Roses combined.
Outside of Japan’s immediate vicinity, distance is going to be a major issue. Pacific islands like the Marianas and Carolines could be jumping off points for a Japanese hegemony in Oceania, but provided the medieval Japanese even know these isolated and far-off islands exist, holding them when they’re separated from the mainland by a vast expanse of ocean would be hard, and probably not worth it at all. Southeast Asia also has the distance problem, coupled with the fact that the countries in this region are battle-hardened from constant fighting amongst themselves, and won’t go down easy. There’s still Siberia, however, it’s Siberia. not exactly an attractive prize, I don’t think hypothermia is considered an honorable death in bushido.
Southeast Asia is the natural target as the
Nihonmachi of the 16th century were focused there. Vietnam may be a natural Japanese ally, although IMO Japan's best ally in the region is a southern Chinese state like Southern Ming since Japan needs China divided. The ideal scenario is having Qing, Shun (or another northern Han dynasty), and Southern Ming having a decades long standoff.
Siberia is helpful for furs which at this point will be increasingly overhunted by the tribal vassals of Japan. Japan can also make contact with the Russians, assuming no butterflies in their history. Let's keep in mind that the late 18th century European expeditions to Kamchatka recorded that the natives (both Ainu and Itelmen and IIRC Koryak too) were aware of the Japanese, even if it was mostly secondhand knowledge from the Kuril Ainu who appeared to have been very good middlemen.
An alternative direction, following the currents to Alaska, is also natural and probably would be done TTL. It leads to more furs, ivory (from the Chukchi and Yupik), and jade (from the Tlingit). I'm not sure about the ivory trade in Kamchatka, but in Alaska it was highly valued by the Tlingit and procured at great cost and from far away from the otherwise disliked Yupik, Alutiiq, etc. Jade came from the Athabaskan peoples of northern BC and Yukon. I believe that (along with fantastic fishing ports) is what would draw the Japanese to the New World. With experience in Hokkaido, the marginal lands of the Cook Inlet (Anchorage, Kenai Peninsula, Mat-Su Valley) can be settled by Japanese farmers and fishermen. Buddhists will probably spread Pure Land ideology among the natives.
This will open up lands further south to acquire more jade, furs, etc. and if Japan can discover the Columbia River (not a given, it's shrouded in fog and its mouth is famously deadly given the sandbar) than they've become a powerful fur trading nation and can probably seize the Northwest down to the Klamath River or so once they contact and dominate the Chinookans. The Chinookans would essentially be like the Creek and other Five Civilised Tribes were in the east given their dominance as merchants along the Lower Columbia. They'd probably be joined with allied Salish, Sahaptin (Klickitat, Yakima, etc., the tribes who gathered at Wyam/Celilo, the largest trade center between the Rockies and Cascades), and Wakashan-speaking peoples resulting in a situation akin to the 18th/early 19th century (an explosion in population, creativity, and violence among PNW peoples) or the change in Ainu culture and the Ainu construction of
chashi (a very similar phenomena), or for that matter late 18th/early 19th century New Zealand.
Unlike OTL, the PNW is close enough to Japan that it will likely be a place where Japan sends its surplus population, especially since diseases will eventually decimate the natives. East of the Cascades and the Central Valley of California will probably be a buffer, controlled by native allies and mixed-race merchants. East of the Cascades will be easier to seize from European powers, but the Central Valley might be lost. At any rate, Eastern Washington/adjacent areas (possibly the entire Columbia watershed) is an incredible prize given the resources and agricultural potential.
I think the Japanese may very likely follow what is popular called the "French" model of colonisation and their "empire" would be merchants, mixed-race people, and native allies. Considering the history of the PNW, it would be incredibly fascinating what might result from these events. I think the classic "Northwest Indian" elements (i.e. totem poles, house posts, the classic potlatch) would thoroughly spread to the Coast Salish and Chinookans (OTL the former received it very late and the latter barely at all), and possibly to groups further south on the Oregon Coast and further inland (who OTL had only bare elements of it, if any at all) given they'd have an extra century or so compared to OTL.
In the end, the natives and *Metis will be subdued by the power of the Japanese or whatever government emerges in the New World, but that will pave the way for a very powerful local nation. Many of Japan's needs, from wood to coal to good agricultural land, are present in the Pacific Northwest.
Japanese trade before the 1500s was entirely between it and Korea/China, there is little reason to expand elsewhere, the Emishi subjugations were mostly to stop their raids in northern Tōsandō, once they retreated into Ezo expansion virtually stopped.
As with most of those scenarios, the main question is: why is Japan expanding outside the home islands? Conquering for conquering's sake wasn't really a thing before the 19th century.
I think the Imjin War and the plans to invade both Taiwan and Manila show Japanese leaders could easily find an interest in expansion. The challenge is giving Japan a reason, and while it's become cliche, the Mongols winning in Kyushu is probably exactly the spur since inevitably the Yuan will start pressing on the Nivkh and Ainu. At the very least, they'll end up with Hokkaido from it and the OTL equivalent with one Japanese clan controlling native chiefs extracting furs in exchange for Japanese goods will move further north to Karafuto, the Kurils, and Kamchatka.
but japan before middle ages was neither politically united nor literate
It was in a loose sense given how Heian Japan worked.