Yes in 1941 Torpedoes are some of the most expensive and complex weapon systems then in existence
A bomb and a depth charge are staggeringly simple by comparison and to store your best weapons in another nations base would require USN maintenance armorers to be present there.
Also 20,000 lbs is not a lot of weapons
OTL, Singapore was Hart's preferred fall back base. Hart suffered a massive insufficiency in fleet train. He had one large seaplane tender ,the Langley, that at times served also as an aircraft transport,which could handle 12 aircraft, 2 converted destroyers each capable og handling 4 Aircraft,,and a converted mine craft capable of handling 4 aircraft. Hart had 28 PBYs. 5 J2Fs, 5 SOC-3, and 1 OS2U.
Hart moved those munitions because his Tenders were full. He had also 1 destroyer tender, 3 submarine Tenders, ( one a converted cargo ship) and 2 oilers.