I don't think preventing invasion was possible unless Pol Pot cuts ties with Communist China, which would require a radically different Pol Pot I guess (one not crazy enough to think that the Cultural Revolution was worth copying).
Furthermore, it's probably safe to say that postwar Vietnam considered the former French Indochina as basically its 'strategic sphere' - given the immense 'help' the two countries were for the Vietcong/Viet Minh - and so it's probably inevitable that at some point, Vietnam was going to try and control Laos and Cambodia. So the only way for Pol Pot to not fight the Vietnamese was to align with Vietnam & the Soviet Union.
Given this, it's probably safe to say that such a dictatorship could not have gotten as out-of-hand as it did in reality. The Soviet Union and Vietnam would probably have used their influence to stop the massacres.