Environmental damage is an issue too. Availability of hardy crops means lands that used to be left alone for being marginal will be cultivated, which in turn means the native vegetation (ie. trees) will be wiped out to make room. Since most marginal lands are the hilly areas at the top of the watersheds, it can have pretty big knock-on effects. When deforestation of the uplands gets widespread it leads to fun things like increased soil erosion (landslides), increased water flow (flash floods), leaching of the soil (crop failures, wasteland), etc.
The Empire has pretty stringent rules about forest management so it shouldn't be too bad, but the government doesn't have eyes everywhere and laws can be repealed. What's really needed is for forestry and agriculture to be banned near running water.