They had no idea that it was the human homeworld until they had already invaded. They found Earth based on searching for old Forerunner artifacts, and that's why they were there: they had manage to find out that there was supposed to be some important Forerunner remnants on the third planet of a particular system.
I think, I haven't actually played the games.
You are correct; the first covenant invasion of Earth was kind of an accident; they didn't expect it to be the human homeworld, and the heavy resistance that it contained. They paid for that; the flagship was the only ship that escaped out of the 15-ship fleet that arrived.
However, the doomed Covenant fleet figured out that this was the human homeworld, AND the site of the Ark, sent a message to High Charity (the Covenant capitol) and a much more MASSive fleet was organized and invaded Earth a few weeks later-several hundred ships, at least.
By the time Halo 3 starts, the Covenant have smashed formal human resistance, as well as several cities, but instead of continueing their campaign, they withdraw almost completely to East Africa, and begin diggin up the Ark. However, it should be noted that there were still several major human armies in hiding, as well as ships; it can be presumed that they either fled into space (assuming that the Covennant left the rest of Sol intact in their search for the Ark, which is not unlikely, IMO) or hid themselves, as shown in the first few levels (a large human military base).
