The official casus belli was over impressment, but several of the War Hawks were openly pro-annexation; Senator Richard Mentor Johnson for instance openly called for it. Others may have been less enthusiastic.
Given that everything else about the American participation in the War of 1812 was an incoherent mess, why should it be surprising that motives were equally incoherent?
Here are the Canadian historian Patrick White's views of why Canada was mentioned so frequently in the congressional debates leading up to the war in *A Nation on Trial: America and the War of 1812*:
"If war were declared, where was it to be fought?..The answer to this
conundrum lay at hand, however...Porter admitted that it would be folly to
contend with Britain on the ocean, but...'We could,' he suggested, 'deprive
her of her extensive provinces lying along our border to the north.' Not
only were these colonies valuable in themselves, but they were also
'indispensable' to Britain now that she was cut off from Europe. William
King, who opposed territorial expansion, insisted that Canada be taken in
order to 'wound our enemy in the most valuable part...' ...Lowndes of
South Carolina advanced the same argument. He ridiculed Randolph's view
that England could not be injured. American ships could damage British
trade, and American troops could conquer Canada. Harper of New Hampshire
insisted that the Canadas as well as New Brunswick, Nova Scotia and the
Bahamas should be taken. This conquest was the only way that the United
States could exact payment for past damages and guard against injuries
from future acts. Even the writers of a memorial from Massachusetts
admitted that Canada was the only point at which Britain was assailable.
"These speeches explain the real reason for the persistent references to
Canada in the Twelfth Congress. It was not land hunger or Indian unrest
that drove the United States to envisage the conquest of British North
America. The idea that the United States would enter into a war with
Britain for land that she did not need does not stand scrutiny. The
theory that a coalition of frontier and Southern interests supported war
so that one would take Canada while the other secured Florida is also
unsubstantiated. In 1810 West Florida was occupied by America without
incident. The rest of that area could be taken in the same manner.
There was simply no need to declare war on Britain in order to despoil
Spain. Indeed, to have done so would have been costly, devious, and
senseless. And whatever we may think of Madison and the members of
Congress, they were not complete fools. They were in fact deeply
distressed men who were desperately anxious to find the means to combat a
formidable foe. Together they struck upon the obvious and only method to
wage a war [i.e., to invade Canada]....Only Randolph's tortuous mind could
read motives of greed and expansion into their discussions. There have
been wars fought for profit (and the cost of these have usually far
exceeded the gain) but the War of 1812 was not one of them." (pp. 105-6)
I would add one thing: The South was avidly pro-war, yet except for Kentucky (which feared British influence over the Indians of the Northwest Territory), the South had no material interest in acquiring Canada, which obviously would not be promising ground for her peculiar institution...The reason Southerners talked about Canada was quite simply because it was the only place the hated British could be fought.
