In terms of difficulties with domestication, let's take foxes as an example. An animal that, while like dogs, has not shown the capacity to auto-domesticate the way the original post-wolf canines are supposedly to have done, finding an ecological niche based around following human bands around and experiencing natural selection for traits well suited to that niche. However, foxes can be domesticated; as that Russian breeding experiment with Siberian Foxes showed after 45 years of work.
Actually, I saw a Fox crossing the road last night, in an urban downtown area of Winnipeg. I was startled. I've seen raccoons occasionally, and rabbits, but this was the first fox ever.
Fox's are exquisitely effective vermin predators - rats, rabbits, mice, earthworms, you name it. They mature rapidly, have large litters.
So why aren't they domesticated?
Cats. The Verminator Niche in human society is occupied by Cats. So there's no real economic or social reason to go through the effort of domesticating Foxes.
Interestingly, I've read that the Ferret was probably a European domesticate. What happened? Cats pushed them out of the Verminator Niche.
It's simple economics really. A domesticated animal has to pay its way, it has to be valuable in the human economy. It has to contribute a use or a dollar value or a benefit that cannot easily be otherwise obtained and can't be supplied by harvesting the wild ones.
If you have one very efficient Verminator, you don't actually need a second. Once cats managed to occupy the 'catbird' seat in the economy, its hard to domesticate a second verminator. That's time, effort and short term unreliable results, when its cheaper and faster to buy a cat.
Now, there's a couple of interesting wrinkles. Ferrets managed to hang on as a domesticate or semi-domesticate, even though they largely got pushed out of their niches by cats. And a number of specific breeds of dogs - terriers and dachschunds were bred as Verminators, when the Vermin - big ass rats - were too nasty for cats.
The interesting thing about the fox in the city, is that clearly the animals are habituating to human presence. The Foxes are slowly meeting us half way. The trouble is that we don't really have a job opening for them in terms of the stuff they do and what we need.
Here's what I think though: If Feline HIV wiped out cats, then Foxes would be the new domesticate in a generation. And the henhouses would be armour plated.
But with potential food sources that can be more easily kept penned, eating the more ornery but still pen-able ones until the fight has been bred out of them and they are suitable towards the yoke is perhaps less outlandish?
The sad thing is that these days, all we want from animals is food sources. And sometimes a nice leather jacket.
Remember the good old days when they carried packs, we rode them around, they pulled plows, herded sheep, stood watch, killed mice and rats and generally put in an honest days work? Hell, its getting harder to come up with an economic use of animal labour that you can't get a machine to do cheaper and easier.
It's getting so bad, I hear that Cat geneticists are conspiring to build better mice, and technologist sheepdogs are working on ways to make sheep incompatible with GPS.
Anyway, if you want an interesting source on potentially untapped domesticables, check out this link on Micro-Livestock.
http://www.nap.edu/openbook.php?record_id=1831&page=R1
Though I would really like to know more about how OTL beasts of burden and food animals were domesticated, as I know nothing about that as compared to knowing a little bit about the dog situation. Also, did an agrarian society first domesticate goats, or were goats just tamed by pastoralists? That is, I'm having a hard time seeing the transitory period of domesticating or at least taming wild animals working out very smoothly for a nomadic people; with that not being a problem in the case of nomadic humans and the auto-domestication of dogs, by comparison.
There are multiple pathways. But what it seems to come down to is that you need an interface zone. An area on the borders of both settled and wild territory. In the heartland of settled territory, the wildlife is killed off - its free protein, no one is going to domesticate it. In wild territory, lots of wildlife, very few humans, and no one willing to make the effort.
So you generally need a geographical or ecological zone on the peripheries of core habitats. So even if you kill all the local wildebeest or llama, more keep drifting in slowly every year. Multiple points of contact over a long long span mean more domestication opportunities.
Second, generally people underestimate the factor that self domestication plays. Here's how it works. Startle a dog, and he'll jump and then look at you with a hurt expression. Startle a wolf, and five miles later, he's still running.
Now, that's partly a temperment issue. But what it really is, is human habituation. The potential domesticate starts getting used to human presence. Human presence is associated with opportunities, so they start to calm down around us, they're hanging around, chillaxing.
Look at it from the cats point of view: Human agriculture was a mouse buffet! Hot and cold running rodents, with an all you can eat menu. For Dogs, we were the ultimate picky eaters, you could get fat following them around and eating the leftovers. For horse and cattle, they were looking at huge cleared fields of millet, sorghum, barley, and the feebs were only eating the seeds. Free grass, particularly in winter.
In short, there was an advantage to them to hang around humans and get used to human presence.
These weren't the only ones. In the city, I've come within a dozen feet of a wild raccoon. There's 700,000 people, a downtown of high rises. And the raccoons are good with that. They know we are there, they're not bothered by that, they see all sorts of opportunities for the good life.
There are a lot of animals that habituate to human presence - seagulls, pigeons, the pariah dogs of India, raccoons, foxes, skunks, even bunnies and coyotes. They're all urban dwellers like the rest of us.
So that's half the battle. And the interesting thing is that when you model it out, most of the domesticates probably were domesticating themselves. They were animals who saw enough advantage to human presence and works that they just started hanging around and got used to us.
The other half the battle, what separates the cats and dogs from the raccoons and skunks, is that one group - in terms of their natural behaviour and habits, turned out to be useful to us. An animal that wakes up in the morning and lives and breathes for the purpose of killing mice? Hell, that's the answer to a farmer's prayer. A social animal that is highly territorial and inclined to functioning cooperatively? Hallelujah hunters, shepherds, etc.
On the other hand, what the hell is a raccoon going to do for anyone? They'd be a terrific domesticate, if someone could figure out something useful for them to do that's within their natural behavioural range.