I have encountered some libertarian atheists, and a video of Penn Jillette talking about his version of libertarianism I screamed BULLSHIT!
1. It's impossible. Libertarianism is impossible except for survivalist nutters and hermits. Humans need a social structure because we're a social species. Trusting that humans can moderate their own behavior on their own is just a fantasy. If we were as sparsely distributed as wolf packs, we could get along in our small groups without intervention from a higher authority in theory.... but the matriarch or patriarch would be the higher authority so even that isn't strictly libertarian. Anyway, we're long past the point in evolution where we could manage our behavior without a formal structure. Michael Shermer theorizes the optimal size for a human community to manage without any oversight is about 150. Those days are gone.
2. It's naive. It assumes people are basically good. This is a nice thought, and a refreshing break from the Christian belief that all people are sinners who need to be saved, but it's just plain wrong. Just as we differ in our DNA we differ in our personalities. Some of us will go through life making very few decisions that negatively impact others, and some of us are sociopaths. At its best, government protects the truly good from the sociopaths. Without a government, we would be reduced to lynch mobs which can only avenge bad deeds, not prevent them. And we certainly wouldn't have something like the FBI, which can trace the path of a serial killer from one area to the next based on DNA evidence, etc. I think the people who believe that "survival of the fittest ergo libertarianism" probably assume they are the fittest themselves. They don't think that they would be the victims of a sociopath. Bernie Madoff counted on this kind of hubris to make his illegal millions.
3. It's cold-hearted. For example, regulations about safety in cars aren't needed because over time car companies would be forced to make safer cars or they'd go out of business. So the people who died in fires caused by exploding gas tanks in Ford Pintos, or in wrecks caused by the design of their Corvair were just collateral damage in the evolution of better cars. People who died because of unregulated businesses did nothing to deserve that fate, except perhaps not be able to afford better cars. And the pseudo-Darwinism of libertarianism really doesn't care what the strong do to the weak. Rich and powerful people are good and deserve to be rich and powerful. The poor and powerless deserve what they get.
4. It ignores history. We haven't always had a U.S. government. It's only a little more than 200 years old. But we do know earlier forms of society. We've had monarchies. We've had theocracies. We had the ancient Roman & Greek systems that privileged people with money. Modern democracy certainly has its failings, but we really be better off returning to "less" government considering what our previous systems gave us?
5. It's not natural. The underlying assumption of libertarianism is that government is an artificial construct that interferes with natural behavior, which they believe works just fine on its own. There's no evidence that humanity could have survived without some form of social organization. The instinct for survival that causes some to climb to the top of the heap and others to hide from the climbers just doesn't result in a society that works for large numbers. It probably won't work for small numbers, either.