Bill Glod, a philosopher who works at the libertarian Institute for Humane Studies (I used to work there, too), offers an interesting short discussion on the limits of the common libertarian conception of liberty as non-interference:
[M]any libertarians themselves endorse liberty as just being non-interference. But I think this framework limits how we conceptualize liberty: in terms of being left alone from interference versus having an equal share of voice in a democratic decision procedure, or being left alone versus being free of arbitrary power. I think sometimes the temptation to dichotomize possible conceptions of liberty leads us to overlook the important connection between recognizing social rules and lawful authority on the one hand, and the normative reasons we have as moral agents on the other hand.
So, what about another alternative conception that people sometimes overlook: freedom as not being subject to coercion whose rationale fails to accord with one’s reasonable normative standards (one’s values, beliefs, etc.)? Any contender authority – democratic, civic republican, or a liberal property rights regime premised upon giving people jurisdiction to make their own decisions free of interference – must accord with a person’s reasonable normative standards or else the person subject to this authority is unfree. Freedom is not about merely being always free of interference (if that is even possible or desirable). Nor is freedom about being free of another’s arbitrary will – as if being subject to a _non-arbitrary_ will solves the problems that motivate us to seek a proper conception of freedom. Rather, freedom is about not being subject to an external will at all. It’s about self-rule.
I like Bill's notion of freedom as self-rule, but I think specifying the meaning of "self-rule" is not at all easy, and does not obviously lead to what most of us would consider a "libertarian" conception of freedom. Of course, we're all constantly subject to the wills of others. People are constantly enjoining and entreating and wheedling and shaming and peer-pressuring and so forth. One doesn't want to say that self-rule or autonomy requires total immunity from the influence of others. And it's plainly circular to say the problem is being subject to an external will in a way that limits our freedom. But I think noting that helps us to see that the question is not really one of being subject to an external will or not, but of the way in which one is made subject to an external will.
It seems pretty plausible that subtle psychological manipulation, "brainwashing," and even just internalizing the norms of a culture that discourages the development of a sense of independent self-efficacy can make one the subject of a will not really one's own. If the capacity for self-rule has developmental and ongoing material and psychological preconditions, freedom as self-rule can require a whole lot more than immunity from physically coercive interference. Suppose self-rule requires a certain level of self-regulation or self-control which can be destroyed by addiction. In that case, it seems plausible that paternalistic measures to prevent or break addiction could be freedom-enhancing. Or suppose certain religious sects prevent girls from developing their capacities for effective self-rule. In that case it seems plausible that a ban on certain religions or religious practices could be freedom-enhancing. But this doesn't seem very libertarian, does it?
It seems to me that every conception of freedom or liberty when stated in broad outlines is relatively indeterminate. In order to arrive at a recognizably "libertarian" version of a conception of freedom requires filling out the conception in not-at-all obvious ways. This is true even of the classic libertarian conception of liberty as non-coercion. Generally, libertarians rely on a tendentiously loaded conception of coercion that simply stipulates that commonsense forms of emotional, psychological, and social coercion aren't really coercive in the relevant sense. And then they tend to want to tack onto their conception of liberty a notion of fraud that has no obvious connection at all to their narrowly-specified notion coercion.
I don't mean to pick on libertarians. Everyone who professes to care about liberty does the same sort of thing. It seems to me that most of our high-level political concepts like "freedom" or "equality" are tailored and tweaked to justify the kind of political regime we already tend to favor. If you are offended by taxation, you'll settle on a conception of liberty according to which taxation is a violation. If you think a relatively high level of taxation is necessary to give people what you think they ought to get, you'll settle on a conception of liberty according to which taxation is not a violation, but not giving people what you think they ought to get is. That's why abstract political philosophy is so often futile. It's probably more useful to start out arguing over regime types in the first place, since mostly what we do is choose our favorite regime type and then reason backwards to conceptions of liberty, equality, and so forth that justify our pick.