After distinguishing his topic from free will, John Stuart Mill launches into a history-in-a-nutshell of the origins of the idea of liberty. The starting point for this history, is the point made in Leveling Up: Making the Transition from Poor Country to Rich Country and echoed in The Government and the Mob:

Designing strong but limited government that will prevent theft, deceit, and threats of violence, without perpetrating theft, deceit, and threats of violence at a horrific level is quite a difficult trick that most countries throughout history have not managed to perform.

Or as I wrote in “Why Thinking about China is the Key to a Free World,”

Freedom is a rarity in human history, and still too much of a rarity in the world today. This should be no surprise. Would-be tyrants abound, and it is not easy to establish a system that keeps them all in check.

Chaos and anarchy tend to make life nasty, brutish and short. So some kind of government is necessary (even if that government is called a “security firm” as in some discussions of anarcho-capitalism). But anyone or any institution strong enough to keep order is strong enough to be a potential danger to the freedom of everyone else from being bossed around or worse. Thus, a key aspect of liberty is what limits can be placed on the ruler. This is the gist of the rest of the remainder of the first two paragraphs of On Liberty:

A question seldom stated, and hardly ever discussed, in general terms, but which profoundly influences the practical controversies of the age by its latent presence, and is likely soon to make itself recognised as the vital question of the future. It is so far from being new, that, in a certain sense, it has divided mankind, almost from the remotest ages; but in the stage of progress into which the more civilized portions of the species have now entered, it presents itself under new conditions, and requires a different and more fundamental treatment. The struggle between Liberty and Authority is the most conspicuous feature in the portions of history with which we are earliest familiar, particularly in that of Greece, Rome, and England. But in old times this contest was between subjects, or some classes of subjects, and the Government. By liberty, was meant protection against the tyranny of the political rulers. The rulers were conceived (except in some of the popular governments of Greece) as in a necessarily antagonistic position to the people whom they ruled. They consisted of a governing One, or a governing tribe or caste, who derived their authority from inheritance or conquest, who, at all events, did not hold it at the pleasure of the governed, and whose supremacy men did not venture, perhaps did not desire, to contest, whatever precautions might be taken against its oppressive exercise. Their power was regarded as necessary, but also as highly dangerous; as a weapon which they would attempt to use against their subjects, no less than against external enemies. To prevent the weaker members of the community from being preyed on by innumerable vultures, it was needful that there should be an animal of prey stronger than the rest, commissioned to keep them down. But as the king of the vultures would be no less bent upon preying upon the flock than any of the minor harpies, it was indispensable to be in a perpetual attitude of defence against his beak and claws. The aim, therefore, of patriots was to set limits to the power which the ruler should be suffered to exercise over the community; and this limitation was what they meant by liberty. It was attempted in two ways. First, by obtaining a recognition of certain immunities, called political liberties or rights, which it was to be regarded as a breach of duty in the ruler to infringe, and which, if he did infringe, specific resistance, or general rebellion, was held to be justifiable. A second, and generally a later expedient, was the establishment of constitutional checks, by which the consent of the community, or of a body of some sort, supposed to represent its interests, was made a necessary condition to some of the more important acts of the governing power. To the first of these modes of limitation, the ruling power, in most European countries, was compelled, more or less, to submit. It was not so with the second; and, to attain this, or when already in some degree possessed, to attain it more completely, became everywhere the principal object of the lovers of liberty. And so long as mankind were content to combat one enemy by another, and to be ruled by a master, on condition of being guaranteed more or less efficaciously against his tyranny, they did not carry their aspirations beyond this point.

Here, John Stuart Mill neglects another key part of the historical background for the concept of liberty. Until remarkably recently in human history, most people had some encounter with slavery–whether subjected to it, imposing it, or observing it. To those who know slavery first hand or even second hand, one of the foremost meanings of liberty is “not slavery.” And indeed, those fighting for political liberty of the sort John Stuart Mill describes above often used literal slavery as an analogy for how they would feel about a ruler who imposed his will on them too much. So any history of the idea of liberty should pay attention to slavery as well as to more overtly political activities.

In the history of the United States, one of the most poignantly troubling moments was when agreement on the Constitution–that has done so much in the end to protect all of our liberties–was secured by sacrificing the liberty of enslaved Americans for an additional several generations. And in understanding the tragedy of that decision, consider that not only was the path of immediate emancipation not taken, even paths that would have taken at least a generation to end slavery–such as ending the importation of newly enslaved human beings and declaring that the children of slaves would be free–were roads not taken.