A recent study explains why birds fly in a “V” formation: probably to save energy. Each bird in the formation takes advantage of the slight air currents its leading neighbor makes to increase soaring time and decrease flapping time. They are using a principle known as drafting, which is used by competitive cyclists to share the task of breaking air resistance. A group can move forward faster by allowing most members to shelter or “draft” behind one or more others. Bird flocks probably change direction in seemingly random fashion to share the burden of breaking air resistance (as well as in response to changes in air flow around them).

As I wrote a few years ago in “Swarm Theory and Web Communities” recent research shows that “swarms follow leaders who take risks”. The risk-takers most likely to be followed are those who find a better solution for a problem. The problem might be that a predator is chasing the group; or the problem might be that the group suddenly encounters a headwind; or the problem might be that a group’s immediate food supply becomes exhausted. In each of these situations the well-being of the group is threatened and the group seeks (through consensus) to find a way to defeat or escape the threat.

If the threat is immediate enough to destroy an individual in the group (a selective threat) then the group may form multiple strategies for individual survival. Typically, a group of prey will scatter in as many directions as possible. This has the effect of confusing the predator and increases chances of survival for most if not all members of the group (other factors affect those chances of survival). A group of prey, if cornered or angered, may also turn upon the predator and challenge it. This has the effect of shifting at least some of the burden of survival to the predator.

Swarm Behavior Seeks an Equilibrium Point

It has become evident through research that groups work together for mutual benefit, thus allowing individual members of the groups to prosper and work toward their own benefit. Thus, there are two levels of competition within every group: the group level of competition ensures the survival of the group; the individual level of competition ensures the survival of the individual within the group. In some animal cultures groups may expel members, changing the dynamics within the groups while preserving the groups’ abilities to compete and survive.

This is essentially the theory put forth by John Forbes Nash, Jr. that is most commonly known as the “Nash Equilibrium”, where all the members of the group make the best possible decisions with full knowledge of what the other members of the group are doing. See the movie “A Beautiful Mind” starring Russell Crowe if you want an elegant illustration of the principle, where Nash imagines what happens if he and his friends all try to dance with a blond-haired girl versus their ignoring her to dance with her friends.

A swarm, in order to function, must be stable in a stable environment. Viral Propaganda Theory tells us that a group becomes unstable if two opposing points of view divide the members of the group in about equal numbers (or strength). In “Using Swarm Theory to Balance Viral Propaganda Theory” I proposed that “The search engine optimization community has … on numerous occasions formed the wrong collective opinion about many things ….” Some examples of these bad choices include:

PageRank Hoarding PageRank Sculpting Keyword Matrix-driven Content Production Scalable Link Placement via Blog Networks Use of targeted link anchor text Use of Flat Site Architecture

I think a strong argument can be made that — in every one of these schemes put forth by SEO pundits and widely adopted by the SEO community and the businesses who followed their advice blindly — individual needs were emphasized over “group” needs. The “individuals” are collections of Web documents. The “groups” are groups of collections of Web documents.

Think of your content portfolio as a corpus of documents that comprise an individual (a “collection” as described above). The individual’s purpose is to advance your personal goals for success. The individual’s component documents may be published on a single Website or multiple Websites. If the individual (document collection) behaves selfishly, with no consideration for other individuals (document collections) around it, the individual is taking a risk beyond normal behavior.

The group may be a vertical (topic) or a small group of similar/related verticals. The group must normally function in some sort of an equilibrium, where all the members contribute value to the overall value of the group. In this model the value is whatever value a searcher places on the group. For example, given 1,000 documents about credit cards you’ll find more information collectively in the entire group than in any individual document. But search engines have to order the documents for presentation to the searcher and the searcher must read as many of the documents as are necessary to obtain adequate information.

When an individual (document collection) changes its normative behavior within the group (all related document collections) the group must re-establish its equilibrium. The search algorithms attempt to find the equilibrium point. No matter how biased the algorithms may be, they nonetheless seek an equilibrium point.

Breaking the Equilibrium Forces a Recalibration

To adapt to the new situation the group must make new choices, whether to follow the risk-taker or to ignore it. On the Web we primarily express these choices through Links and Citations. Hence, the search algorithms must incorporate any changes in links and citations into their presentations. These incorporated changes occur over time (not all at once) due to crawl priorities and possibly also due to search engine-internal issues (some documents may require more processing than others).

A search result for a query represents a calibrated assessment of the documents with respect to the query. It may not be the best possible calibration but it establishes a baseline for future similar queries. A search engine may ignore that calibration or change it for any number of reasons. We can say that search results display a tendency to blend calibrations toward their own equilibrium point. In simple terms, we can say that “average rankings tend to remain about the same given the same data to work with”.

Any change, no matter how small, has the potential to substantially alter a search result. Clearly not every change does this but to realize or fulfill that potential the change needs a catalytic influence (such as PageRank or trust).

In Search Engine Optimization You Change Links or Content

Changing links in a document corpus may force a recalibration of PageRank distribution.

Changing text in a document corpus may force a recalibration in idiomatic measurement.

If a link includes anchor text then changing the link’s anchor text may force a recalibration in idiomatic measurement.

Idiomatic measurement loosely describes everything a search engine does with the words in a document: determining their frequency of use within the document, within the document collection, and within the entire corpus of the search index; determining their structure, grammatical use, and possible relevance to other words; assigning contexts, etc.

The content around the link may include or consist of other links. Changing one or more adjacent or nearby links can therefore change the context of the link. A single link on a page may be assessed differently from a similar link (pointing to the same destination with the same anchor text) included on a page filled with many links simply because the presence of the other links creates a different context.

The page on which a link is placed is the Originating Context. There is a Receiving Context as well. It is right to ask if a search engine measures the relevance of the link to the receiving context. In some search engines the link may pass anchor text to the destination, thereby fundamentally changing the composition of the destination document as the search engine defines it. Hence, in such search engines all links are relevant to their destinations. There is an inherent problem in that kind of system, though: any group of documents can point manipulative links at an otherwise unlinkworthy destination and in practice that is exactly what has happened on a wide scale.

The existence of deceptive links forces search engines to evaluate the links in terms of contextual relationships, thus measuring the Originating Context and the Receiving Context may have become very important. An out-of-context link may be a sign that something is amiss. However…

What Happens When Many Link Contexts Are Changed?

What if a document about, say, horse racing, is suddenly singled out by a large number of other documents as humorous? The other documents could all link with variations of “this is the funniest thing I ever heard”. How is the search engine to assess the new influx of links?

It stands to reason that if the Originating Contexts all pass current filtration tests that their anchor text will be passed to the horse racing document. In an honest relationship that should be the end of the issue. But what if all the new links are deceptive? The search engine may have to alter its filtration process to identify and adapt to a new manipulative technique.

To change the filtration process the search engine would have to assess the relationships between the new links and their originating contexts; furthermore it would have to determine if the corpus of originating contexts share some patterns of similarities or dissimilarities. For example, do the links all fit naturally into the originating contexts or do they disrupt the expected behavior of those contexts?

The originating contexts themselves must be assessed to determine if they conform to expectations for manipulative content or if they are somehow discordant with expectations for natural content. In such an assessment a swarm of manipulative contexts would break away from the main corpus and create its own identity. That identity may be distinct enough for a search engine to measure and respond to.

But what if a new body of contexts appears naturally, discordant from the corpus and yet appropriately framing the new links? In this case the corpus of documents must be assessed on how it behaves. For example, are the documents all too generous toward the shared destination? Do the documents create a distinctive value for themselves as a collection (such as a clump of news articles about a breaking event)? Do the documents behave as other documents published by their hosts? Do the hosts behave as other hosts that the search engines trust?

By analyzing the behavior of groups of documents search algorithms should be able to detect sudden changes in direction, even if only in retrospect, given a large enough sample size of documents that depart from their hosts’ established norms. The swarm of documents will behave like other swarms of documents; the swarm of links will behave like other swarms of links.

Doing What Others Do is Both Safe and Risky

By following the examples set by others when we create content and place new links in that content we are following the swarm. A swarm that has existed for a long time has probably found a safe route or environment and it should continue surviving. But every now and then a smaller swarm breaks off from the main swarm, following a new (risk-taking) leader. The smaller swarm may eventually grow and thrive but within the context of all other swarms it may draw undue attention to itself.

A search engine acts like a predator chasing the swarms. The risk-taking swarm may be the object of the search engine’s attention. In a natural biological context swarms can change direction and break up to avoid a predator’s approach; in the searchable Web ecosystem, however, the swarm cannot react. It continues to do what it does blind to the approach of the predator until the search engine strikes.

For the search engine, then, deindexing manipulative links that conform to a newly identified pattern is equivalent to shooting fish in a barrel. Once the swarm’s unique characteristics have been identified the swarm is essentially doomed.

In order to survive such a calibration event a link must change throughout all of its contexts in order to escape the swarm. Leaving the swarm before the swarm has been swallowed up (deindexed) is the individual link’s only optimal strategy for survival.

See also: