Your onsite search engine optimization goes well beyond meta tags, page titles, and article URLs. In fact, real onsite SEO has very little to do with the (meta) content you put on the page. You need to know the difference between “onsite SEO” and “onpage SEO”. What you do for the site is different from what you do for the page.

Onsite SEO: Crawl Management

Crawl management is the most important part of onsite search engine optimization. Judging solely by the articles you find on popular blogs like Moz and Search Engine Land you could easily conclude that hardly anyone actually pays attention to crawl management. Worse, too many people avoid looking at the data that is most critical for effective crawl management.

You have to look at the raw server logs to create a crawl management strategy or to identify a crawl management problem, much less to propose a solution. Depending on anything other than a raw server log file for crawl management analysis is equivalent to gambling with your Website’s search visibility.

You can muddle your way through a crawl management crisis by looking at the reported crawl errors in search engine Webmaster accounts. This has become the standard practice. But crawl problems are not found only in 404 errors and redirect loops.

Problems you have to look for in crawl management include:

Aggressive crawler activity

Inefficient recrawling

Redirect chains

Server error responses

Bow-to-stern crawl latencies

Targeted crawler access

Aggressive crawler activity applies to situations where a crawler is bringing down your server, crawling too much content for the amount of traffic it sends to your site, or is some sort of spambot. You have to know how to identify all three types of aggressive crawler issues and you have to be able to deal with them efficiently.

Inefficient Recrawling is similar to aggressive crawl from a major search engine, but not quite the same thing. The crawler may not be hammering your server but it’s fetching the wrong page too many times. This is a classic hallmark of two navigational problems: PageRank Sculpting (always a bad idea) and Redirect Engineering. You use Redirect Engineering as a band-aid to fix wide-scale problems temporarily. If you don’t address the core problems, however, the Redirect Engineering itself becomes a problem.

Redirect Chains are the most well-documented crawl problem for the SEO community. Most people now seem to understand that putting users and crawlers through multi-hop redirect chains wastes server resources and visitor time. Sure, some people fear the loss of PageRank but if you’re focusing on PageRank when you see a redirect chain you’re not thinking like an SEO. Create an optimal search experience and stop fussing about PageRank.

Server Error Responses are powerful signals that someone or something is in the wrong place at the wrong time. In a perfect Website management world you should never see server error responses. Whether you are looking at 302s, 301s, 404s, 500s, or something else, spend some time figuring out why crawlers or people are receiving these responses. In a world afflicted with rogue crawlers the only server error codes you WANT to see in the log files are 403s, but you have to make sure they go out to rogue crawlers and not to real people.

Bow-to-Stern Crawl Latencies are among the most important (and most difficult) crawl metrics you need to manage. If you don’t even have a glimmer of an idea of how long it takes a major search engine to crawl all the URLs on your site then you’re lost in the woods and surrounded by hungry wolves. You cannot master the art of managing crawl latencies very quickly. You will be tempted to flatten the site and then to create a massive peak in the navigational architecture, but Bow-to-Stern Crawl Latency must be monitored over time. Sometimes it’s okay to leave the old content uncrawled for several months. Sometimes you need to get it crawled as quickly as possible.

When a site has been downgraded by the Panda algorithm, you’ll have to bring out every Bow-to-Stern crawl technique you can get your hands on if your site is very large.

Targeted Crawler Access has taken on new importance as rogue crawlers (including “social media monitoring” tools, “SEO link checking” tools, and various other types of scrapers) become more dominant in your Website experience. On many sites crawlers now constitute more than 2/3 of all the visitors — and some crawlers do execute Javascript so, yes, they ARE inflating your analytics code. Blocking rogue crawlers is very important, but sometimes you’ll have to create special channels for tools that you want to grab your RSS feeds.

You’ll also want to watch for and block attempts by competitors to crawl your site with tools like Screaming Frog and Xenu Linksleuth. Some people in the industry just blithely tell all their readers to go out and hit other Websites with every link crawler they can find. This creates crawl spam and it is an EXTREMELY UNETHICAL PRACTICE. You should NEVER crawl someone else’s Website without their permission, especially not if you are only looking for link opportunities.

Crawl management also entails watching for changes in crawl behavior by the search engines from within their Webmaster dashboards. The occasional “Fetch as Xbot” failure is also an important red flag. If you cannot fetch a page as the search engine that’s a serious flaw in your site’s relationship with the search engine. Don’t leave that relationship on autopilot.

Onsite SEO: Page Delivery Faults

Stop fussing over “page load time”. This is a red herring that has wasted a LOT of time over the past couple of years. Most Websites are serving their content fast enough to satisfy Google.

Page delivery faults, however, are a more serious problem. You can divide them into three categories:

Offsite delivery anomalies Misplaced onsite delivery Onsite delivery failures

Offsite Delivery Anomaly has a huge impact on your search optimization because you may be depending on a remotely hosted tool to track visitor behavior, or to deliver content for you. You need to make sure the site is delivering the content it promises and properly loading tracking tools. Any other remotely hosted widget that either affects the data you collect or the visitor experience also needs to be firing in a timely fashion.

Most of the time when I cannot get a Web page to “load” it has actually loaded in my browser but an advertising or tracking widget has locked up and the browser won’t finish rendering the page. As more remotely hosted services implement asyncronous Javascript this becomes less of an issue but you still need to watch for it.

Malware is the other big problem in Offsite Delivery Anomaly analysis. You won’t find this in the server logs; rather, you have to go through the site’s code and/or public files/directories.

Misplaced Onsite Delivery can be anything from an improperly coded widget to a poorly coded navigational link. Misplaced Onsite Delivery often manifests itself as broken link requests in the server logs but it could be hidden away and causing some other problem. For example, you may be delivering the wrong widgets on some pages. Remember that search engines have been crawling and executing Javascript for YEARS. You need to monitor what they extract from Javascript more than ever before.

Onsite Delivery Failures includes everything from missing scripts to missing images. Whatever it is, your server has failed to deliver it to the visitor. You may have a mistyped redirect in place. You may have broken links in your navigation. You may have the wrong content embedded in a Javascript widget or an iframed file. There are a lot of potential onsite delivery failures. Spend enough time digging into Website code and you’ll compile a large list of these types of problems.

Onsite SEO: Page Composition

Page composition falls into the design area although the SEO technician has to be more concerned about the “hows” and “whys” than the “whats”. In other words, once the business decision has been made to put something on the page (or to remove something) your job as the SEO is not to argue with the decision-maker; your job is to make the decision as successful as possible. Of course this may not always be possible, in which case you want to cover yourself by explaining the risks entailed by the decision. Nonetheless, you still need to propose a plan to compensate for the weaknesses of the decision and to leverage its strengths.

For an SEO page composition includes things like content delivery mechanisms (anything that may delay the publication of changes or additions to the site’s content), compartmentalization of things like navigational widgets (which often are limited in what can be included and how they fit onto the page), and the coding structures used to include content or widgets on the page.

For example, a site may be delivered by PHP, Javascript, static HTML files, Perl, and Java (among other options). A site may also be load balanced or published through a content delivery network. All of these things affect page composition because you may ask for changes that either cannot be implemented or which may be implemented in staggered layers, thus affecting your ability to monitor and measure the impacts of changes made.

Page Composition also includes the number of links on the page, the number of paragraphs of text, the number of images, the number of widgets, the number of navigational groups, etc. In other words, for search engine optimization page composition must catalog and analyze all the groups of things that go on the page. Any one group of things may have unforeseen consequences depending on content (it could be duplicated, look like it’s keyword-stuffed, or in a different language) and placement (it could be hidden) or style (it could be too similar in color or contrast to the background).

Page Composition is not so much concerned with what any one individual page looks like or contains as it is about how the pages are laid out and presented. This includes evaluating and modifying the templates that are used to present pages to the visitor, but there may be theme-level elements that all the templates incorporate into their requirements.

In addition to themes and templates you have to look at interlaced components (such as shopping carts, image galleries, contact pages, store locator tools, site search, etc.). How do all the components of the site fit together? Are they helping each other through reciprocal navigation? Do some of the components form dead end alleys? Where can the crawlers go? Where can visitors go that crawlers cannot?

Finally, you also have to look at “window dressing”. These are things like social media widgets that lead visitors to interact with other Websites, perhaps even leave the site (as advertising and cross-promotional widgets do). How does the window dressing on the site affect the presentation of pages and the site’s crawlability and indexability? People may call this stuff “supplemental content”, “external content”, etc. but it all affects how the page is composed and presented. Do search engines see it? Should they see it? And if they see it, how important should it seem to them? You should have answers to all of these questions even if your answers are trivial.

Search Engine Optimization Isn’t a Question of Links vs. Content

People who speak about SEO in terms of links and linking simply don’t understand what really goes into the optimization process. Every Website has relationships with search engines (and possibly site search tools). Optimization is all about managing those relationships in every way possible. When you just focus on publishing content or placing links you’re not optimizing for search; you’re just publishing content and placing links.

The optimization process includes measurement, monitoring, and adjustment. The SEO Method has always been the same: experiment, evaluate, and adjust. In 2014 it’s a little more complicated than it was in 2004 but the primary goals are the same. You try something, you see what happens, and if you don’t like the results you change something.

The experiments and changes can occur anywhere in the publishing process and the visitor may never realize what is going on. Good optimization improves the relationship between the Website and the search engine without compromising the relationship between the Website and the visitor. Contrary to what many people have come to believe, there are indeed many things that you MUST do for the search engines in order to make their experiences on the Website valuable and informative.

You literally have to speak to the search engines in a different technical language from that which you use to speak to visitors. There are many overlapping elements in these two conversations and you must respect the search engines’ desire to see what visitors see. But search engines are not people and they interact with Websites in very, very different ways from how a typical human being does.