Google feeds its search engine's index with site data from a virtual army of "bots"—Web-crawling applications that scour sites for content. But in the past, Google's bots hit a wall when they ran into interactive content that was loaded through JavaScript—especially on pages that use Asynchronous JavaScript and XML (AJAX) to allow users access to additional content without reloading pages. But now, according to Vancouver-based developer Alex Pankratov, it appears Google's bots have been trained to act more like humans to mine interactive site content, running the JavaScript on pages they crawl to see what gets coughed up.

Google has in the past offered up proposals to make AJAX content more searchable, but this put the burden on Web developers rather than on Google's bots—and the proposals didn't gain as much traction as Google had hoped. During the last quarter of 2011, Google finally started to figure out how to efficiently solve the problem from its end, and began to roll out bots that could explore the dynamic content of pages in a limited fashion—crawling through the JavaScript within a page and finding URLs within them to add to the crawl. This required Google to allow its crawlers to send POST requests to websites in some cases, depending on how the JavaScript code was written, rather than the GET request usually used to fetch content. As a result, Google was able to start indexing Facebook comments, for example, as well as other "dynamic comment" systems.

Now, based on the logs Pankratov has shown, it appears that rather than just mining for URLs within scripts, the bots are crawling even deeper than comments, processing JavaScript functions in a way that mimics how they run when users click on the objects that activate them. That would give Google search even better access to the "deep Web"—content hidden in databases and other sources that generally hasn't been indexable before.