1. Crawling & Indexing

Not even millions or billions, but the search on Google happens in trillions! A query starts earlier than something is searched, where the process of crawling and indexing the hundreds of thousands of documents continues.

Let’s get the facts straight…

Google now processes over 40,000 search queries every second on average, which translates to more than 3.5 billion searches per day and 1.2 trillion searches per year worldwide.

How It Works…

Well well… these are the two founding processes that make it possible to collect and arrange all the info on the world wide web in order to bring only the most meaningful and relevant results matching your search query. Google’s index range is more than 100 million gigabyte and the credit goes to the 1 million hours spent in computing to build it.

The process is simple. The search giant first finds the information with the help of crawling and then organizing is done by indexing them. Check out the detailed info below.

Find Info With The Help Of Crawling

Also known as web crawlers, these are used to find publicly available pages. This software go through the web pages, follow the links on these pages, and ultimately fetch data back to Google servers.

For this purpose, it uses past crawls and sitemaps. Attention is paid to the new websites or modifications in existing sites as well as dead links. Everything from listing the sites to crawl, how often to crawl them, and what numbers of pages to fetch out of each website. Note that Google NEVER accepts any kind of payment for crawling a particular site more than others.

Organize This Info With Indexing

After the info is crawled from different web pages across the web, now comes the turn for organizing this fetched data. The web works more like a public library where there’s no filing system but the continuous addition of new books. During crawling, Google collects web pages and on the basis of their relevancy, it creates an index. After this, you come to see the search results when you enter a query.

From a heading to a single word written on a particular page, Google keeps the record of every info from the most basic level and then with the help of algorithms, it finds the results according to your search query.

That’s where the whole process becomes all the more complex. There would be thousands of pages with the same name yet Google won’t show you that many when you type it, for example, Apple. You might be looking for anything on Apple like images, videos, or info. Here Google algorithms work differently and using its Knowledge Graph, it surpasses keyword matching to bring better results from the pages that may be useful to you.