Washington is not a place to live in. The rents are high, the food is bad, the dust is disgusting and the morals are deplorable. Go West, young man, go West and grow up with the country

— Horace Greeley

In the winter of 1848, a carpenter named James Marshall found shiny flecks of metal in the American River. He identified this as gold and the discovery changed the history of California forever. An estimated 300,000 men migrated with the hope of finding the same shiny fleck that Marshall found. Some lucky ones did. Gold worth 10’s of billions of modern day dollars was found. San Francisco’s population exploded. The secondary effects of the boom was just as important. Newer tools were invented to mine gold faster. Fundamental infrastructure change followed with a railroad constructed between the east coast and California.

Modern day California has had it’s share of Gold rush - semiconductors, software, internet and the most recent of these - Big Data. With the proliferation of Internet/mobile usage, availability of data exploded.

To the untrained eye, the data may not be useful. With the right skills, however, the data can be turned into actionable intelligence -- in other words Gold.

The modern-day art of converting data into Gold is aptly referred to as “Data mining." Just like their 19th century counterparts these miners (often referred to as “Data Scientists” or “Big Data engineer”) sift through huge amount of useless material to find the one fleck of gold that matters.

This modern day gold rush also saw tremendous progress in tools and techniques used to mine data. The fundamental piece of infrastructure needed for pretty much any large scale data processing is a cluster of machines. This would imply setting a distributed system, managing the load, paying for the systems administrators etc.

Naturally, it posed a big barrier for startups.

Amazon democratized big data with the creation of AWS bringing down the barrier. AWS is the railroad for the modern gold rush. It paved the way for startups like Instagram to scale rapidly without worrying too much about data infrastructure.

While it’s natural to draw parallels, the miners seeking different types of wealth in different era divulged on critical aspect -- scale.

Here's an easy way to understand how scale plays into this. Why does LeBron James make so much more money than a cardiologist? The latter saves lives whereas the former merely “entertains”. An objective analysis would point to a basic economic reality -- James in his “job” can service 100’s of millions at the same time, a cardiologist cannot. In other words James scales.

"Scale" is generally used in the Valley to refer to businesses that have the potential to expand massively. Scale is the reason why Airbnb gets funded and a plan to build a single new hotel does not.

There is nothing stopping from extending this concept to professionals. A general rule of thumb is that if you have to work twice as hard to make twice, you don’t scale.

The availability of AWS and related software tools for big data (like Hadoop) implies that engineers can concentrate on their applications. To put it bluntly, an engineer’s productivity just skyrocketed because a whole lot of s**t has been taken care of. A gold miner can only sift through a few pieces of gravel at the time, a big data engineer can push code and expect to see his/her changes make a difference to 100’s of millions of users.

Big data engineer is now LeBron James.

Scalability as a profession is no longer restricted to singers and sports stars. Where do you think the term “rockstar engineer” came from?

Artificial Intelligence (AI) had been around for decades and the fundamental theory behind neural networks was formulated somewhere in the 80s. Without data of the right kind and quantity, neural networks remained the domain of a handful of researchers. With data proliferation and need for complex tasks like image recognition, AI found it’s sweet spot.

Some practical applications include matching faces almost as well as humans (Facebook), IBM’s work on identifying correct treatment plans with the supercomputer Watson (same group which made Jeopardy winning supercomputer), Google/Uber/Audi’s ongoing efforts on self driving cars etc.

It’s no surprise that prominent AI researchers are now in the industry -- Geoffrey Hinton at Google, Andrew Ng at Baidu and Yan LeCun at Facebook. If AI was bread mold, Big Data is penicillin. Big Data made AI relevant. AI made some very complex Big Data problems tractable.

In the Spring of 2012, Facebook announced it’s acquisition of Instagram. Number of users: 26 million; number of employees - 13. Press floated around words like “existential threat." Thirteen engineers had cooked up an application, made it viral and posed an existential threat to the largest social network in the world.

Just when you think this is as crazy as it can get, 2-1/2 later Whatsapp is acquired by Facebook. Number of employees: around 55; Number of users: around 400 million.

Again, around 50 employees had cooked up a system which hit the very foundations of communication.

Here’s the big takeaway: You don’t have to be an early engineer at Whatsapp to make this level of difference. Plenty of jobs within Google/Facebook/LinkedIn/Twitter have the similar potential.

If you are an engineer with the right skills and are not working in Big Data, what are you waiting for? This is your chance to shape your destiny and make impact at scale never seen before (at least for an engineer).

Go Big, Go Data and Go Scale.

[Thanks to John Abell and Erran Berger for helping me with this post. It has come a long way from it's first version]