Recently I have had several conversations with people I know in the data science space. The conversations are always about business and then tend to drift into the state of data science as a whole. One theme has constantly come up in these conversations which I think needs to be addressed. The topic that always comes up is, there are a lot of people currently running data science teams in large organizations and the vast majority (we are talking 80-90%) want to leave their current job. Why is that? With smaller organizations, the number isn’t as great. What is going on in the larger organizations that is causing such a mass exodus?

Having worked in and with many large organizations in a data science leadership capacity, I have my own theories. These are based on experience and observation.

Academia can’t do For-Profit

Many large companies have fallen into the trap that you need a PhD to do data science, you don’t. The top 5 data scientists I have ever worked with, only 1 had a PhD and it wasn’t even in stats or data science, he was a bio physicist. I call this the academia trap because many companies believe they need that stats or data science PhD. There are some smart people who know a lot about a very narrow field, but data science is a very broad discipline. When these PhD’s are put in charge, they quickly find they are out of their league. They were never taught how to run a P&L, manage a team, deal with people, competitive intel, market assessments, building a business case, etc… They were taught numbers and a few tools.

Add to this, the world they come from. Many peer review papers in the academic world that are really good, don’t see the light of day. Why? The reviewers may have a competing theory and don’t want their current established ideas to get superseded. It is shocking how often this happens in the data science space. I always found the academic world more political than the corporate world and when your drive is profits and customer satisfaction, that academic mindset is more of a liability than an asset. Not to mention, I have yet to see a data science program I would personally endorse. It’s run by people who have never done the job of data science outside of a lab. That’s not what you want for your company.

What often happens is that budgets grow and results drop. I have had some of these types just demand more and more from leadership and leadership doesn’t have an idea of what is real or not. Then they find out they are spending $10 million plus a year and getting very little economic return for it. I have never spent that kind of money and have built billion-dollar data products. I’m also not a PhD. Those that make the profits, not a single one I know has a PhD. PhD’s do great work but they often need a lot of help to get to leadership levels and be good at it.

Wrong Expectations

Doing data science and managing data science are not the same. Just like being an engineer and a product manager are not the same. There is a lot of overlap but overlap does not equal sameness. Sometimes, I envy the data scientist who just has to crunch the numbers. Being a data scientist without management duties is pretty easy. Most of the time it is cleaning data sets, testing algorithms and researching new methods. A pretty comfy job. Now compare that to someone who needs to run the practice.

A leader of data science practices needs to focus on: data governance, MDM, compliance, legal issues around the use of algorithms created and ensuring documentation just in case someone sues for wrongful use. There are hiring issues, staffing problems to deal with, budget and funding to gain, P&L to run, business cases to build, market research to conduct, vendor meetings to hold, tech life-cycle management to run, evangelizing of projects (both internal and external) and turning the work into data products that are sell-able by the company and all this while trying to ensure a profit for the company. Big difference.

Most data scientists are just not ready to lead the teams. This is why the failure rate of data science teams is over 90% right now. Often companies put a strong technical person in charge when they really need a strong business person in charge. I call it a data strategist. A data strategist needs to be the person you hire. Right under that data strategist is the strong technical person. And that needs to be a very solid and strong interpersonal relationship between the two. If they are competing, it will cause friction and less than desired results.

Bad Methods

Agile has taken the tech world by storm. It works fairly well for software development and as a result, many companies enforce it on data science. But data science is not software development, it’s really a field of discover whereas software development is about assembly. I have worked with companies that demand agile and scrum for data science and then see half their team walk in less than a year. You can’t tell a team they will solve a problem in two sprints. If they don’t’ have the data or tools it won’t happen.

Data science is a discipline that requires its own methods. Add to that, most companies are still treating data products like they do physical products, the economics are not the same. When I build a recommendation engine, my costs per unit is pretty much zero. Unlike a physical product which has a per unit cost. I can make a million product recommendations with that engine or just one. Other than the electricity, my cost per recommendation is the same. The cost to do the same in the physical world would be a lot more with very different cost variables involved. We have to understand that the economics of data products is different. A lot of large companies don’t even have this conversation. Finance is used to physical products economics which can cause frustration for those running data products as they are having one hand tied behind their back.

These are just a few reasons. I am sure there are more and would love to hear what you think. Feel free to comment below.