Continuing the report from my sojourn among some of the most capable programmers in Silicon Valley…

“You look like someone who might know awk,” said a top software engineer at one of the biggest web companies, where a 12 petabyte data set is a common starting point for analysis. “I think that was a polite way of saying that I had gray hair,” she continued. “Big data is ‘batch processing’ or ‘stuff that requires more work than can be done interactively.’ I’ve seen young programmers for whom big data is their first encounter with batch processing. It is like watching a dog eat peanut butter. It takes them a while to learn that machine learning is simply dividing things into bins and then clustering.” What are her secrets? “Transform everything into tab-delimited text files and use standard Unix tools. This results in much faster run times than code using the shiny new tools and data structures. My favorite algorithm is the gradient boosted decision tree mostly because I like to hear people trying to say it without getting tongue-tied.”

An artificial intelligence specialist said that, from her perspective, machine learning is fundamentally changing programming: “hundreds of lines of code to drive nVidia CUDAs instead of millions of lines of code. AI is both freedom from programming and freedom from understanding. Google is replacing PageRank with RankBrain and when this is complete they won’t know why certain pages are offered as the best results.” In her view a “future-proof skill set does not involve much coding background; it will be more about statistics and domain knowledge.” Some tools to learn? The Torch library and lua.

Is Deep Learning all hype? Perhaps not. The recognition rate on standard image sets has gone up dramatically recently. “There are no new ideas,” said one hardware/software expert. “The guys in the 60s were pretty smart. They just didn’t have fast enough hardware.” Could it be that the Singularity is in fact within sight? The best market-based evidence for this seems to be that people who teach at Singularity University are being offered $20,000 speaking engagements at corporate events.