I’ll talk about it more generally, and then I’ll talk about the Etsy-specific case. More generally, the kind of frameworks that most people are accustomed to using are actually written in Python… But the lie about Python is that none of the fast bits are actually written in Python. It’s all indexing into C and C++, or Cython, or in some cases Fortran. And what Python really becomes is this kind of domain-specific language for gluing these together.

[ ] The ones that are probably the most familiar to everyone on the show are scikit-learn, Tensor Flow, PyTorch, Light GBM, XGBoost… All of those have the core performance pieces written in C and C++, and they also aren’t immune to these problems. You can actually look and find TensorFlow has had to release patches, because they also, by nature of being in C and C++, have these problems with safety and reliability.

The place where Rust tends to have the biggest benefit is by replacing those back-end components with a safer, faster language… And we’ve had a lot of work recently done in this space to make it a little bit easier to integrate with Python. There’s a project called PyO3 out there which makes it much simpler to interface between the back-end and the front-end. That applies to Etsy.

In the learning to rank space we have to do a lot of feature engineering. The state of the art is still gradient boosting models for the most part, and that means that a lot of the benefits you get from neural nets, that [unintelligible ] being deferred to the algorithm - you have to do it manually. And we were running into this case where every night we were training hundreds of millions or billions of records, and we were trying to [unintelligible ] through a whole bunch of different features, and it was taking an exorbitant amount of time.

The second piece that’s really challenging in the search space is that the machine learning algorithms are traditionally written in Python, but Solr and Elastic and these kind of [unintelligible ] systems are actually written in languages like Java. And what we really didn’t wanna have to do was write feature engineering twice, so do something like the hashing trick in Python, and then have to port that same implementation over to something like Java, to get models trained in Python and then deployed on Java.

So we were really looking for a language which would allow us to embed it in both Python and Java at the same time, and that kind of put some restrictions.

You mentioned that you both are gophers… One of the problems with Go is that it actually had managed memory, and Java and Python don’t necessarily work particularly well with managed memory, while managing their own memory. So those types of constraints made it hone down the number of opportunities we had, and we were mostly focused on trying to find one where we felt we could be productive quickly, but at the same time didn’t have t pay a performance penalty.