MXNet Makes Us Faster And Stronger!

Why MXNet and Gluon?

Recently, our organization(@Machine Learning Cell, SK Telecom) started actively using MXNet and Gluon. As part of the DevOps organization, we need not only to develop new models constantly but also to maintain and deploy them seamlessly, even in a disruptive environment.

There are many deep learning frameworks these days: MXNet, TensorFlow, PyTorch, Theano, to name a few. But, from the “Ops” point of view, maintaining many models for all different frameworks can be challenging. Therefore, we need to choose a few deep learning frameworks by considering the following criteria:

Convenience of code design

High-quality reference code samples

Speed of training and inference

How many functions/methods are offered out of the box

Version update compatibility

Supporting multiple programming languages

TensorFlow has many high-quality reference code and tutorials. However, it is hard to debug because the imperative environment (TensorFlow’s eager execution) like PyTorch is still experimental. In contrast, PyTorch is good for research, but presents some issues in maintainability due to the frequent API breaking changes.

Fortunately, MXNet offers both low level-control and high-level APIs such as Gluon. MXNet and Gluon enable us to quickly test, operate and manage experimental models. In particular, Gluon offers more functions via gluon-cv (Computer Vision) and gluon-nlp (Natural Language Processing). The MXNet community is committed to respect semantic versioning, which means that no API breaking change are introduced between minor version updates of MXNet.

Deep Learning Frameworks (source: https://aws.amazon.com/ko/machine-learning/amis/)

How to evaluate a deep learning framework?

The Importance of Benchmark

Scenario 1. How do I know it is a good choice to migrate from Tensorflow to MXNet?

It is often difficult to compare results between frameworks because model performance metrics such as accuracy or speed, are affected not only by the framework but also by many other parameters. As a result, Keras-MXNet is a good choice for standardizing comparisons because we can test the same model definition with different frameworks, simply by changing the keras backend. We can install keras-mxnet easily using pip.

Scenario 2. How do I know if it is a good choice to update from MXNet 1.1.0 to MXNet 1.2.0?

When it comes to comparing different versions of frameworks, it is relatively easy to perform benchmark tests if the framework is backwards compatible and if the benchmark code is already prepared. The most important point in this case to get well-defined benchmark test metrics.

Benchmark Configuration

How do you configure the benchmark on-premise? We recommend to list all the different environments, frameworks, models and metrics that are relevant to your use-case(s). For example: