Mapping Challenge winning solution

Learn how to detect buildings on the satellite imagery with crazy precision. DIY with open source implementation. Brought to you by neptune.ml that, together with deepsense.ai, won the challenge and received top community contributor award 🏆.

No cherry picked predictions. 1st column is input image; 2nd column is model’s prediction; 3rd column is model’s prediction superimposed on the input image; 4th column is ground truth superimposed on the input image. Each row is one example.

Prologue

Deep learning on satellite imagery is increasingly popular topic among data scientists. Satellite data exhibits number of possible applications in both business and research domains. Use cases are ranging from classification and localization, to semantic or instance segmentation. Last one — instance segmentation — is all about detecting an object class on the image, then segmenting each instance of that object (think: select all cats on the image).

This post is about the winning solution to the instance segmentation challenge, called the Mapping Challenge 🌐, organized under the auspices of crowdAI and Humanity & Inclusion. Here, I will share our process, way of working, solution internals and final results. Few words about the tech stack will be said as well 😉

Project dashboard — we were rather busy with our experiments :)

Why it matters?

Satellite images deliver high quality overview of the Earth surface. Such data can be used for public profit. One example is research project: “Combining satellite imagery and machine learning to predict poverty”. Researchers from Stanford University and National Bureau of Economic Research (US) use deep learning approaches (specifically transfer learning technique and VGG network architecture) to estimate household consumption and assets in some countries from the developing world.

Another example is supporting response to humanitarian crisis or natural disasters. How can Mapping Challenge contribute to this goal?

We are in a period of increasing humanitarian crises, both in scale and number (…). Often, accurate maps either do not exist or are outdated by disaster or conflict.*

Despite their importance, new maps are drawn by hand, so there is urgent need for automated methods that can help generate accurate real-time maps. Therefore, significance of the competition may be summarized very briefly:

Many parts of the world have not been mapped; especially (…) those most vulnerable to natural hazards. Obtaining maps of these potential crisis areas greatly improves the response of emergency (…) actors.*

The practical goal of this challenge that is approachable by data scientist is to:

(…) explore how Machine Learning can help pave the way for automated analysis of satellite imagery to generate (…) maps.*

(*all quotes retrieved on 2018–09–12 from competition page)

Few words about the competition

Example input images

If you want to participate in machine learning challenges organized around open data simply go to crowdAI.org and pick your favorite. Some time ago, in early April 2018, we were attracted by the Mapping Challenge. We are specialized in the image-based machine learning and prior to this one we participated in the Data Science Bowl 2018, where we worked mostly with U-Net and Mask R-CNN architectures. During DSB’18 we developed clean implementation of the U-Net that is highly parametrizable. This competition is also about segmentation, however here, we work with satellite imagery, instead of microscope images. We have decided to participate in Mapping Challenge because of several reasons:

Explore new platform for machine learning competitions. We all know Kaggle well but crowdAI has different profile.

Contribute our piece to the broader public profit (see section: Why it matters?).

Further improve U-Net implementation with custom encoders, multiple outputs and other modifications.

Extend our portfolio of open data science projects.

The Team — Data Science Quartet

Data Science Quartet participated in the competition, that is: Jakub Czakon, Kamil Kaczmarek, Andrzej Pyskir, Piotr Tarasiewicz (alphabetical order). Since it was team work entire time, thus — for simplicity — I will refer further achievements to the team as a whole, not to any particular person. Here, at neptune.ml we strongly believe that this success would not be possible without strong team cooperation. Our team of 4, just like the Gang of Four (GoF — these four authors are famous for their Design Patterns book), consisted of enthusiasts who are passionate about machine learning. We come from two companies: Jakub and Kamil are from neptune.ml, Andrzej and Piotr are from deepsense.ai — machine learning and AI consultancy company.

How did we work? How did we design the process?

We had regular weekly planning sessions (to report progress and discuss future directions), daily stand-ups (to let everybody know about latest work), slack-like communication channels for ad hoc conversations and Pull Requests for comments directly in code (and code review of course 😄).

Prior to entering the challenge we had some principles, that we are eager to realize in any project that we pursue.

clean code — we think that single competition is just one step towards something bigger. Hence, we are not tempted to sacrifice engineering quality for immediate gains on the competition leader-board. Well designed implementations with nice interfaces will pay off later in future competitions and commercial data science projects.

️team work — data science projects are — by definition — team work. Smooth progress with constant iteration of new ideas (some of them worked well, some not so much) was one of the crucial factors that gave us 1st place in this competition. Anybody can propose the idea, pick task that she likes most, and think about their contributions to the project. This open environment was critical for our success.

fast experimentation — thanks to the clean solution development process we were able to iterate over ideas smoothly. Beginning with day 1, all experiments were run on neptune.ml, our platform for data science projects. Single place for all your artifacts related to the experiment: results, outputs, saved models, code, charts, hyperparameters — was very convenient and created common place to compare results and look at each others work. It became handy, because number of experiments quickly crossed 1 hundred (to date we ran more that 1000 experiments, all publicly available 😎).

Single experiment with three interactive charts displayed.

public sharing — neptune.ml team has the vision to open and show the entire process related to data science projects. That is why code base was open source from day 1. Everybody could observe how it grows, when new models are developed and how we build routines to efficiently handle competition data. It was quite a challenge because of number of images: 280741 in train, 60317 in validation and 60697 in test sets. Moreover, everybody could observe our experiment running live on neptune, browse charts, results. Finally, we let everybody reproduce our results, because we shared complete set of information and guidance how to do it. We named our idea Open Solution.

Practical advice for working on data science projects

Here, we share what worked well from team perspective and what we continue to do in current challenges (for insight regarding deep learning check “Winning solution description” section):

Pick single platform for all experiments’ artifacts (saved models, code, results, hyperparameters). This saves you great deal of time when things are getting complicated (and they will surely get 😉).

neptune.ml dashboard, where we keep all experiments. Notice competition metrics, Precision and Recall in the middle (two green columns).

Schedule weekly planning session to report progress and discuss two topics: what next (actions for next 7 days) and ideas. Our deepsense.ai team members came with many tricks regarding loss function and architecture design and this was proper venue to discuss them.

to report progress and discuss two topics: what next (actions for next 7 days) and ideas. Our deepsense.ai team members came with many tricks regarding loss function and architecture design and this was proper venue to discuss them. Schedule fixed time — 15 minutes — for daily sync . This will let everybody know what is being processed right now.

. This will let everybody know what is being processed right now. Put all tasks and problems in single place like GitHub issues and boards or trello boards and cards. This should be curated by team leader and maintained by everybody in the team.

like GitHub issues and boards or trello boards and cards. This should be curated by team leader and maintained by everybody in the team. Every team member work on git branch. Once finished her task, she publish Pull Request to the release development branch. Once a incremental solution is finished release development branch is being merged to master. Never touch master, unless it is PR from release development branch. This ensures that you have incremental solutions to your problem on master. It also allows to you to keep track of the project history.

Enforce peer code review. Team gets educated about software engineering in general and about each other’s style of work in particular.

Tech stack

Hardware

At the level of hardware, we develop code on regular laptops and train models either in the cloud (neptune.ml handle communication with Google Cloud Platform. User can select her favorite GCP machine with either Nvidia K80 or P100 GPU) or locally on Nvidia GeForce GTX 1070 GPUs. In the later case, we use neptune.ml to monitor experiments that we compute locally (see section “Neptune.ml” below for more details).

Software

We work with Python 3.5 and this is our most basic block of the tech stack. We can distinguish between few logical layers, that will be described in the following section.

Python Packages (some are project-specific)

At the top, we have number of Python Packages that we use daily to simplify common tasks in data science. Below, most important are presented, together with few words of justification why them matters 😉.

PyTorch (v0.3.1). We found it easy and intuitive to develop deep models in PyTorch. We are aware of numerous blog posts and conference talks that compare Keras and PyTorch. We pick PyTorch because user can develop custom architectures intuitively.

ImgAug — for images augmentations, because it offers simple API and a lot of augmentations. Take a look at our use of ImgAug: augmentation.py.

Specifically in this challenge we worked with pycocotools — to compute competition’s metric.

scikit_image, opencv_python and imageio for operations on images.

(all requirements are on our repository in the file requirements.txt)

Steppy and steppy-toolkit

Final solution pipeline developed using steppy.

Our team participated in competitions and commercial projects before winning Mapping Challenge. Back then it became clear that there is a need for some simple and clean interface for building complex machine learning pipelines. Steppy is lightweight, Python library that enables clean machine learning pipeline design. It is complementary to the neptune platform, but is 100% independent technically. Steppy-toolkit in turn is curated set of transformers that makes your work with steppy faster and more effective.

Neptune.ml

It is community-friendly platform supporting data scientists in creating and sharing machine learning models. Neptune facilitates teamwork, infrastructure management, models comparison and reproducibility. We ran all our experiments on neptune.ml to ensure that we can reproduce them and compare later in time. Neptune is independent from all layers mentioned above and it just works with arbitrary Python code.