Photo by Artak Petrosyan on Unsplash

Introduction

I am fully aware that this could be considered an exaggeration but treating generative adversarial networks as a small step in creating a computer with a sense of creativity really stimulates my own imagination. This especially applies to the conditional flavor of GANs. These, in simple words, offer the possibility of creating a plausible real outcome, an image, by following a set of predefined rules. Those rules can be constructed in a form of a mask image in which different color areas denote distinct classes. Conditional GANs can interpret these constraints to produce an image that not only meets those criteria but also is realistic.

The goal of a cGAN is to produce realistic images using masks that define the conditions. In this case left picture presents a mask that contains rules regarding land cover i.e. where buildings, fields or streets have to be located. The main task is to achieve a result similar to this on the right by using the mask as an input.

Recently this has helped us tackle a problem from the remote sensing and geoinformatics domain. The story so far is that we’ve started preparing ourselves to write a scientific paper on the modifiable areal unit problem or MAUP. tl;dr The effectiveness of spatial analysis directly depends on how the data is aggregated. In remote sensing the issue revolves around the resolution of an image. To be precise, the main concern is how large the area covered by a single pixel is e.g a satellite sensor can produce images in spatial resolution of 10 m (four visible and near-infrared bands), 20 m (six red edge and shortwave infrared bands) and 60 m (three atmospheric correction bands). This means that a single pixel represents an area of 100 square meters for rgb + nir channels. Without going into details, we have a plan on how to deal with some of the problems associated with MAUP. However, for this to be possible we need a very large data set. You can argue that the Earth offers quite a vast amount of diverse land samples and satellite data sources are also publicly available but the problem is more complex. We care not only about the volume, but also we have to be sure that each of acquired samples meets strict conditions regarding cloud coverage, season and land cover. In fact, selecting a huge number of samples with specific parameters is a time consuming process. Therefore, we decided to use a GAN to augment our dataset.

Data preparation

Sentinel satellite images and land cover data from the CORINE Land Cover database and Data Base of Topographic Objects were used in the study. The Sentinel satellite images were downloaded from Copernicus Open Access Hub. Sentinel 2 and 3 images with a maximum cloud coverage of 5% were selected. Non-clouded images were preferred. The main source of land cover data was CORINE Land Cover (CLC) sets from 2012 and 2018, provided by Copernicus Land Monitoring Services. The third level of accuracy of CLC data types corresponding to the map on a scale of 1:100 000 with a minimum area of land coverage patch of 25 ha were used. The second, auxiliary source of land cover data was The Data Base of Topographic Objects valid for 2015, obtained from Head Office of Geodesy and Cartography of Poland. The accuracy of this data corresponded to a map with a scale of 1:10 000 with a minimum area of land coverage patch of 0.1 ha.

Data preparation scheme

ArcGIS Pro software was used to prepare the data. In the first stage, using the Composite Bands tool Sentinel data bands: 4 (red), 3 (green), 2 (blue) and 8 (nir) all with a resolution of 10 meters, were combined and a four-band raster was created. Then the land cover vector data with the polygon topology were changed to raster data using the Polygon to Raster tool. The single-band raster was created with the same resolution as Sentinel data — the raster cell size was set to 10 meters. Choosing the right environment settings was a very important task. First of all, it was crucial to match the coordinate system and extent with the data acquired from Sentinel. The extent value read from Sentinel data allowed us to process only a portion of a larger land cover dataset. It was also important to determine Snap Raster environment setting. This allowed us to adjust the extent of output land cover raster to match the cell alignment of the specified Sentinel Image. In the second stage Sentinel Image was covered with a grid of squares using the Create Fishnet tool. In the third stage, by iterating through the grid items and using the Clip Raster tool, the Sentinel Image and Land cover raster data were cut into parts, overlapping with the consecutive mesh. The sample set was created. The same procedure was used for land cover data at both scales and to create samples of different sizes by changing the mesh parameter.

Model

The choice of the neural network architecture was determined by two factors. First one was the problem that we wanted to solve i.e. generating an artificial satellite image that meets precisely defined conditions in terms of land cover. Second one was the data sources that were at our disposal at the time of research. Such data comes in a form of single band TIFF masks where different pixel values represent land cover types. Also, Sentinel satellite, multi band data is publicly available. After defining our inputs (land cover masks) and outputs (artificial multiband satellite images) we were sure that Pix2Pix architecture will be a great starting point. We’ve based our implementation on the original paper describing this network, on additional sources presenting a possible implementation and this awesome article. You can find the code on github and download training data from our s3 bucket. Suggestions regarding the implementation are more than welcomed.