UI2code uses deep learning to analyze a UI design and convert it directly into skeleton code for front-end developers, saving them huge amounts of time when implementing designers’ work into apps. The first stage of this conversion process is layout analysis, and for an app with a simple white background, slicing image information into GUI elements can be as satisfyingly simple as slicing a cake.

To get an overview of Alibaba’s UI2code, find out here: Introducing UI2CODE: An Automatic Flutter UI Code Generator

However, in the majority of cases, it can feel more like trying to gut a fish for the first time: the results can be a bit messy until you’ve fine-tuned your approach. This was the case for staff working on Alibaba’s Xianyu app, a platform for users to sell second-hand goods.

Specifically, they encountered difficulties with the background analysis and foreground analysis parts of the process, which is shown in its entirety in the following figure.

Background/foreground analysis entails the slicing of UI visuals into GUI elements using context separation algorithms:

· Background analysis: uses machine vision algorithms to analyze background color, gradient direction, and connected regions of the background.

· Foreground analysis: uses deep learning algorithms to collate, merge, and recognize GUI fragments.

This article presents the experience and results of the Xianyu team in fine-tuning the background/foreground analysis algorithms. Foreground analysis is more complex and required much more fine-tuning; for this reason, the majority of this article is devoted to this topic.

Background Analysis

The key to background analysis is to find the connected areas and area boundaries. This section goes through the background extraction process step by step, using the actual Xianyu UI as an example.

Step 1

This step seeks to obtain solid-color and gradient background regions by:

· Identifying background blocks

· Calculating the direction of the gradient change through edge detection algorithms (such as sobel, Laplacian, canny, etc.)

The background region extraction algorithm, which is based on the Laplacian operator, is as follows: