This article is part of Alibaba’s Utilizing Flutter series.

In computing, as in much of life, any given method can see a lot of use before its latent flaws reach a decisive impasse. For Alibaba, discovering one such flaw in software development kit Flutter meant the difference between success and failure in the group’s recent work on a mobile app for its Xianyu(闲鱼) second-hand trading platform.

Now the Alibaba team has successfully optimized Flutter for a new range of uses unique to Xianyu’s marketplace, implementing an OpenGL process for the entire UI rendering process to reduce CPU and GPU resource overhead.

In today’s article, we look at the group’s optimization efforts alongside a detailed view of the inner workings and “external texture” of Flutter that technical audiences can explore in their own work.

Flutter Rendering Framework

The Flutter rendering framework is organized into a series of layers, with each layer building upon the previous layer. The architecture design of the Flutter Rendering Framework is shown below:

The Flutter rendering framework, by layer

1. Layer Tree: The rendering pipeline is a tree-like structure output by the Dart API at runtime. Each leaf node on the tree represents an interface element, such as buttons, images, and so on.

2. Skia: A cross-platform rendering framework sponsored and managed by Google. It serves as graphics engine for iOS/Android applications. The bottom layer of Skia is known as OpenGL drawing. Vulkan support is very limited and Metal does not provide the support.

3. Shell: A platform feature that includes iOS/Android platform implementations, EAGLContext management, returning data on a screen, and the external texture implementations.

The Layout process is executed at the Dart runtime and outputs a Layer tree. Each leaf node of the Layer tree is traversed in the pipeline to call the Skia engine for completing the drawing of the interface elements.

After completing the above-mentioned process, do the following:

1. iOS: Run the glPresentRenderBuffer command to display the render buffers contents on the screen.

Android: Run the glSwapBuffer command to perform a buffer swap on the layer in use for the current window.

2. Click the Complete display-on-screen link.

Based on this principle, Flutter can implement UI isolation on Native and Flutter Engine. It also captures the UI code without analyzing the platform implementation on the cross-platform solutions.

Implementation Problems

There are pros and cons to this implementation. Flutter is isolated from Native, which can sometimes make it feel like there is a mountain separating Flutter Engine and Native. This poses issues when Flutter wants to capture high-memory images of the Native side, such as camera frames, video frames, album images, and so on.

Traditional applications (RN, Weex, and so on) can directly obtain this data by bridging the NativeAPI. Flutter, meanwhile, determines whether the data can be directly captured and sends a notification message based on the defined channel mechanism, inevitably leading to huge memory usage and CPU utilization while transmitting the data.

Bridging the Gap with External Texture

Flutter provides a special mechanism known as an external texture. Note that textures are images that can be applied to an area of the Flutter view. They are created, managed, and updated using a platform-specific texture registry. This is typically done by a plugin that integrates with host platform video player, camera, or OpenGL APIs, or similar image sources.

The architecture diagram of LayerTree is shown below:

LayerTree architecture

Each leaf node represents a control of dart code layout. TextureLayer node at the end corresponds to the texture control in Flutter. When a texture control is created in Flutter, it represents the data displayed on this control that needs to be provided by Native. Note that this texture is different from GPU’s texture. It is for Flutter’s control.

The following is the final drawing code of the TextureLayer node on the iOS platform. The code for the Android platform is similar, but its method of obtaining texture is slightly different.

It is recommended to run the code following these three steps:

1. Make the call of external texture’s copyPixelBuffer function to get CVPixelBuffer

2. Create the OpenGL ES Texture — CVOpenGLESTextureCacheCreateTextureFromImage

3. Capture the OpenGL ES texture into a SKImage and make the call of Skia’s DrawImage function to complete drawing.

void IOSExternalTextureGL::Paint(SkCanvas& canvas, const SkRect& bounds) {

if (!cache_ref_) {

CVOpenGLESTextureCacheRef cache;

CVReturn err = CVOpenGLESTextureCacheCreate(kCFAllocatorDefault, NULL,

[EAGLContext currentContext], NULL, &cache);

if (err == noErr) {

cache_ref_.Reset(cache);

} else {

FXL_LOG(WARNING) << "Failed to create GLES texture cache: " << err;

return;

}

}

fml::CFRef<CVPixelBufferRef> bufferRef;

bufferRef.Reset([external_texture_ copyPixelBuffer]);

if (bufferRef != nullptr) {

CVOpenGLESTextureRef texture;

CVReturn err = CVOpenGLESTextureCacheCreateTextureFromImage(

kCFAllocatorDefault, cache_ref_, bufferRef, nullptr, GL_TEXTURE_2D, GL_RGBA,

static_cast<int>(CVPixelBufferGetWidth(bufferRef)),

static_cast<int>(CVPixelBufferGetHeight(bufferRef)), GL_BGRA, GL_UNSIGNED_BYTE, 0,

&texture);

texture_ref_.Reset(texture);

if (err != noErr) {

FXL_LOG(WARNING) << "Could not create texture from pixel buffer: " << err;

return;

}

}

if (!texture_ref_) {

return;

}

GrGLTextureInfo textureInfo = {CVOpenGLESTextureGetTarget(texture_ref_),

CVOpenGLESTextureGetName(texture_ref_), GL_RGBA8_OES};

GrBackendTexture backendTexture(bounds.width(), bounds.height(), GrMipMapped::kNo, textureInfo);

sk_sp<SkImage> image =

SkImage::MakeFromTexture(canvas.getGrContext(), backendTexture, kTopLeft_GrSurfaceOrigin,

kRGBA_8888_SkColorType, kPremul_SkAlphaType, nullptr);

if (image) {

canvas.drawImage(image, bounds.x(), bounds.y());

}

}

Where does the external_texture_object come from? Before making the call of RegisterExternalTexture on the Native side, create an object for implementing the FlutterTexture protocol, which is assigned to the external_texture object. The external texture is a bridge between Flutter and Native, used to continuously obtain the image data to be displayed.

As shown in the figure, the PixelBuffer is the carrier of data transmitted by Flutter and Native while using the external texture. The data source (camera, player, and so on) of the Native side transmits the data into PixelBuffer. Flutter takes PixelBuffer and converts it into OpenGL ES texture for Skia to complete the drawing.

At this point, Flutter can easily draw all the data that the Native side wants to draw. In addition to the dynamic image data (the camera player), the display of the image provides another possibility beyond the Image control, especially for the Native side when there are large-scale image loading libraries like SDWebImage. This process is very time consuming and laborious when there is a need to write a copy with dart on the Flutter side.

Optimizing Processing Speed

The entire process described above seems to solve Flutter’s problem of displaying native-side big data, but it has the following limitations: