Graphics and UI

The application composes the graphic user interface by creating a window with a toolkit. The display server composites an application’s windows and shows the result on the screen. For this procedure, the graphics and UI middleware offers the following 3 modules for both client and server:

• Tizen Buffer Manager (TBM)
• Tizen Display Manager (TDM)
• TPL-EGL

Figure: Graphics UI diagram

The modules are hardware abstraction layers for graphics and UI. They allow the client and server to render with the GPU, share buffers with other processes, and organize hardware output devices for various chipsets. Their backend module needs to be implemented for the new hardware device:

• TBM provides an abstraction interface for the Tizen graphic buffer manager.
• TDM provides an abstraction interface for a display server, such as X or Wayland, to allow direct access to graphics hardware in a safe and efficient manner as a display HAL.
• TPL-EGL is an abstraction layer for surface and buffer management on the Tizen platform, aimed to implement the EGL porting layer of the OpenGL ES driver over various display protocols.

For an application to handle input device events, the Input Manager is provided, and is mainly comprised of libinput and a thin wrapper around it. It handles input events in Wayland compositors and communicates with Wayland clients.

Buffer management

TBM has a frontend library and a backend module. The TBM frontend library is hardware-independent and provides a generic buffer interface. On the other hand, the TBM backend module is hardware-dependent and provides a buffer interface dependent on the target system. Chipset vendors have to provide their own backend modules in order for TBM to work well on the Tizen platform. This is because the way each vendor manages the graphic buffers can be different between various chipset devices. TBM already has several reference backends, such as libtbm-dumb, and libtbm-shm.

Figure: TBM backend

With TBM, the client and server can allocate buffers and share buffers between them. For example, a client allocates a graphic buffer, draws something on it with GL and sends it to the display server for displaying it on the screen without buffer copying. The TBM backend module is implemented as a shared library and the TBM frontend finds the libtbm-default.so file and loads it from the /usr/lib/bufmgr directory at runtime:

sh-3.2# ls -al
lrwxrwxrwx  1 root root    14 Jul 28  2016 libtbm_default.so -> libtbm_sprd.so
lrwxrwxrwx  1 root root    20 Jul 28  2016 libtbm_sprd.so -> libtbm_sprd.so.0.0.0
lrwxrwxrwx  1 root root    20 Jul 28  2016 libtbm_sprd.so.0 -> libtbm_sprd.so.0.0.0
-rwxr-xr-x  1 root root 26728 Jun 29  2016 libtbm_sprd.so.0.0.0

Initializing TBM backend module

The TBM backend module must define the global data symbol with the name, tbm_backend_module_data. The TBM frontend reads the global data symbol at the initialization time. In addition, the TBM backend module calls init() of tbm_backend_module_data. For more information, see tbm_backend.h:

typedef struct _tbm_backend_module {
	const char *name;           /**< The module name of the backend module */
	const char *vendor;         /**< The vendor name of the backend module */
	unsigned long abi_version;  /**< The ABI version of the backend module */
	/**
	 * @brief The init function of the backend module
	 * @param[in] bufmgr: A TBM buffer manager object
	 * @return The backend module data
	 * @see tbm_backend_bufmgr_data
	 */
	tbm_backend_bufmgr_data *(*init)(tbm_bufmgr bufmgr, tbm_error_e *error);
	/**
	* @brief deinitialize the bufmgr private data
	* @param[in] bufmgr_data : The backend module data
	*/
	void (*deinit)(tbm_backend_bufmgr_data *bufmgr_data);
} tbm_backend_module;

#include <tbm_backend.h>

static tbm_backend_bufmgr_data *bufmgr_data;

tbm_backend_bufmgr_data*
tbm_shm_init(tbm_bufmgr bufmgr, tbm_error_e *error)
{
    bufmgr_data = calloc(1, sizeof(tbm_backend_bufmgr_data));

    return (tbm_backend_bufmgr_data*)bufmgr_data;
}

void
tbm_shm_deinit(tbm_backend_bufmgr_data *bufmgr_data)
{
    free(bufmgr_data);
}

tbm_backend_module tbm_backend_module_data = {
	"shm",
	"Samsung",
	TBM_BACKEND_ABI_VERSION_3_0,
	tbm_shm_init,
	tbm_shm_deinit
};

The TBM backend must register tbm_backend_bufmgr_func and tbm_backend_bo_func with tbm_backend_bufmgr_register_bufmgr_func() and tbm_backend_bufmgr_alloc_bo_func() in init() of tbm_backend_module:

#include <tbm_backend.h>

tbm_backend_bufmgr_data*
tbm_shm_init(tbm_bufmgr bufmgr, tbm_error_e *error)
{
	bufmgr_func->bufmgr_get_capabilities = tbm_shm_bufmgr_get_capabilities;
	bufmgr_func->bufmgr_bind_native_display = tbm_shm_bufmgr_bind_native_display;
	bufmgr_func->bufmgr_get_supported_formats = tbm_shm_bufmgr_get_supported_formats;
	bufmgr_func->bufmgr_get_plane_data = tbm_shm_bufmgr_get_plane_data;
	bufmgr_func->bufmgr_alloc_bo = tbm_shm_bufmgr_alloc_bo;
	bufmgr_func->bufmgr_alloc_bo_with_format = NULL;
	bufmgr_func->bufmgr_import_fd = tbm_shm_bufmgr_import_fd;
	bufmgr_func->bufmgr_import_key = NULL;

	err = tbm_backend_bufmgr_register_bufmgr_func(bufmgr, bufmgr_func);
	if (err != TBM_ERROR_NONE) {
		TBM_ERR("fail to register bufmgr_func! err(%d)\n", err);
		if (error)
			*error = TBM_ERROR_INVALID_OPERATION;
		goto fail_register_bufmgr_func;
	}
	bufmgr_shm->bufmgr_func = bufmgr_func;

	bo_func->bo_free = tbm_shm_bo_free;
	bo_func->bo_get_size = tbm_shm_bo_get_size;
	bo_func->bo_get_memory_types = tbm_shm_bo_get_memory_type;
	bo_func->bo_get_handle = tbm_shm_bo_get_handle;
	bo_func->bo_map = tbm_shm_bo_map;
	bo_func->bo_unmap = tbm_shm_bo_unmap;
	bo_func->bo_lock = NULL;
	bo_func->bo_unlock = NULL;
	bo_func->bo_export_fd = tbm_sprd_bo_export_fd;
	bo_func->bo_export_key = NULL;

	err = tbm_backend_bufmgr_register_bo_func(bufmgr, bo_func);
	if (err != TBM_ERROR_NONE) {
		TBM_ERR("fail to register bo_func! err(%d)\n", err);
		if (error)
			*error = TBM_ERROR_INVALID_OPERATION;
		goto fail_register_bo_func;
	}
	bufmgr_shm->bo_func = bo_func;

	return (tbm_backend_bufmgr_data *)bufmgr_shm;
}

Porting OAL interface

TBM provides the header files to implement the TBM backend module.

Table: TBM backend module header files

TBM backend interface

The following table lists the bufmgr backend interface functions of tbm_backend_module. For more information, see tbm_backend.h:

Table: bufmgr functions

The following table lists the bo backend interface functions of tbm_backend_module. For more information, see tbm_backend.h:

Table: bo functions

The following table lists the TBM buffer manager capability, tbm_bufmgr_capability:

The following table lists the TBM buffer memory types, tbm_bo_memory_type:

The following table lists the TBM buffer device types, tbm_bo_device_type:

The following table lists the TBM buffer access options, tbm_bo_access_option:

TBM DRM helper functions

If the target uses the drm interface, the client needs to get the authenticated fd from the display server and the display server must share the drm master fd with the TDM backend module. The TBM frontend provides the helper functions for drm authentication with the Wayland protocol and shares the master fd with the TDM backend module.

Table: DRM helper functions

TBM backends

The following table lists the TBM backends.

Table: TBM backends

Testing porting result

TBM offers tbm-haltests that allows you to test and verify the porting result. The tbm-haltests tool is included in the libtbm-haltests package that can be downloaded from the platform binary’s snapshot repository. It depends on the gtest package and it can be downloaded from the platform’s snapshot repository.

Checking TDM log messages

TBM uses dlog to print the debug messages. To show the TBM run time log, use the following message:

$ dlogutil -v threadtime TBM

Reference

For more information about TBM and TBM backend, see Tizen Buffer Manager (TBM).

Display management

The display server composites and shows the client’s buffers on screen. The display server sometimes needs to convert or scale an image to a different size or format. To make it possible for various chipset devices, the display server needs the display hardware resource information and control over the resources. Tizen Display Manager (TDM) offers these functionalities for the display server with the unified interface for various chipset devices.

Figure: TDM backend

With TDM, the display server can perform mode setting, DPMS control, and showing a buffer (framebuffer or video buffer) on the screen in the most efficient way. If the hardware supports the m2m converting and capture device, the display server can also convert an image and dump a screen including all hardware overlays with no compositing.

The vendor has to implement the TDM backend module. The TDM backend module has the responsibility to let the TDM frontend know the display hardware resource information. The display server gets this information and controls hardware devices through the TDM frontend APIs. TDM already has several backends for reference, such as libtdm-drm and libtdm-fbdev.

The TDM backend is implemented as a shared library. The TDM frontend finds the libtdm-default.so file and loads it in the /usr/lib/tdm directory at runtime:

sh-3.2# ls -l /usr/lib/tdm
total 40
lrwxrwxrwx 1 root root    14 Jul 28  2016 libtdm-default.so -> libtdm-drm.so
-rwxr-xr-x 1 root root 37152 Jul 12  2016 libtdm-drm.so

Initializing TDM backend module

The TDM backend module must define the global data symbol with the name tdm_backend_module_data. The TDM frontend reads this symbol at the initialization time. TDM calls the init() function of the tdm_backend_module_data. For more information, see tdm_backend.h:

typedef struct _tdm_backend_module {
    const char *name; /* The module name of the backend module */
    const char *vendor; /* The vendor name of the backend module */
    unsigned long abi_version; /* The ABI version of the backend module */
    tdm_backend_data *(*init)(tdm_display *dpy, tdm_error *error);
    void (*deinit)(tdm_backend_data *bdata);
} tdm_backend_module;

#include <tdm_backend.h>

static tdm_drm_data *drm_data;

tdm_backend_data*
tdm_drm_init(tdm_display *dpy, tdm_error *error) {
    drm_data = calloc(1, sizeof(tdm_drm_data));

    return (tdm_backend_data*)drm_data;
}

void
tdm_drm_deinit(tdm_backend_data *bdata) {
    free(bdata);
}

tdm_backend_module tdm_backend_module_data = {
    "drm",
    "Samsung",
    TDM_BACKEND_SET_ABI_VERSION(2,0),
    tdm_drm_init,
    tdm_drm_deinit
};

The TDM backend must register the tdm_func_display(), tdm_func_output(), and tdm_func_layer() functions with the tdm_backend_register_func_display(), tdm_backend_register_func_output(), and tdm_backend_register_func_layer() functions in the tdm_backend_module_data init() function:

#include <tdm_backend.h>

tdm_backend_data*
tdm_drm_init(tdm_display *dpy, tdm_error *error) {
    memset(&drm_func_display, 0, sizeof(drm_func_display));
    drm_func_display.display_get_capability = drm_display_get_capability;
    drm_func_display.display_get_pp_capability = drm_display_get_pp_capability;
    drm_func_display.display_get_outputs = drm_display_get_outputs;
    drm_func_display.display_get_fd = drm_display_get_fd;
    drm_func_display.display_handle_events = drm_display_handle_events;
    drm_func_display.display_create_pp = drm_display_create_pp;
    ret = tdm_backend_register_func_display(dpy, &drm_func_display);
    if (ret != TDM_ERROR_NONE)
        goto failed;

    memset(&drm_func_output, 0, sizeof(drm_func_output));
    drm_func_output.output_get_capability = drm_output_get_capability;

    ret = tdm_backend_register_func_output(dpy, &drm_func_output);
    if (ret != TDM_ERROR_NONE)
        goto failed;

    memset(&drm_func_layer, 0, sizeof(drm_func_layer));
    drm_func_layer.layer_get_capability = drm_layer_get_capability;

    ret = tdm_backend_register_func_layer(dpy, &drm_func_layer);
    if (ret != TDM_ERROR_NONE)
        goto failed;

    return (tdm_backend_data*)drm_data;
}

After loading the TDM backend module, the TDM frontend calls display_get_capability(), display_get_outputs(), and output_get_capability() to get the specific hardware information. The TDM backend module has to set TDM_OUTPUT_CAPABILITY_HWC on the output, when TDM_OUTPUT_CAPABILITY_HWC supports TDM HardWare Compositing (HWC). In the latest version (supports version 2.9) of libtdm, TDM recommends that the TDM backend module must support TDM HWC, which means that TDM backend has to register tdm_func_hwc and tdm_func_hwc_window. If the TDM backend module does not support TDM HWC, the TDM backend module implements output_get_layers() and layer_get_capability(), and also registers tdm_func_layer().

In addition, if a target has a memory-to-memory converting hardware device and the capture hardware device, the TDM backend module can register the tdm_func_pp() and tdm_func_capture() functions with the tdm_backend_register_func_pp() and tdm_backend_register_func_capture() functions.

Porting the OAL interface

TDM provides the header files to implement the TDM backend module.

Table: TDM backend module header files

The display backend interface is mandatory. For more information, see tdm_backend.h.

Table: Display backend interface functions

The output backend interface is mandatory. For more information, see tdm_backend.h.

Table: Output backend interface functions

The HWC backend interface is mandatory. For more information, see tdm_backend.h.

Table: HWC backend interface functions

The hwc window backend interface is mandatory. For more information, see tdm_backend.h.

Table: HWC window backend interface functions

The layer backend interface is mandatory. For more information, see tdm_backend.h.

Table: Layer backend interface functions

The pp backend interface is optional. For more information, see tdm_backend.h.

Table: pp backend interface functions

The capture backend interface is optional. For more information, see tdm_backend.h.

Table: Capture backend interface functions

TDM backends

There are several backends which can be used as reference when implementing the TDM backend.

Table: TDM backends

Testing porting result

TDM offers tdm-haltests that allows you to test and verify the porting result. The tdm-haltests tool is included in the libtdm-haltests package that can be downloaded from the platform binary’s snapshot repository. It depends on the gtest package, and it can be downloaded from the platform’s snapshot repository.

Checking TDM log messages

TDM uses dlog to print debug messages. To show TDM runtime log messages:

$ dlogutil -v threadtime TDM

References

For detailed information about TDM and the TDM backend, see Tizen Display Manager (TDM).

Input management

The input manager supports a libinput-based input device backend. libinput is a common input library for the Wayland compositor. With libinput, the input stack is simpler without the Xorg input drivers. Since Tizen 3.0, the input manager is not a HAL component.

Figure: Tizen 3.0 input management

libinput

The libinput library handles input devices for display servers and other applications that need to directly deal with input devices:

• Device detection
• Device handling
• Input device event processing
• Scaling touch coordinates
• Generating pointer events from touchpads
• Pointer acceleration

For more information, see the libinput wiki.

libevdev

The libevdev library handles evdev kernel devices. It abstracts the evdev ioctls through type-safe interfaces and provides functions to change the appearance of the device. For more information, see https://en.wikipedia.org/wiki/Evdev.

mtdev

The mtdev standalone library transforms all variants of kernel MT events to the slotted type B protocol. For more information, see http://www.linuxfromscratch.org/blfs/view/svn/general/mtdev.html.

libinput Backends

libinput: platform/upstream/libinput

OpenGL

This section describes the essential elements of the Tizen platform-level graphics architecture related to OpenGL ES and EGL, and how it is used by the application framework and the display server. The focus is on how graphical data buffers move through the system.

The Tizen platform requires the OpenGL ES driver for the acceleration of the Wayland display server and the wayland-egl client. This platform demands an OpenGL ES and EGL driver, which are implemented by the Tizen EGL Porting Layer.

Tizen OpenGL ES and EGL architecture

The following figure illustrates the Tizen OpenGL ES and EGL architecture.

Figure: Tizen OpenGL ES architecture

• CoreGL

  An injection layer of OpenGL ES that provides the following capabilities:

  • Support for driver-independent optimization (FastPath)
  • EGL/OpenGL ES debugging
  • Performance logging
  • Supported versions
    • EGL 1.4
    • OpenGL ES 1.1, 2.0, 3.0, 3.1

  CoreGL loads the manufacturer’s OpenGL ES driver from the /usr/lib/driver directory. CoreGL provides libEGL.so, libGLESv1_CM.so, and libGLESvs.so driver files in the /usr/lib directory.

• GPU vendor GL/EGL driver

  The Tizen platform demands that the GPU vendor implements the GL and EGL driver using libtpl-egl. The GPU vendor GL/EGL driver (libEGL.so, libGLESv1_CM.so, libGLESv2.so) must be installed in the /usr/lib/driver path.

Tizen Porting Layer (TPL) for EGL

TPL-EGL is an abstraction layer for surface and buffer management on the Tizen platform. It is used for the implementation of the EGL platform functions.

Figure: TPL architecture

The background for the Tizen EGL Porting Layer for EGL uses various Tizen window system protocols. Therefore, there is a need to separate the common layer and backend.

Tizen uses the Tizen Porting Layer for EGL, as the TPL-EGL API prevents burdens of the EGL porting on various window system protocols. The GPU GL Driver’s Window System Porting Layer can be implemented by TPL-EGL APIs, which are the corresponding window system APIs. The TBM, Wayland, and GBM backends are supported.

Tizen Porting Layer for EGL object model

TPL-EGL provides interfaces based on an object-driven model. Every TPL-EGL object can be represented as a generic tpl_object_t, which is reference-counted and provides common functions. Currently, display and surface types of TPL-EGL objects are provided. A display, like a normal display, represents a display system which is usually used for connecting to the server. A surface corresponds to a native surface, such as wl_surface. Surfaces can be configured to use N-buffers, but are usually double-buffered or triple-buffered. A buffer is what you render on, usually a set of pixels or a block of memory. For these 2 objects, the Wayland, GBM, TBM backend are defined, and they correspond to their own window systems. This means that you do not need to care about the window systems.

The TPL-EGL has the following core objects:

• TPL-EGL Object

  Base class for all TPL-EGL objects.

• TPL-EGL Display

  Encapsulates the native display object (Display *, wl_display). Like a normal display, this represents a display system which is usually used for connecting to the server, scope for other objects.

• TPL-EGL Surface

  Encapsulates the native drawable object (Window, Pixmap, wl_surface). The surface corresponds to a native surface, such as tbm_surface_queue or wl_surface. A surface can be configured to use N-buffers, but they are usually double-buffered or triple-buffered.

TPL-EGL objects and corresponding EGL objects

Both TPL-EGL and vendor OpenGL ES/EGL driver handles tbm_surface as the corresponding TPL surface buffer. It is represented by the TBM_Surface part in the following figure.

Figure: TPL-EGL architecture

The following figure illustrates the OpenGL ES drawing API flow.

Figure: OpenGL ES drawing API flow

TPL-EGL frontend API

TPL-EGL Object is a base class for all TPL-EGL objects. It provides common functionalities to all TPL-EGL objects.

Table: TPL-EGL Object functions

TPL-EGL Display encapsulates the native display object (Display *, wl_display). Any other objects created from TPL-EGL Display inherit its backend type.

Table: TPL-EGL Display functions

TPL-EGL Surface encapsulates the native drawable object (Window, Pixmap, wl_surface). The main features of the class are retrieving the buffer for a frame and posting the surface to a screen.

Table: TPL-EGL Surface functions

The following code snippet shows a simple example of the Tizen Porting Layer:

dpy = tpl_display_create(...);
sfc = tpl_surface_create(dpy, ...);

while (1) {
    buf = tpl_surface_dequeue_buffer(sfc);

    /* Draw something */

    tpl_surface_enqueue_buffer(sfc, buf);
}

In the GPU vendor driver, the GPU frame builder handles the drawing. TPL-EGL exposes the native platform buffer identifiers and managers so that the buffer can be used in other modules. Currently, dma_buf/DRM is supported for these purposes. The EGL porting layer calls TPL-EGL functions to execute commands requested of it, and returns the results to the GPU vendor driver. TPL-EGL performs all protocol-dependent actions. Such protocol-dependent parts can be separated into TPL-EGL backends. TPL-EGL backend can also be configured at runtime, and you can specify which type of backend to use when initializing a display object.

TPL-EGL and Wayland server and client

Tizen uses the wl_tbm protocol instead of wl_drm. The wl_tbm protocol is designed for sharing the buffer (tbm_surface) between wayland_client and wayland_server. Although the wayland_tbm_server_init and wayland_tbm_client_init pair is a role for eglBindWaylandDisplayWL, the EGL driver is required to implement the entry points for eglBindWaylandDisplayWL and eglUnbindWaylandDisplayWL as dummy. For more information, see https://cgit.freedesktop.org/mesa/mesa/tree/docs/specs/WL_bind_wayland_display.spec.

Figure: TPL-EGL and Wayland

Buffer flow between the Wayland server and OpenGL ES/EGL driver

The following figure shows the buffer flow between the Wayland server and the OpenGL ES/EGL driver. The passed buffer is of the tbm_surface type.

Figure: Buffer flow between Wayland server and OpenGL ES/EGL driver

Project Git repository

The following table lists the available project Git repositories.

Table: Git repositories

libtpl-egl reference driver

The Emulator YAGL (OpenGL ES/EGL driver for the emulator) is implemented by libtpl-egl.

The following commit explains how to port the driver with libtpl-egl from the traditional drm-based driver:

• Porting YAGL to the Tizen platform https://review.tizen.org/gerrit/c/platform/adaptation/emulator/emulator-yagl/+/67921
• Porting MESA to the Tizen platform https://review.tizen.org/gerrit/c/platform/upstream/mesa/+/228724

Testing and verifying OpenGL ES driver

The Khronos OpenGL ES CTS supports wayland-egl. libtpl-egl has a test case for the libtpl-egl. tpl-novice of ws-testcase has the sample code for libtpl-egl.