Since our successful demonstration at SIGGRAPH, ARM and Collabora have continued to work together on providing the best possible platform for media playback and presentation. Numerous applications such as digital signage, IVI, tablets, remote monitoring, and more, all require accurate, high-quality and low-power video presentation, with as low a thermal envelope as possible.
Collabora has made significant contributions to, and maintenance of, both the standard open-source GStreamer media framework, and the next-generation Wayland window system. Combining the two has allowed us to bring out the full extent of the capabilities of GStreamer, for years used in broadcast television with its exacting standards, and Wayland's lightweight and flexible design, which above all else emphasises accuracy and perfect end results.
The result is a system providing perfectly synchronised network video presentation. Three displays, all powered by separate ODROID-XU3 systems using the Samsung EXYNOS 5422 SoC with an ARM MALI-T620 GPU. Each system displays one segment of the video, with one acting as the co-ordinator to keep timing consistent across all three segments. From the user's point of view, the video appears as one consistent whole.
Network synchronisation with GStreamer
GStreamer is the reference open-source media framework, used in everything from audio playback on embedded systems, to huge farms powering broadcast TV. GStreamer's pipeline concept provides a flexible and lightweight transport to suit almost any usecase. In this particular instance, we are using GStreamer to load H.264 content from disk, pass it to a hardware H.264 decoder, feed the resulting frames to Wayland, and then feed the timing information from Wayland back to the master device.
Core to GStreamer's flexibility and applicability has been its excellent support for clock control, being able to synchronise multiple disparate sources. Its clock control supports both hardware and software sources and sinks and allows the most precise matching possible between input audio and video clocks, and the output device's actual capabilties.
GStreamer's measurement was then supplemented by an open-source distributed media control system called Aurena, which uses these measurement reports and targets from GStreamer to synchronise playback across all three devices.
The work we did with GStreamer to enhance its Wayland and H.264 hardware decoding support is both already merged to the upstream open-source project, and fully hardware-independent.
Accurate display with Wayland
The next-generation Wayland window system allows us to make the most efficient possible use of the hardware IP blocks, not only maximising throughput (thus increasing the highest achievable resolution, or number of streams, without sacrificing quality), but also providing predictable presentation.
Wayland's design goal of 'every frame is perfect' means that the content shown to the user must always be complete, coherent, and well-timed. The frame-based model employed is a significant stride over legacy X11 and DirectFB systems, and the consideration given to timing concerns allows us to make sure that the media is always delivered as close to on time as possible, without unsightly visual artifacts such as tearing.
Building on this solid and well-tested core, Collabora developed multiple extensions to Wayland. The first ensures that no copies of the video data are made in the compositing process, preserving precious memory bandwidth, latency, and overall system responsiveness. This extension uses the latest Khronos Group EGL extensions, as supported by ARM's MALI GPU.
However, even without this copy stage, as video usage continues to push at the margins of hardware performance – one recent customer project involved 4K output of nine 1080p H.264 streams on an embedded system – we realise that it might not be physically possible to obtain full frame-rate at all times. To compensate for this, Collabora developed an additional Wayland extension, allowing not only real-time feedback of actual hardware presentation time, but ahead-of-time frame queueing.
This feedback mechanism allows GStreamer to dynamically adjust its clock to obtain perfect synchronisation both across devices and between audio/video, whilst the ahead-of-time queueing gives the hardware the best possible chance to make frame deadlines, as well as preserving power by allowing the hardware to enter sleep states for longer.
This work is all either included with current Wayland releases, or actively being discussed and developed as part of the upstream open-source community.
Hardware enablement
Neither GStreamer nor Wayland required any hardware-specific development or tweaking. However, in order to make this work possible, Collabora has worked extensively on the kernel drivers for the Exynos 5422 SoC found inside the ODROID-XU3. Bringing the Exynos hardware support up to speed with the latest developments in the Kernel Modesetting and Video4Linux 2 subsystems, as well as fixing bugs found in our automated stress-testing laboratory, allowed this work to proceed without a hitch.
Far from being throwaway, this work is being merged into the upstream Linux kernel and U-Boot projects, as part of our ongoing commitment to working closely with the open source community to raise the bar for quality and functionality. As this and other platforms rapidly adopt these improvements and become able to run this work, device manufacturers are able to select from the greatest possible choice of vendors. Our work with our partners, including ARM, the wider open source community, and membership of the Khronos Group, continues to deliver benefits for the entire ecosystem, not just one platform or device.
This open standards-based approach allows platform selection to be driven by the true capabilities of the hardware and cost/logistics concerns, rather than having to fret about software capability and vendor lock-in.
Further development
Of course, the power of GStreamer, Wayland, and a standards-based Linux system does not just stop there: its extensibility includes support for OpenGL ES and EGL, as well as arbitrary client-defined content. Not only can applications like web browsers take advantage of this synchronisation in order to embed seamless media content in HTML5 displays, but the source data can be anything from a single OpenGL ES application, a web browser, or anything else imaginable. Whether providing for immersive gaming experiences or large-scale digital signage, the underlying technology is flexible and capable enough to deal with any needs.
ARM is a registered trademark of ARM Limited (or its subsidiaries) in the EU and/or elsewhere. All rights reserved.