In a previous life, I was given the challenge of building a responsive, expandable UI framework that would run at high performance on multiple devices from desktop, tablets and phones all the way to Set Top Boxes and Smart TVs. I was thinking "Wow - what a great challenge!", then I was told that it had to be implemented in HTML5 and my heart sank. At the time I had no HTML development experience, and had also heard how large companies such as Facebook were moving away from HTML because they were unable to get the required performance to achieve a good user experience.
But the engineer in me was curious. All the target platforms were capable, they had good ARM® Cortex® CPU's and ARM® Mali™ GPUs: there must be a way to get the desired performance. The fact that Mark Zuckerberg had stated that the "Biggest mistake we took as a company was betting too much on HTML5 as opposed to native ... We were never able to get the quality we wanted", was like a red rag to a bull, I had to try and prove him wrong! If however you delve a little further into what Mark was saying, HTML5 was not, at the time, the right choice to target mobile platforms. The variance in spec conformance at the time was large, the rendering pipelines were not well optimized, and knowledge on the idiosyncrasies of developing for an embedded platform were not well understood at the time.
HTML5 was new to me. I had done a little work with Flash® Lite® 4, but the only takeaway I had from that experience was that I never wanted to work with ActionScript®, an ECMAScript based language again! This is probably not being fair to Adobe, who had traditionally done a great job with Flash, but from a developers perspective, the version ActionScript that came with FlashLite 4 could never work out which API it wanted to conform to. So the realization that Javascript (JS) was ECMAScript based, but without some of the good parts of ActionScript such as Object Oriented (OO) constructs like Inheritance, was not a good start. Also, understanding the concept of separation of content and layout through HTML and CSS was new. As you can see, there was a lot to learn!
But here lies a strength. If you approach a problem with very little knowledge, you are not bringing any previous prejudices with you. You are facing the problem with fresh eyes. This is me trying to give that great big cloud a silver lining!
So, development started... slowly... and I soon stumbled across the concept of the Polyfill. It turns out that due to the incredibly flexible nature of JS, you are able to add functionality that previously did not exist in either the JS or HTML specification. I was able to recreate some of the benefits of OO based inheritance by use of polyfills. An example of writing Polyfills can be found here.
Next I discovered was that there is a great community of HTML5 developers out there and a huge number of blog posts and tutorials to help the budding HTML5 App developer on their way. The greatest discovery here was how unbelievably fabulous the Chrome Developer Tools were! I'm not going to go into much detail about them here because there is already a huge amount of material online to get you started. If you have never used them before, I would suggest starting here, also checking out some of the tutorials at http://www.html5rocks.com and following the hashtag #perfmatters.
So, now I have the power tools for development and debugging, the polyfill to smear over the gaps in the HTML and JS APIs and a great set of libraries to help me build complex HTML5 applications (my choices here were http://requirejs.org/, http://backbonejs.org/, http://underscorejs.org/ and the ubiquitous http://jquery.com/). We are finally ready to start development.
The first thing I noticed was that, even though things were looking great on the desktop platforms, performance on the target embedded devices was shocking! Doom and gloom! Maybe Mark was correct - HTML5 is not yet ready for the prime time? But this was where the developer tools came to the rescue. Quick analysis of the application whilst running animations using the Chrome Devtools Timeline feature showed that I was spending a huge amount of time painting the contents of the page.
I found this strange because I was only moving assets across the screen, not updating their contents. In the image above, you can clearly see large areas of the screen being repainted. Surely this should be a simple composition job? Why was the whole screen being re-rendered on every frame?
It turns out that you have to be more expressive in your CSS to help the browser know which elements are going to be moved. This can be done by promoting the DOM (Document Object Model) element into its own layer. A good description of this method is covered here, but the basic premise is that if you apply a null 3D transform to the DIV (Document Division Element), for example:
<code>
.animate {
transform: translate3d(0,0,0);
}
</code>
will cause a new layer to be created for the DIV to be rendered into. If you then want to change the position of that item, you can update that same 3D translation to move the item around the screen. Because the DIV is now represented in a layer, which in turn is backed by a GPU texture, there is no need to repaint any part of the DOM when re-positioning the DIV and the animation boils down to a simple GPU backed composition job. Now we are working to the GPU's strengths!
Great! Problem solved!! Well, almost...
The use case being implemented in this example is a scrolling list. One factor of a scrolling list is that items are added to the front and removed from the end of the list. There are two ways in which I could have implemented the list:
- Have all list items available in the DOM when the screen loaded
- Dynamically add and remove items as the list scrolls
The problem with #1 is that there is no way of knowing how many items there will be in the list. I could be representing a channel lineup on TV, where we could have 800 items. If each of these items has a channel logo, and maybe an image representing what is currently showing on the channel, the memory requirements of the list will be huge! The likelihood would be that the application would abort with a failure and become unresponsive.
So #2 was chosen, and a framework built to manage the adding and removing of items. But there are problems with this approach as well. Every time we change the DOM, we get a 'judder' in the UI as the DOM recalculates layout, downloads and decodes images for new assets and renders the contents into a new layer. So every time we move the list, we are triggering major layout and repaint events.
Bring on the Magician
So, the title of this blog talks about illusion.... This is where we need a magician, and a good one at that!
The solution to this particular problem is to understand the difference between 'interaction time' and 'browser time'. I'm not sure if these are industry standard names for what I am about to talk about, but they have become how I refer to the issues.
Interaction time is the time when I am interacting with the UI. This is also the time at which my perception of application framerate is at its highest. Imagine I am using a touch device and I am sliding my finger across the screen. If the item I am moving does not track my finger immediately and without lag, I will notice and the experience will be poorer as a result.
Browser time is the time in-between interaction time. During this time, the framerate could fluctuate drastically and I wouldn't notice. There would be no impact on my experience during this time.
The idea is to defer any operations that would impact the framerate of the application into browser time. By doing this, you are disguising the inefficient paths of the browser and making much better use of the correct resources when they are needed.
This blog does not cover the whole of the story, I will be posting more in the weeks to come. The bulk of this post was covered in a presentation I gave at the Developer Summit at ARM TechCon™ earlier this year.
I will also be giving a version of the presentation again at DevCon5 in California on the 10th December, it would be great to see you there.
Conclusion
----------
So, did I prove Mark wrong - well yes and no.
It turns out that efficient HTML5 development is hard, but its not impossible. So long as you are willing to put in the effort to understand what is happening under the hood, you stand a much better chance of crafting an application that performs well on your target devices. The great news is that the ARM Mali Ecosystem team are working hard with a number of browser vendors to make sure that future browser engines perform well on ARM Mali based GPU's.
I found the experience challenging but ultimately rewarding, and I would encourage you to get involved in the HTML5 community and build beautiful bodacious web apps for our ARM based devices.