March 8, 2022

Lightning Performance Analysis

"How does Lightning compare to HTML5?"

When selecting an Application Framework for your next project it is important to understand what the performance benefits are. One of the main recurring questions is "How does Lightning compare to HTML5?" and in this page we'll answer the performance aspect of Lightning versus HTML5.

We'll use 2 different classes of embedded devices, one on the low-end and one of the higher-end spectrum of performance (at the time of writing this article).

Please note these devices are very bespoke and differ greatly from your typical target PC. Albeit Lightning can be perfectly used for PCs or high end devices, the performance gain is higher on low-end devices.

Test conclusion (brief)

Test overview

The devices used in this analysis:

Low-end Set-top Box

CPU: Broadcom SOC 7428 (Dual-core MIPS CPU 3000 DMIPS) Mem: 1 GB Shared (~200 MB available) GPU: VideoCore 3 (~195 MB Gfx memory available)

See software stack details below. For those unfamiliar with STB hardware this particular device has a more or less equivalent processing power of an iPhone 4G (4k DMIPS on Apple A8 ARM based) released in 2010.

High-end Set-top Box

CPU: Broadcom SOC 7268 (Quad-core ARM CPU 14k DMIPS) Mem: 2 GB Shared (~250 MB available) GPU: VideoCore 5 (~195 MB Gfx memory available)

The equivalent processing power of the high-end STB is roughly of an iPhone SE released in 2016. Though the iPhones get a very powerful PowerVR GPU's compared to the embedded platforms making it harder to do a direct compare.

Software stack

Both devices are running an RDK software stack which is built on a custom Yocto embedded Linux distro using the 4.1 Linux kernel.

The browser used is an embedded Safari/WebKit: WPEWebKit version 2.22.

Test applications

To run the performance test we're running an IMDB application with a custom design. One application written in LightningJS and the other application in HTML5 using VueJS.

Test scenario

Both apps are loaded onto the STBs, wait until the splash screen ends and are ran using a pre-scripted set of key sequences.

The average/current Frame per Second (FPS) is captured using a requestAnimationFrame counter on screen.

At the end of the test sequence the average FPS is noted for comparison.

Each test sequence is recorded using a direct HDMI capture to avoid delays/processing of a TV using an elgato capture device.

Test 1 - Low-end STB

These are the results of the two applications on the low-end STB:

Video captures:

As you can see the frame drops of the HTML5 version on the Low-end STB are very frequent. Resulting into a degraded or "stuttering" experience of the UI. Whereas Lightning stays relatively stable through all the animations.

So even though the average FPS is pretty close, the HTML5 version frequently drops below 10 FPS. Especially when there is a lot happening at the same time.

Test 2 - High-end STB

These are the results on the high-end STB:

Video captures:

The high end STB shows a different picture. Where the HTML5 drops are still present it does seem to run a but smoother then on the Low-end STB. Overall it does struggle to get a higher average FPS. The Lightning version is rock stable in the high 50 FPS zone and rarely dips and the overall experience on the Lightning app is buttery smooth.

Test conclusion

In both tests the Lightning experience is much better. With less frequent dips the average FPS stays well above the threshold, keeping the FPS on both the low-end as well as the high-end as high as possible resulting in a smooth UI.

The impact of Lightning becomes more apparent on low-end devices where it prevents dips below 10-15 FPS, which is where users start noticing the frame drops and the experience comes across as "choppy" or "slow". On the high-end device the experience becomes really smooth and likely able to handle a lot more animations/effects for more added eye candy. However with every project you have to balance the target devices and the desired UX as a whole.

Overall the Lightning experience is a lot more pleasant and smooth, due to the higher average FPS and far less (to almost none) FPS dips during animations.