Towards GUI latency benchmarking

I am a huge believer in latency as an important part of UI feel. I am also a believer in optimizing against what you can measure. In this regard I see a hole in the open-source ecosystem: We have no structured ways to measure GUI lag.


When I say lag I mean the time between you pressing a physical button on your keyboard/mouse, and a change becoming visible on screen. I worry that this measurement in particular is suffering on modern systems with modern techniques, and with no cheap and easy way to measure we can’t even discuss it.

Therefore, I was thrilled when NVIDIA announced the Reviewer Toolkit for Graphics Performance, because it included a tool (the LDAT) for measuring total system lag. From the pictures and videos it looked to be a pretty simple circuit inside a 3D printed enclosure. It wouldn’t be hard for NVIDIA to open the design up to the community for the benefit of everybody. Instead NVIDIA productized it in the form of a proprietary monitor for GAMERS.

On the positive side, that means I get to make it myself!


This is where I introduce my Holiday project: FrameTime

Frametime is exactly the tool I think I need to measure end-2-end latency on modern graphical systems. It emulates a keyboard to send input. Using a photodiode it then watches the monitor for a change in brightness. The time between those two events is defined as the lag.

FrameTime is free open-source software.

Terminal comparison

So what kind of data can we get out of this device? I’ve taken measurements of a few terminal emulators. All tests were taken by configuring the terminal emulator to a black background with a white cursor. The terminals were all placed at roughly the same spot on the monitor, with the FrameTime device placed such that the cursor would be measured when the button was pressed. The fonts were configured to be Monaco in size 100. The device ran the test 10000 times for each terminal emulator, I then trimmed the outer percent of each end to eliminate outliers.

title signal lag_min lag_delta rise_mean rise_stddev
xterm 18.876 136379.000 130050.000 65343.759 4494.641
alacritty 20.532 370491.000 133613.000 57126.701 6083.740
termite 18.242 277158.000 143515.000 67793.107 5793.508
urxvt 18.657 413547.000 131844.000 66218.334 5249.306

The signal column is the change in brightness. The other columns are in cpu cycles. To get that into seconds the value must be divided by 16MHz. The lag is the value defined earlier, and the rise is the time it takes the monitor to stabilize on the new brightness value.

I think it’s interesting to note that lag_delta is around 133Kcycles for all terminals. If we convert that values to seconds, 133000/16MHz = 8.3ms which happens to be the time between frames on my 120Hz monitor. I think that speaks to the accuracy.