Single Board Computer Benchmarks

Contributors: Shawn Hymel

The Tests

For the majority of the benchmarks, we rely on a few tests within the Phoronix Test Suite (PTS), which is a free, open-source benchmarking tool for Linux (other operating systems are supported). The slick part about PTS is that it can automatically upload results to Open Benchmarking, which we will use liberally. Open Benchmarking is an online platform where users can store their results from PTS, publicly or privately. The power test is not part of PTS, so we will perform it separately.

Phoronix Test Suite

PTS will handle most of our tests. We first need to make sure the PHP 5 command interpreter is installed. On Debian systems, use:

sudo apt-get update
sudo apt-get install php5-cli

We can then download and install v5.8.1 of PTS:

git clone https://github.com/phoronix-test-suite/phoronix-test-suite/
cd phoronix-test-suite
git checkout v5.8.1
./install-sh

To run any of the tests, install it first using the install command with the name of the test followed by calling benchmark and the test name.

phoronix-test-suite install pts/<TEST>
phoronix-test-suite benchmark pts/<TEST>

The test can be selected from any of the profiles listed on the Open Benchmarking Test Profiles page. We will be using only a few of these tests.

For example, we can run the Himeno test on a BeagleBone Black Rev C (BBBrC). Once it completes, it will provide a text output of the results, which we can choose to upload to openbenchmarking.org. A link will appear in the console showing you where to find your results (http://openbenchmarking.org/result/1507305-BE-BBBRCDEMO13 in the example below).

CPU (Single Core)

The backbone of any computer is the ability to execute instructions and perform arithmetic. We will test simple number crunching abilities with the encode-flac test. This simple test times how long it takes the CPU to encode a sample WAV file into a FLAC format.

At this time, the default FLAC encoder only uses one thread, which means we are able to compare single core speeds.

Run the encode-flac test with:

phoronix-test-suite install pts/encode-flac
phoronix-test-suite benchmark pts/encode-flac

CPU (Multi-core)

While testing a single core is great for measuring raw speed, many newer computers (including SBCs) are created with multi-core processors. Having multiple cores allows a computer to run threads simultaneously. If a program is written to be multi-threaded, it can take advantage of more than one core and significantly speed up execution time.

We can measure multi-core performance using the Himeno benchmark. According to Open Benchmarking:

The Himeno benchmark is a linear solver of pressure Poisson using a point-Jacobi method.

If you are masochistically curious what the pressure Poisson equation is, see this.

Run the Himeno benchmark with:

phoronix-test-suite install pts/himeno
phoronix-test-suite benchmark pts/himeno

RAM

Another important feature of a computer is how fast it can read and write to random-access memory (RAM). RAM is great for storing short-term or volatile data. How fast we access that is another useful indicator of system performance.

For RAM, we will use the RAMspeed benchmark. This test is capable of running with a variety of parameters, but we will use the simplest one: copying a bunch of integers to and from RAM. This should hopefully give us an idea of the combined read and write speed.

phoronix-test-suite install pts/ramspeed
phoronix-test-suite benchmark pts/ramspeed

Flash Memory

RAM is great for storing data that requires shorter access times, but hard disk drives (HDDs) allow for non-volatile storage (i.e. we don't lose the data when power is removed). Most SBCs do not use a true "hard disk" but rely on SD cards or onboard eMMC for non-volatile storage, which we will refer to as "flash memory."

Like RAM, we can test the performance of the file system and flash memory. The IOzone test reads or writes large amounts of data to the non-volatile memory. We will test both read and write as separate tests using 1 MB blocks and a 2 GB test file (we want the test file to be larger than RAM so our operating system does not cache it in RAM).

phoronix-test-suite install pts/iozone
phoronix-test-suite benchmark pts/iozone

Graphics (2D)

As SBCs become more powerful, they are seeing use as home theater PCs and even low-end gaming machines. Many SBCs come with a version of OpenGL or OpenGL ES to handle graphics.

While we were unable to find a good 3D benchmark that ran well on the SBCs, we can test the computer's ability to draw simple 2D shapes. We used GtkPerf found within PTS to accomplish this. Specifically, we use the circles drawing test, which times how long it takes to draw a set number (in the thousands) of filled circles in a window. This fairly quick test allows us to compare the relative 2D graphics speed of SBCs.

phoronix-test-suite install pts/gtkperf
phoronix-test-suite benchmark pts/gtkperf

Power Test

Another important factor for SBCs is power consumption. Many users want to leave their SBC running continuously and want to know how much current the SBC will draw. How does it compare, for example, to leaving a light bulb on?

To test power consumption, we created a current logging device, lovingly named the "Current Log-O-Matic," using an mbed LPC1768, an INA169 Breakout, and a microSD Card breakout.

The schematic for the Current Log-O-Matic can be found below and the code can be found on mbed's site.

The Current Log-O-Matic, on boot, records 10 seconds of measured current use (flowing through the INA169 breakout's sense resistor) in about 20 ms increments. This information is put into a spreadsheet and the average current over those 10 seconds is computed.

We wanted to compute the idle and loaded power consumption. To do that, we booted the SBC under test and logged in to the console using a serial terminal. We disabled any extraneous features such as the GUI (e.g. X Windows), WiFi, and Bluetooth. While idle (doing nothing in the console), we took a voltage measurement at the input of the SBC (we want to test the power usage of the whole board) and ran the Current Log-O-Matic. We computed the average idle power usage with P = V x I using the measured voltage (with the assumption that the voltage did not vary much) and average current.

We repeated the test while the SBC was running the Himeno multi-core CPU test to measure the average current usage under load. The average power consumption was also computed using P = V x I.