openHAB: Quality and Evolution

Table of Contents

1. Software Quality Processes
2. Continuous Integration Processes
3. Test Processes
4. Hotspot Components
5. Quality Culture
6. Technical Debt

Software Quality Processes

The health care system is full of fixed processes that are in place to ensure patient safety. A patient is tested for a penicillin allergy before taking the antibiotic. A pre-anaesthesia checkup is in place also to test the sedation before an operation. Similarly, systems have software quality (assurance) processes to ensure that the system meets specific quality objectives before releasing the code into the world. Within openHAB, the following process is used to ensure software quality.

Any developer should adhere to the coding guidelines, which apply to all Java code inside the project and aim for easy readability and maintainability. These guidelines are used as a checklist for code reviews on a pull request. Some of the topics are the structure of a typical OSGi bundle project, code format for Java and XML files, JavaDoc, and logging usage patterns.

openHAB has devoted a repository¹ to static code analysis using Maven plugins for FindBugs, Checkstyle and PMD (sometimes referred to as ‘Programming Mistake Detector’). This tool tweaked to openHAB’s needs. FindBugs detects and ranks possible bugs. Checkstyle creates a report regarding the code style. PMD finds frequent flaws such unused variables. This tool is available in a separate repository that generates an HTML report.

openHAB has a Jenkins server that is used for Continuous Integration. It is used to make sure all repositories are able to be built and pass all the necessary tests². The next section goes more into detail.

openHAB uses jFrog to optimize processes within the CI pipeline and stores the binary output of the build data. In addition, jFrog provides automation and standardization across the repositories which can lead to a faster workflow.

The community highly appreciates pull requests, but also has a process in place to ensure quality. Changes can only be merged after an approval by one of the openHAB maintainers³. Maintainers focus on a specific code repositories and every repository has multiple maintainers. Apart from reviewing PRs they also define the code guidelines previously mentioned, decide the code’s architecture and features, and more. To become a maintainer, one should be heavily involved, be nominated by another maintainer, and receive at least three positive votes in the election⁴.

Continuous Integration Processes

Most of openHAB’s development and deployment processes are integrated into the GitHub organization, using GitHub Actions. As most openHAB’s containers have their own repository, they all have their own Continuous Integration Processes, even though most of their integrations are shared.

openHAB’s primary development integrations are based on Jenkins⁵, an open-source automation server. The Jenkins instance is publicly available ⁶. Each container has its own Jenkins project. Here the instance lints, builds, and tests the complete openHAB source on a schedule and whenever a commit or merge request has been pushed. On top of Jenkins, openHAB uses relativeCI for quality control of their web bundles.

Besides the primary CI, openHAB uses Crowdin⁷ for their internationalization process. Crowdin has automated this process and automatically creates merge requests using the openHAB-Bot, whenever language updates are available.

Test Processes

openHAB requires code to be well tested and has several test processes where they apply Unit tests and integration tests⁸. The Unit tests are applied to test individual components, while the integration tests test the combination of multiple software modules. For both the core system (openhab-core) and add-ons (openhab-addons), the tests are stored in a folder called itests. Writing tests for every line of code is not mandatory in openHAB, but is highly encouraged.

Mockito, an open source mocking framework, is used for the Unit tests⁹. openHAB’s documentation gives an example of how this can be applied⁸:

public class MyBindingHandlerTest {

    private ThingHandler handler;

    private @Mock ThingHandlerCallback callback;
    private @Mock Thing thing;

    @Before
    public void setUp() {
        initMocks(this);
        handler = new MyBindingHandler(thing);
        handler.setCallback(callback);
    }

    @After
    public void tearDown() {
        // Free any resources, like open database connections, files etc.
        handler.dispose();
    }

    @Test
    public void initializeShouldCallTheCallback() {
        // we expect the handler#initialize method to call the callback during execution and
        // pass it the thing and a ThingStatusInfo object containing the ThingStatus of the thing.
        handler.initialize();

        // verify the interaction with the callback.
        // Check that the ThingStatusInfo given as second parameter to the callback was build with the ONLINE status:
        verify(callback).statusUpdated(eq(thing), argThat(arg -> arg.getStatus().equals(ThingStatus.ONLINE)));
    }

}

Hamcrest assertions can be used to assist in writing Unit tests. It allows the tester to quickly create complex assertion matchers, instead of being stuck to assert(x == y); for instance. A few examples of Hamcrest assertions are stated below.

assertThat("myString", is(instanceOf(String.class)));
assertThat(Arrays.asList("one", "two"), hasItem("two"));
assertThat(new Object(), is(not(nullValue())));

Some components use the OSGi core services, which makes it hard to test them. Luckily, by using integration tests test classes which extend JavaOSGiTest can be run inside an OSGi runtime. Note that these test have a long run time, which should be taken into account.

Hotspot Components

Hotspot components are components that are often changed. These types of components are usually the most critical part of the system. Since openHAB does not have any known roadmap, we, unfortunately, cannot say much about the future’s hotspot components. We can, however, say something about which components have been the hotspot components in the past.

The two most active repositories for openHAB are openhab-core and openhab-addons. We used git effort to find out what are the most actively developed components. This command shows the number of commits and days worked on for all files in the repository. These are the topmost actively developed components for both repositories..

openhab-core

openhab-addons

For this repository we take a look at which bindings as a whole received the most commits, since that give a better representation of their importance than separate files give.

Quality Culture

Most changes that come to openHAB start out as an issue. Creators of such issues can choose to add labels to their issue to identify them easier. In the repositories openhab-addons and openhab-webui, the issues are clearly labeled. However, in the openhab-core and openhab-distro many do not go for this approach, and as a result the list of issues is a list of titles of the problem they would like to have fixed, see the issues page.

For a pull request (PR), all repositories use the same policies. A (future) contributor is expected to clearly explain the context of their contribution and sign off their work in their commit messages. Testing your code is encouraged, however we see that many pull requests do not test their pull requests (PR#2133, PR#1814).

A pull request can only be merged when maintainers of openHAB have accepted the pull request. They do this using LGTM (Looks Good To Me) in the comments of the merge request. Dependent on the repository the maintainers might differ, there needs to be an accept from the absolute majority of maintainers on both repositories.

Most issues identify a problem someone has experienced when using openHAB. These issues get explored by contributors and then either get resolved (issue#2216) or eventually converted to a pull request to fix the issue (issue#2200 to PR#2212). Some of the issues address test cases that fail on certain systems or are unstable (issue#1955, issue#1069). What stands out is that these issues are all posted by top contributors and resolved within a few weeks(PR#2093,PR#1148).

Very few PRs actually have in improvement in the code quality or optimization (PR#1068) and are mostly fixated on additions to the system , bug fixes or updating dependencies (PR#1216 PR#1242, PR#1245).

The main focus for most contributors is through the openhab-addons repository. This is the place where extra bindings to systems are implemented to allow for the widest range of connected components allowed with openHAB. The issues and PRs in this repository add new bindings, fix an issue with bindings or add functionality to a binding (bindings are the connection with the system and IoT devices) (PR #10277, #10355, #10302).

Technical Debt

Technical Debt is a popular concept which came into being when Ward Cunningham used it as a metaphor to explain motivations for refactoring in 1992¹⁰. The term covers the costs of additional work in future when choosing a faster and cheaper short-term solution now. The image below perfectly illustrates this¹¹. Although technical debt is hard to quantify, this section considers multiple aspects of it to at least get an idea of the current status.

Bus Factor

Unlike the term technical debt which came from a financial metaphor, the term bus factor is a bit more to the point. It represents the number of people who would have to drop out, or more subtly ‘be run over by a bus’, before the project would be in hazard.

To analyze this factor, we take a look at core system and leave out the rest including applications and add-ons since these are less vital. The following graph shows the contributors who have had over 20 commits in total between 2016 and present day (March 22, 2021).

As we can see, there are just seven contributions who meet this definition of active. Some of these contributors have only been active in a particular year. Especially between 2016 and 2018 openHAB’s founder Kai Kreuzer was the only active contributor on the openhab-core repository. After that it improved slightly, however, the number of active contributors is still low. Looking at 2019 and 2020, there are just four contributors with over 20 commits within that year. Therefore, the bus factor is four for the core system.

Code Quality Analysis

As mentioned before, openHAB also comes with a static code analysis repository¹. This generates a report of possible bugs in the code of the openhab-core.

Reducing Technical Debt: 1.x Compatibility Layer

By releasing openHAB 3.0 much technical debt that originated from the second version was paid. openHAB 1 was just a clean and simple architecture. The second version become more complicated as comprises were introduced between ‘old’ first version and the new UI features from Eclipse SmartHome¹². Bindings and rules worked quite differently in both approaches, but they did have to work together. As a result a 1.x compatibility layer was introduced¹³. openHAB 3.0 removed this unnecessary layer all at once¹⁴.

References