libGDX - Quality and Evolution

The quality behind software systems is an interesting topic to study, as it is a very broad concept. One important factor is, for example, the quality of documentation, which plays an important role in understanding the system. Also, the quality of software itself should be assured, which is done, for example, by writing test suites or by making use of continuous integration. In this post, we discuss the libGDX project in terms of quality and evolution.

First we discuss the documentation that libGDX provides. This is followed by a further look at its test suites. Afterwards, we consider the continuous integration of the project. Finally, an overall analysis is presented of the quality of the past and future of libGDX.

Quality of documentation

When using a framework such as libGDX, it is essential for the developer to get a deep understanding of how the framework is organized and how they should use it. libGDX has different types of documentation which are easy to find. On their website they have an entire page dedicated to referring the developer to the documentation1. libGDX’s documentation is spread over three different sources; each of these sources is targeted at the different users libGDX might have:

  1. The documentation section on their website1. Here they give short guides that explain to developers how to get started with libGDX; the topics include “Set Up a Dev Env”, “Generate a Project” and “Demos & Tutorials”. This is also the documentation a beginning developer will stumble onto first, since they might already have experience with using Github or reading Javadoc. This page also contains some guidelines regarding reporting issues and contributing.
  2. The Javadocs2. This documentation is provided at a programming level and annotates classes, methods and functions within Java itself. This documentation helps developers understand what certain functions do while they are working on code inside their project. Lots of feature-rich IDEs like Eclipse3 and JetBrains IDEA4 enable hints that help the user with understanding the description of functions or classes that they are using. Assuming that methods and classes are well-annotated with Javadoc, this can be a very useful tool in understanding and navigating the libGDX codebase.
  3. The documentation on the Github Wiki5. The Github Wiki contains documentation that goes into more technical depth compared to the documentation on libGDX’s website. libGDX themselves call this documentation their “Developer’s Guide”. Developers who want a more thorough understanding of libGDX and its various components can read up on it here, which is most useful to people that seek to contribute.

It is nice to note that the overall quality of documentation is good. The documenation for beginners on the website is quite extensive and easy to follow along with the wiki on the Github page, although it might lack some architectural choices. We also analyzed the code in terms of Javadoc. We have determined that a total of 34% of production code (which excludes extensions and tests) has Javadoc attached to them. This might seem like a low percentage, but we must keep in mind that this also includes classes that override methods or implement interfaces. However, we have found numerous sparsely documented classes; the choice of when a method contains Javadoc is sadly not a consistent one. When we attempted to add Javadoc ourselves as a contribution we were told that this was “overdocumentation”. This is a rule that libGDX follows that we respect, but might not entirely agree with.

Quality of testing

Each platform supported by libGDX has a dedicated folder for testing platform-specific code. The folder structure is shown below6.

Each of the platform-specific tests spin up their own “application” in which certain functionality is “tested”; they do not use an actual testing framework. Therefore this platform-specific code does not have actual test coverage, even though the folder is populated with many different classes and scenarios. In total the tests folder contains 828 files, some of the files are resources and others utility classes, while the majority of the content is used for scenarios. These are mainly used for manual testing the merge request7.

The cross-platform tests are structured differently:

These tests are actually tested run jUnit and thus a coverage can be retrieved of the files that make up the core code of libGDX. The following is the code covered of libGDX by test coverage.

It is clear from the test coverage that the core code of the project has not seen much testing. Only a staggering 45 of almost 1000 classes are located inside the test coverage. The actual number of classes that have a test implementation is lower, as some of these 45 classes are implemented once for an intregation test. The awful test coverage is likely a result of contributors that are not forced to fulfill any test coverage criteria for their implementation or changes to the project.

The positive effect of a poor test coverage is that it is extremely fast to run, as can be seen above. The libGDX project cares more about the platform-specific code than the core code. It also applies integration tests over unit tests. Which is an odd choice, as unit tests should serve as a foundation8.

The above is the testing of a project in the ideal case: unit tests form the base, integration tests should be less and manual testing should rarely be used and even avoided where possible. Not only does the project not follow the ideal case of the testing pyramid, it follows an anti-pattern of testing as seen below.

The anti-pattern fits the project perfectly. It relies heavily on manual testing and integration testing and very little on unit tests. This anti-pattern follows when unit testing is not embraced from the start of the project8. Which supports the claim that proper testing was not applied from the start.

Quality of continuous integration

The libGDX project has continuous integration in place. The pipeline that is used, is shown below.

One thing that becomes clear after reviewing the pipeline, is that the continous integration is mainly focused on releases of the framework. There is room for improvement in terms of quality checking.

One point that would be a good addition for the continuous integration process, is to run tests. This gives the opportunity to see if the codebase still functions as intended after the change.

Another possible addition is that of Checkstyle 9. This would be a good addition, in order to keep an eye on the general style of the codebase.

Analysis of the past and the future

We have analyzed the codebase of libGDX by inspecting the repository and calculating common code quality metrics over the code itself. This can give insights into the evolution of the codebase and its development process.

The analysis of the git repository indicates the development intensity for each file. We iterated over the commits from the master branch on March 16, 2021 (commit 95fc6c710) all the way to the first commit on March 6, 2010 (commit 69c97c511). For each file we recorded the number of lines added and removed and calculated the size of the changes relative to the length of the file at that commit. We also counted how often a file was referenced by a commit. For additions, we took the length of the file after the commit, while for removals we took the length before the commit. This caps the relative change to 100%.

Filename Times referenced
tests/gdx-tests/…/tests/utils/GdxTests.java 172
gdx/…/graphics/BitmapFont.java 150
tests/gdx-tests-lwjgl/…/tests/lwjgl/LwjglDebugStarter.java 135
gdx/…/graphics/g2d/SpriteBatch.java 128
gdx/…/scenes/scene2d/ui/TextField.java 127
gdx/…/scenes/scene2d/ui/ScrollPane.java 114
gdx-backend-android/…/backends/android/AndroidGraphics.java 112

As can be seen in the above table, the file that was commited to the most was GdxTests. Instead of using a common testing framework, libGDX has its own framework and any test classes that should be run are registered in this class. The list contains 210 test classes, with one disabled via a comment, which largely explains the 172 commits touching this file.

The next file is for a non-test class BitmapFont, referenced by 150 commits. Font handling is a large part of libGDX and there are still outstanding issues1213, so it is likely that more changes will follow.

Filename Cumul. rel. changes
gdx-backend-android/…/backends/android/AndroidGraphics.java 18.9
gdx-backend-android/…/backends/android/AndroidApplication.java 16.0
tests/gdx-tests-lwjgl/…/tests/lwjgl/LwjglDebugStarter.java 16.0
backends/gdx-backend-lwjgl/…/backends/lwjgl/LwjglInput.java 13.0
gdx-backend-lwjgl/…/backends/desktop/LwjglGraphics.java 12.6
gdx/…/graphics/BitmapFont.java 12.5
backends/gdx-backend-lwjgl/…/backends/lwjgl/LwjglApplication.java 12.5

The classes that saw the most intense changes, where commits often changed large portions of the file, are AndroidGraphics and AndroidApplication. This should come to no surprise, since Android is one of libGDX’s major target platform. Both files have, in total, many times more changed lines than they are long and can therefore be seen as the most intensely edited files in the project.

Although there is a sharp dropoff in intensity after these two files, files that come close are LwjglInput, LwjglGraphics and LwjglApplication, mirroring the patterns seen for the Android backend. It is safe to conclude that the Graphics and Application interfaces are the most active parts of libGDX and that they will continue to be so in the future.

A special case here is LwjglDebugStarter. The developers use this class to quickly start certain test cases and because many commits touch on different parts of libGDX, which test is started and with which parameters this is changes often. The file itself is rather short, but it changes often and on many lines, which gives it a high development intensity score.

Code quality

Code quality metrics were calculated over every git tag. We will look at a few key metrics.

In the first two years of development, there was a rapid increase in the number of .java files in the repository. After that, it started plateauing, which hints towards feature stability as no additional modules are added. With libGDX 1.0.0, released in April 2014, the codebase was cleaned up and demos were moved to separate repositories14, causing a noticable drop.

The test-to-production ratio is also climbing, but at a much faster rate. It should be noted that automatic detection of test classes is error-prone and the percentage shown here may not reflect the actual proportion (it is likely the ratio is greater). However, it shows that the project is moving towards more standard methods of testing, which are easier for the metrics software to detect.

The mean cyclomatic complexity of the project is fairly high. We took the average cyclomatic complexity of all methods and later some methods, depending on a lower bound. Because idiomatic Java uses many getter and setter methods, we tried to filter these out by only including methods from a certain complexity and up.

The global cyclomatic complexity is dropping, which is likely due to the addition of more getters and setters. When these are excluded, the overall complexity is still decreasing, as can be seen by the yellow, upside-down triangle “cycl. cxty >= 2” line. However, the larger complexities remain worryingly high, with methods surpassing 20 and one even reaching 170. For the extremely complex methods, with a complexity of at least 10, the median is at 14. These complexities may severely impact the testability and maintainability of these components.

The number of lines of code in each method is, on average, decreasing, likely also due to the growing numbers of getters and setters. Meanwhile, files are growing in size, caused by the same phenomenon as well as general additions to the codebase.

These metrics show that the codebase is fairly healthy, save for some complexity issues. It is growing at a moderate pace and maintenance is keeping up. Analysis of the intensity of development shows that there are few classes that form the core of the project. We also showed that the project is using strange testing methodologies, but also that this is improving.

  1. libGDX. (n.d.). Source & Documentation. Retrieved March 19, 2021, from https://libgdx.com/dev/ ↩︎

  2. libGDX. (n. d.). Javadoc Overview (libgdx API). Retrieved March 19, 2021, from https://libgdx.badlogicgames.com/ci/nightlies/docs/api/ ↩︎

  3. Eclipse Foundation. (n.d.). Eclipse homepage. Retrieved March 22, 2021, from https://www.eclipse.org/ ↩︎

  4. JetBrains. (n.d.). Intellij IDEA. Retrieved March 22, 2021, from https://www.jetbrains.com/idea/ ↩︎

  5. libGDX. (n. d.). Wiki Home. Retrieved March 19, 2021, from https://github.com/libgdx/libgdx/wiki ↩︎

  6. libGDX. (n.d.). Platform-specific test. Retrieved March 18, 2021, from https://github.com/libgdx/libgdx/tree/master/tests ↩︎

  7. libGDX. (n.d.). Contributing. Retrieved March 18, 2021, from https://libgdx.com/dev/contributing/ ↩︎

  8. A. Bowles. (2017). Willow Tree. Retrieved March 18, 2021, from https://willowtreeapps.com/ideas/test-early-test-often-automation-testing-with-the-test-pyramid ↩︎

  9. checkstyle – Checkstyle 8.41. (n.d.). Checkstyle. Retrieved March 22, 2021, from https://checkstyle.sourceforge.io/ ↩︎

  10. obigu. (2021). Fix #6454 (#6455). Retrieved March 22, 2021, from https://github.com/libgdx/libgdx/commit/95fc6c7 ↩︎

  11. (no author) (2011). Initial directory structure. Retrieved March 22, 2021, from https://github.com/libgdx/libgdx/commit/69c97c5 ↩︎

  12. geertjan-apptornado. (2016). BitmapFont draws out of bounds when first character has negative xoffset. Retrieved March 20, 2021, from https://github.com/libgdx/libgdx/issues/3755 ↩︎

  13. J. Piegsa. (2015). Problems with bitmap fonts with pixel identical glyphs. Retrieved March 20, 2021, from https://github.com/libgdx/libgdx/issues/ ↩︎

  14. M. Zechner. (2014). removed demos from main repo, jippie. Retrieved March 22, 2021, from https://github.com/libgdx/libgdx/commit/940fa1a22d4a63da86558cf4cd0614c35c25a039 ↩︎