VSCode - Quality and Evolution

The software quality of a codebase is very important for the longevity of a software system. The quality of the code, the testing process and the amount of technical debt all factor into the ease in which a system can evolve and be maintained. In this essay we will discuss the ways in which the team working on Visual Studio Code have handled the challenges of maintaining such a large codebase. We start by taking a look at the overall software quality processes of the project.

Overall software quality processes

Being one of Microsoft’s top priorities, software quality processes exist at multiple levels, starting from well-documented guidelines to automated testing and code reviews. The development’s process intrinsic structure facilitates high quality software by having monthly iterations and weekly milestones. The relative low-frequency iteration rate and thorough continuous integration pipeline assure the system’s functional suitability, performance efficiency, compatibility and usability 1. As an overview, the main software quality processes are:

  1. Well-documented guidelines for development & contributions
  2. Code reviews
  3. Automated & manual testing
  4. Continuous integration

The well-documented guidelines 2 consist of project management outlines (roadmap, iteration plan, release process etc.), contributions guidelines (code contribution specific, testing etc.), and documentation (API, extensions, etc.). Each of these components makes possible the production of quality software, as they provide more context & help to developers/contributors, which can adhere to VS Code’s specific software standards & requirements. For example, if one has a specific coding style, one must adapt to VS Code’s Coding Guidelines 3. In regards to code reviews, as software quality is directly influenced by code quality, when a PR is created, a developer is automatically assigned to it - it is his responsibility to review the code (commit) and, where appropriate, to merge. Automated and manual testing represent very important processes, as they asure software’s quality 4 - we will detail this more in the section which refers to the rigor of test processes. In the next section, we will present the continuous integration process.

Continuous Integration

To maintain the software quality laid out in the previous section, the team at Microsoft has set up GitHub Actions 5 workflows. In particular we will take a look at the two types of Continuous Integration [^ciyml] jobs and its triggers. The Continuous Integration script has two triggers, it starts jobs on pushing to a branch or when creating a pull request. Both of these only trigger when the main or a release/* branch is involved, however, since running the jobs can take a long time. Running checks on every iteration of a feature branch would be wasting resources. When the CI is triggered, it has multiple jobs to run. First of all it runs both integration and unit tests on three operating systems: Windows, Linux, and MacOS. Also part of the CI are the hygiene checks. This means running type checkers, compiling and bundling the code, checking dependencies and running lint checks, among others. The result of these checks is shown along with the commit on GitHub, or on the Pull Request pages where it is used as a safeguard to prevent “bad” code to be merged.

Rigor & role of test coverage

The role of test coverage is essential, as it not only provides the certainty that the software behaves as required, but also gives stakeholders information about the software’s quality 4. Thus, VS Code’s testing process rigor is of utmost importance. As mentioned earlier, the continuous integration tools make sure nothing is broken on a new commit. Therefore, the following tests are being run. Tests 6 that run on new commits:

  • Unit tests
  • Integration tests (API tests)
  • Smoke tests (automated UI tests)
  • UI tests (manual UI tests)

In order to help contributors execute the tests, for each (except UI tests), guidelines exist. Prior to creating a PR, it is recommended to create coverage tests, to increase the chances of a merge 7.

Hotspot components

Described below are some of the main hotspots of the Visual Studio Code codebase. This list was handpicked from a collection of files affected by the most commits over the years. The left number denotes the amount of commits that changed the respective file.

474     src/vs/code/electron-main/app.ts
        Singleton main application

344     src/vs/code/electron-main/main.ts
        Main VS code entry point can have multiple instances

493     src/vs/platform/windows/electron-main/window.ts
        Class describing a code window and its properties

349     src/vs/workbench/contrib/debug/common/debugModel.ts
        Contains the main class definitions of the debugger

345     src/vs/workbench/browser/layout.ts
        Describes the main layout class, layout of the editor subwindows

323     src/vs/workbench/services/textfile/common/textFileEditorModel.ts
        Handles when to save files based on text changes.

317     src/vs/platform/actions/common/actions.ts
        Defines a base Action class, classes which implement run

313     src/vs/editor/common/model/textModel.ts
        Defines a textModel class containing information of a text block

As can be seen from this list most of the hotspots include big parent classes and other building blocks for a lot of other parts of the codebase. It is not that strange that these modules have become hotspots as these are the foundation for many other files. The above files all are files containing code implementations, the files which have changed even more over the years are the files mostly describing javascript interfaces.

1123    src/vs/vscode.d.ts
1043    src/vs/monaco.d.ts
541     src/vs/editor/common/modes.ts
486     src/vs/editor/common/editorCommon.ts
441     src/vs/workbench/common/editor.ts
435     src/vs/workbench/contrib/debug/common/debug.ts

These interface files describe the API’s of the different components and as such every new exposed API definition would require a change in one of these files.

From the included extensions the git extension is by far the one which has had the most changes over the years. Some of the big hotspots within the git extension can be seen below.

606     extensions/git/src/commands.ts
336     extensions/git/src/repository.ts
327     extensions/git/src/git.ts

These files contain all the commands which are being used for git from the Visual Studio Code user interface.

As these files are all pillars of the code base we will probably keep seeing them change over the years. In some of these files clear divisions can be seen between parts of the defined code, as such it could be possible to split these up into separate modules at a later time, but in general the Visual Studio Code codebase is already split up a lot over all it’s different components.

The code quality is safeguarded through the use of code linters. This is also true for the hotspot components. They are only accepted when using correct coding practices, both automatically and through manual review in pull requests.

Quality Culture

The quality culture, as evidenced in actual discussions and tests taking place in architecturally significant feature and pull requests (identify and analyze at least 10 such issues and 10 such pull requests) In order to guarantee the quality of VS Code, the developers from microsoft are constantly reviewing and commenting on feature requests and code contributions made by the community. To determine how successful this system is, and how much technical debt it avoids, we will look at several recent issues and pull requests that accompany them. Some of which eventually got merged and some of which did not. The first example is of an issue presented by a community member who proceeded to fix it himself. Apparently, when installing VS Code on a Raspberry Pi, some post install script would add the whole repository even though that was not required. On the pull request for this issue, two of the Microsoft developers had a small discussion. One of them added a few minor “lipstick” changes and asked the other to review those. After this, the original author had to sign the CLA8 and this PR was merged. The next issue concerns an API-proposal by a member of the community. This issue is interesting because it started quite a lengthy interaction chain between the developers. Before any actual implementations were presented, the discussion goes through several stages.

  1. The issue’s author presents the current problem
  2. The issue’s author proposes a solution.
  3. The developers evaluate the sensibility of the solution and debate whether it is beneficial to general consumers.
  4. With the help of the Microsoft developers, the issue’s author updates her original proposal. After this, the API proposal undergoes one more revision until developers start on its implementation.

In order to identify the quality culture, it is also interesting to consider pull requests that got rejected and the reasons for that. This PR targeted at this issue did not make it into the main branch. The problem is that SVG files won’t show up in the webview because the resource gets blocked. The pull request mentioned above proposes a workaround for this problem. However, one of the Microsoft developers kindly closes the PR with the remark that the provided solution goes against one of the design decisions. This shows that not all working solutions are blindly accepted.

In the list below we present some more examples of issues and pull requests with a brief analysis on their status:

PR #117085:

  • Corresponding issue: #115933
  • Problem description: Very long URLs cause the width of a dialog box to overflow
  • PR status: Closed, the proposed solution involved adding new line characters to the url which was deemed insufficient and caused problems with copying. The developers decided to wait for a better CSS solution.

PR #119253:

  • Corresponding issue: #118824
  • Problem description: Adding support for a new command line argument.
  • PR status: Merged, a Microsoft developer left a comment on the issue to help the issue author on his way. After a small discussion about the naming scheme, the PR was merged by the same developer.

PR #118976

  • Corresponding issue: #117596
  • Problem description: An error should throw for an invalid bound check in Splitview.
  • PR status: Closed, the presented solution was deemed incomplete. The author added a single bound check, but the Microsoft developer states that similar checks are required in more places. He proceeds to explain what types of issues would be a better place to start contributing.

Issue #116540

  • Problem description: Support needs to be added for the JSON $ref node to resolve external file references.
  • Status: In progress. This is an issue that our team is currently working one. Our personal experience is that the senior developers are quite willing to answer our questions, as evident from the comments on the issue. However, in this case they did not provide us with any insights or guidance on their own, something we did observe on some other issues. The change requires us to change the existing API of the JSON language service 9.

Assessment of Technical Debt

Out of approximately 4700 open issues, 157 are labeled as technical debt. These issues mostly involve more intricate code changes and are often handled by developers from Microsoft rather than the community. The debt label is used for simple problems, such as an ambiguous naming scheme, but also for more difficult tasks like changes in the dependency hierarchy of some class or to fix “dirty” behaviour with IO-operations. Regarding pull requests, those labeled technical debt get merged about once every ten days. By looking at these numbers in the repository, we could easily conclude that the current state of VS Code is quite healthy and that the amount of technical debt is very reasonable. For a more objective view on the matter, we decided to run the codebase through SonarQube’s 10 analysis tool. The results are displayed in the figure below.

Figure: SonarQube Analysis

The first number tells us that a total of 329 man-days are required to fix all the technical debt. This sounds like an awful lot, but as visible on the right, SonarQube still assigns an A rating to this score. Provided how large the VS Code project is, this technical debt only amounts to 0.8% (another statistic from the analysis) of the complete code base. Combining both of the presented views, we can conclude that the levels of technical debt in the VS Code project are perfectly acceptable.

References

  1. ISO/IEC 25010, Software Product Quality. https://iso25000.com/index.php/en/iso-25000-standards/iso-25010?start=0 ↩︎

  2. VS Code, Wiki. https://github.com/microsoft/vscode/wiki ↩︎

  3. VS Code, Coding Guidelines. https://github.com/microsoft/vscode/wiki/Coding-Guidelines ↩︎

  4. Software testing. https://en.wikipedia.org/wiki/Software_testing ↩︎

  5. GitHub Actions. https://docs.github.com/en/actions ↩︎

  6. VS Code tests. https://github.com/Microsoft/vscode/blob/main/test/README.md ↩︎

  7. How to contribute, Pull Requests. https://github.com/microsoft/vscode/wiki/How-to-Contribute#pull-requests ↩︎

  8. Contribution License Agreement. https://cla.opensource.microsoft.com/microsoft/vscode ↩︎

  9. VS Code JSON Languageservice. https://github.com/microsoft/vscode-json-languageservice ↩︎

  10. SonarQube. https://www.sonarqube.org/ ↩︎