ThingsBoard - Architecture

From Vision to Architecture

Previously, we introduced ThingsBoard1, an open source IoT platform on which you can read out and interact with all your IoT devices. We did this by describing its goals and the context in which it operates2. In this post, we will talk more about ThingsBoard’s underlying system architecture.

The ThingsBoard application is built following a layered, service-oriented architecture, according to the standards for Java EE applications3

A typical layered architecture contains a presentation, business, persistence, and database layer3. These layers could be traced in the ThingsBoard application according to the Web UI, Rule Engine, Core and Transports, and database. As an IoT platform, ThingsBoard’s main focus is on developing the business and persistence layers. We can also identify multiple design patterns4 in the system architecture, such as Model-View-Controller (MVC)5, Representational state transfer (REST)6, specification pattern7, and microservices8. All of this will be explored further in the remaining sections.

Containers & Components

Figure: ThingsBoard Architecture

For a good understanding of the system’s architecture, we first present a high-level schematic overview of the ThingsBoard architecture as depicted in ThingsBoard’s documentation9. The containers with a white background are out of scope, as they are not controlled by ThingsBoard directly. The containers with a blue background represent ThingsBoard components which are decoupled and deployed separately. The four main components are described in the following table.

Containers Description
ThingsBoard Web UI1011 This is where the end user interacts with the system. This is a two-way street: the user uses the interface to extract data and gain insights into the connected devices, and can also feed new data or commands into the system.
ThingsBoard Transports12 The container forms the bridge between physical IoT devices in the network and the application’s internal state. This container can be strictly divided into separate subcomponents, one for each supported transport API and is easily extensible to support more APIs. This container is crucial for a solid end product because it is incharge of stable, reliable connections.
Rule Engine13 This container contains all user-defined logic. It allows to filter, enrich and transform incoming messages originated by IoT devices. Then, users can define behaviour following these messages. Additionally, users can connect to external services via the Rule Engine. For users, the rule engine is easily and visually modifiable in the Web UI.
ThingsBoard Core14 This container connects to all others. It is responsible for REST API calls, WebSocket subscriptions15, attribute changes, and monitoring device connectivity states16. In short, this container acts as the main backend and source of business logic.

Moving on, we will dive a bit deeper into the components that make up the containers and we will also explain how they are connected.

Thingsboard Web UI components

The ThingsBoard Web UI is a single-page web application and it consists of three main parts: the authentication module, home module, and dashboard module.

  • The authentication module is responsible for authenticating the login credentials and setting new passwords.
  • The home module consists of user menus and toggles.
  • The dashboard module consists of widgets and graphs.

The Web UI uses a Websocket API which duplicates the functionality of REST API so that users can make changes to the rules and subscribe to the device’s data changes.

Figure: Diagram showing the Web UI component

ThingsBoard Transports components

The Transports container includes 3 components: HTTP, MQTT, and CoAP servers. The servers connect the devices(clients) on one side, to the ThingsBoard Core on the other side. The communication between the devices and servers is done using the respective protocol and the communication between the servers and the ThingsBoard Core is done by queues using Apache Kafka.17 ThingBoard also provides an API for MQTT gateways. An MQTT gateway allows the group of several MQTT devices to one device and connects it to the MQTT server/broker. Furthermore, loadbalancing can be done in case multiple servers are used, for which HAProxy is recommended.

Figure: Diagram showing the Transport component

Rule Engine components

The ThingsBoard Rule Engine is a configurable system for processing incoming messages from devices. The basic unit of ThingsBoard Rule Engine is a Rule node. A Rule node processes one incoming message at a time and produces one or more outputs. There are 5 different types of Rule nodes: filter nodes, enrichment nodes, transformation nodes, action nodes, and external nodes. The filter nodes are used for filtering and routing incoming messages. The enrichment nodes are used for updating the meta-data of the incoming messages. The transformation nodes are used to change the format of incoming messages. The action nodes are used to trigger actions based on the incoming messages. And the external nodes are used to interact with external systems. In addition to these, users can also build their Rule nodes if required.

Figure: Diagram showing the Rule Engine Component

The Rule Engine uses an actor model consisting of Rule Chain actor and Rule Node actor.18 The Rule Node actor can be a custom node or one of the previously mentioned node types and is responsible for the processing of the incoming messages. The Rule Chain actor is a logical group of Rule nodes and their relations as configured by the user and is responsible for rule node configuration, routing messages between rule nodes, and handling queue commands.

ThingsBoard Core components

The ThingsBoard core is responsible for handling REST API calls, Websocket subscriptions, processing messages via Rule Engine, and monitoring device connectivity states. Several identical ThingsBoard core nodes can make up a ThingsBoard cluster. And the processing load is distributed between these core nodes. HAProxy is used for load balancing and Zookeeper19 is used for service discovery. Messages between the Core nodes are passed through queues using Kafka. Furthermore, ThingsBoard uses Redis for caching assets, entity views, devices, device credentials, device sessions, and entity relations.

Figure: Diagram showing the Core component

Development view

Thingsboard consists of several main components, as discussed in the previous chapter. These components interact and share logic. From a development point of view, it is best practice to split these components up3 as much as possible, so that when one thing needs to change in component A, it doesn’t require a change in components B or C. In ThingsBoard, the four main components are thus split from each other. The UI is split from the core logic, but also the Rule Engine and the transport layer have their separate logic. From a development perspective, all of these components reside in the same repository on Github, with each of the components having its folder in the repository.20

The logic which is used throughout the components is split off from the rest of the code, residing in its folder named “common”.21 Creating such a common folder is, well, a common practice in that it removes the need to duplicate code. The pitfall of having duplicate code is that once someone makes a change in location A, it never gets updated in location B, resulting in bugs of which the developer thought they had fixed it.22 23 24

The rest of the components' logic is split from each other as much as possible and only communicates via their respective APIs.

Relation between architecture and quality attributes

ThingsBoard has its logic split into several components, which improves individual scalability. For example, if there are many messages sent, while the front-end is only visited once a year one can choose to scale the Transport component, without affecting the (cost of the) UI-hosting. However, splitting the components has a negative side too. Each of the individual components now needs to have a certain boilerplate code to run.

This boilerplate code both needs to be stored, as well as maintained, resulting in an increased cost. When designing such a system, you have to take into account if the profits gained from this splitting exceed the time- and storage setbacks during development. For a large system such as ThingsBoard, this is an easy choice to make; the development of the system is done once, but the individual scaling can happen on all the systems running the code. The profit gained from having the individual scalability on each machine thus greatly exceeds the one-time setback. A more detailed analysis of the development view and the codebase is investigated in the quality and evaluation essay25.

Runtime interaction

The four key components during the runtime are: Netty26, Cassandra27, Actor System28 and Kafka. The IoT devices are connected to the ThingsBoard server via MQTT and ‘publish’ ‘commands with a JSON payload are issued with a size of roughly 100 bytes. MQTT is an efficient publish/subscribe messaging protocol which offers many advantages over HTTP request/response protocol.

ThingsBoard supports on-premise and cloud deployments and has currently more than 5000 servers running all over the world. These servers include AWS, Azure, GCE, and private data centers.

Figure: ThingsBoard Data Flow

A ThingsBoard server processes the MQTT requests from the devices, these are then handled by Netty and Akka and subsequently stored to Cassandra asynchronously. The server can also push data to the WebSocket subscriptions if needed. ThingsBoard is supporting MQTT QoS level 1, meaning that the client receives a response only after the data is stored in the Cassandra database. It makes use of multiple message queue implementations such as Kafka, RabibitMQ, AWS SQS, etc. These queues allow ThingsBoard to implement both back-pressure and load balancing in case of peak loads. Further, special attention is paid to the correct implementation of the asynchronous Cassandra driver API.29

Server-side API and design principles

ThingsBoard exposes a REST API that can be explored using Swagger UI.30 We will analyze the API through the following principles: easy to understand, defensive API, meaningful error messages, and compatibility.

Easy to understand

The API structure is well organized with independent controllers and consistent naming. Further, the API endpoints follow the general guidelines of the naming conventions31. This makes the API predictable for the developer. Several good practices used by ThingsBoard are: nouns to represent resources, pluralized resources, query parameters where necessary and punctuation for lists, etc. Moreover, ThingsBoard uses CamelCase. Although this is still clearly readable, it is not in compliance with the best practices.

Defensive API

To ensure a defensive system and protect each API call, ThingsBoard uses JSON Web Token (JWT)32. JWT is a well-established standard method (RFC 751933) to claim secure communication between two parties. Additionally, JWT incorporates different encryption mechanisms for token communication available in libraries for different programming languages.

Meaningful error messages

The basic part of the error responses is characterized by the meaningful returned codes. Further, additional information could be included in the response body of the error messages. According to the default spring error responses34, this should be the title, the message, the timestamp, and the URL path where the error occurred. In the exception handler, ThingsBoard defines the standard HTTP statuses according to the spring framework, a short error message, an extra error code according to the ThingsBoard specification, and the timestamp in full date format. Even if the path is absent in the error response, ThingsBoard logs the error trace which allows a full error debugging. However, in case of multiple errors, more information could be helpful. The best practices for REST API Error handling recommends more information to be included. Standardized response error bodies could include the type, title, status, detail, and instance35.

Compatibility

In terms of API design compatibility36, old clients should be able to operate with the same ThingsBoard API version and when they want to update the API version, they should be able to do so easily. The oldest version which is still supported till May 1st, 2021 is v2.5, with the newest one v3.0.1 37. However, the jump from v.2.5.6 to v3.0, introduced major changes. The changes involved a frontend rewrite from AngularJS 1.5.8 to Angular 9. This establishes a new JS framework where the custom widgets and actions will have to be consequently adjusted individually by clients. Further, ThingsBoard migrated from Cassandra to Hybrid DB Approach. Meaning that clients using the pure Cassandra database will have also to install the PostgresSQL database.

tags: Thingsboard DESOSA Software Architecture TUD Delft University of Technology

  1. https://thingsboard.io/ ↩︎

  2. https://2021.desosa.nl/projects/thingsboard/posts/1.-product-vision/ ↩︎

  3. https://www.oreilly.com/library/view/software-architecture-patterns/9781491971437/ch01.html ↩︎

  4. https://leanpub.com/software-architecture/ ↩︎

  5. https://en.wikipedia.org/wiki/Model%E2%80%93view%E2%80%93controller ↩︎

  6. https://en.wikipedia.org/wiki/Representational_state_transfer ↩︎

  7. https://en.wikipedia.org/wiki/Specification_pattern ↩︎

  8. https://microservices.io/ ↩︎

  9. https://thingsboard.io/docs/reference/monolithic/ ↩︎

  10. https://www.youtube.com/watch?v=L_geyNzS7tM ↩︎

  11. https://thingsboard.io/docs/reference/msa/#web-ui-microservices ↩︎

  12. https://thingsboard.io/docs/reference/msa/#transport-microservices ↩︎

  13. https://thingsboard.io/docs/user-guide/rule-engine-2-0/re-getting-started/ ↩︎

  14. https://thingsboard.io/docs/reference/msa/#thingsboard-node ↩︎

  15. https://thingsboard.io/docs/user-guide/telemetry/#websocket-api ↩︎

  16. https://thingsboard.io/docs/user-guide/device-connectivity-status/ ↩︎

  17. https://kafka.apache.org/ ↩︎

  18. https://thingsboard.io/docs/user-guide/rule-engine-2-0/re-getting-started/ ↩︎

  19. https://zookeeper.apache.org/ ↩︎

  20. https://github.com/thingsboard/thingsboard ↩︎

  21. https://github.com/thingsboard/thingsboard/tree/master/common ↩︎

  22. https://medium.com/thinkster-io/code-smell-duplicated-code-c82590858057 ↩︎

  23. https://www.informit.com/articles/article.aspx?p=457502&seqNum=5 ↩︎

  24. https://itsm.tools/using-code-smell-for-better-software-development/ ↩︎

  25. https://2021.desosa.nl/projects/thingsboard/posts/3.-quality/ ↩︎

  26. https://netty.io/ ↩︎

  27. https://cassandra.apache.org/ ↩︎

  28. https://en.wikipedia.org/wiki/Actor_model ↩︎

  29. https://thingsboard.io/docs/reference/ ↩︎

  30. https://thingsboard.io/docs/reference/rest-api/ ↩︎

  31. https://www.oracle.com/java/technologies/javase/codeconventions-namingconventions.html ↩︎

  32. https://jwt.io/ ↩︎

  33. https://tools.ietf.org/html/rfc7519 ↩︎

  34. https://www.baeldung.com/exception-handling-for-rest-with-spring ↩︎

  35. https://www.baeldung.com/rest-api-error-handling-best-practices ↩︎

  36. https://cloud.google.com/apis/design/compatibility ↩︎

  37. https://thingsboard.io/docs/reference/releases/ ↩︎