The basics of balenaCloud device logging
Devices managed by balenaCloud have a number of options when it comes to logging. The easiest and most straightforward logging for a device is via the Logs Terminal in the balenaCloud dashboard.
Here you can quickly view anything written to stdout and stderr of your application services on the device. You can also filter these logs, hide/show timestamps and toggle the timestamp in UTC.
However, as convenient as this log window is, it has some limitations:
- The maximum limit of logs displayed in the dashboard is 1,000 lines.
- If the device is generating too many log rows too quickly, some rows may be suppressed.
These limitations are by design: to quote from our documentation concerning device logging: “Device logging in balenaCloud isn’t meant for long-term, reliable storage of logs.” It is designed as a debugging feature for balena devices.
Checking the journal logs
The logs from different services, including the Supervisor and others, can be found using the journalctl utility. Journalctl is a command-line utility used to query and filter log messages from the systemd journal, the central logging service used in modern Linux that uses systemd as its init system.
To access these logs on the device, you can ssh into the device via the web terminal window in balenaCloud, balena device ssh from the balenaCLI, or by using a standalone SSH client.
You can find examples of querying the journal logs using journalctl in our Debugging Masterclass.
To give your container read-only access to the host’s journal log directories, you can use the io.balena.features.journal-logs docker compose label in your docker-compose.yml file.
Persistent Logging
Since journal logs in balenaOS are cleared on a reboot, balena provides a persistent logging feature which can be enabled on a device’s balenaCloud dashboard. Once persistent logging is enabled, the logs are stored as part of the data partition on the device with a default size of 32MB. On some versions of balenaOS this size can be increased, however it is not usually recommended due to the excessive wear it can inflict on a device’s storage medium (especially SD cards.)
Programmatic access to logs
For more versatile and unfettered access to device logs, you can use one of the following two APIs to stream, filter, parse, and/or ship your logs:
- The balena API: This API is the core of the balena platform and is used by the balenaCloud dashboard and CLI. You can use it to subscribe to a device’s logs and stream them to wherever you need to. However, note that it has the same restrictions as the dashboard log terminal: log lines can be suppressed if they are too numerous and/or too fast, or if the client can’t read them fast enough. Typically a client will utilize the API’s Node or Python SDKs to subscribe to logs.
- The Supervisor API: The Supervisor is balena’s agent that runs on devices to ensure your app is running and to communicate with the balenaCloud API server. The Supervisor itself has its own set of APIs that include the ability to retrieve a stream of the journald logs on the device. (This is equivalent to running journalctl –no-pager.)
Unlike the balena API/SDKs, this on-device service is not rate limited. With this option, you can save the logs on the device or send them to a third party for log storage/analysis. Note that the Supervisor is not designed as a high-availability service, meaning that among other things it can restart without notice. Normally this won’t have much effect on a device unless you are using it to stream critical logs where you can’t miss even one line. In that case, use the feature label mentioned below.
io.balena.features.journal-logs feature label
As mentioned previously, this label, when added to your docker-compose file, will bind mount the host’s journal log folders into your container as described on this page. At that point, your container will have journalctl access to all of the device’s logs that you can do with as you please.
This example project takes logs from the journal (courtesy of the feature label) and outputs (mirrors) them to the container stdout/stderr. Because the Supervisor can read container logs, that means those logs also get sent to the backend.
Using this label provides the most flexibility and robustness when working with logs on a balena device.
Third-party alternatives
For a simple example of how to use the Supervisor API to gather logs and either save them on the device or send them to the Loggly cloud service, check out this example.
We also have a more comprehensive example of capturing journal logs on a device via an OpenTelemetry Collector and sending it to a Loki backend. (In addition to device monitoring metrics sent to a Prometheus backend.)
An older but still informative example exists that demonstrates telemetry and logging data being sent from a device to Datadog for rendering in their cloud-based dashboard. It uses the Datadog IoT agent which lets you select which telemetry, logs, and information you want to collect, and then balance that with your device’s bandwidth and capabilities.
Logging out
There are many options for inspecting the logs on your balena device. For quick and dirty troubleshooting, the web terminal in balenaCloud can be an invaluable resource. We’ve also outlined some more comprehensive options for reading, saving, or shipping your logs as well.
Do you have any suggestions or additional methods for logging on balena devices? Do you have a shortcut or handy tip that may benefit our readers? If so, you can reach out to us via balena forums, paid support or by adding a comment below.
The post All About Logging on balenaOS appeared first on balena Blog.