Restart and Watchdog

One of the advantages of using systemd is that you can control how your program is automatically monitored and restarted if there is a problem. Our sample unit file has two parameters relating to restarting in the [Service] section:

Restart=on-failure
RestartSec=10

This means that our program will only be auto-restarted if it exits with a failure code, i.e. any non-zero value. The number of seconds specifies how long systemd will wait before restarting and the default is 100ms. If you want your program to be restarted if it exits for any reason then you can use:

Restart=always

If you make this change to the example unit file and restart it with the new settings:

sudo systemctl daemon-reload
sudo systemctl start myService

then a status check will confirm that it is running. You can find the service’s process identification number (PID) from the status listing - the number in square brackets:

Mar 24 11:08:29 MyServer myservice[11419]: Hello systemd world

You can now use this to kill the service:

sudo kill 11419

and when you check its status you should see something like:

● myService.service - My Hello Service
   Loaded: loaded (/etc/systemd/system/myService.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Sun 2019-03-24 11:06:14 UTC; 2s ago
  Process: 11419 ExecStart=/home/pi/myservice (code=killed, signal=TERM)
 Main PID: 11419 (code=killed, signal=TERM)

If you check the status after 10 seconds you should see the program running again.

It is important to know that systemd gives up restarting your service if it fails to start more than 5 times within a 10 seconds interval. There are two [Unit] options that control this:

StartLimitBurst=5
StartLimitIntervalSec=10

A simpler way is to use the RestartSec parameter. For example, if you set it to restart after 3 seconds, then you can never reach 5 failed retries within 10 seconds.

Often restart is enough but imagine that your program has a tendency to go into a useless infinite loop. In this case it doesn't end abnormally or normally and it isn't restarted. To protect against running but otherwise ineffective programs, you can set a watchdog.

If you include:

WatchdogSec=5

in the [Service] section your program has to send a keep-alive-ping every 5 seconds or it will be restarted. To be more precise, if the ping doesn't happen after 5 seconds the program is stopped using a SIGABRT and the program is restarted.

To send a keep-alive-ping you need to use the sd_notify() system call. There are several strings that you can pass which inform systemd of various states or state changes in your program, but if you send "WATCHDOG=1" then this resets the watchdog timer.

To make use of sd_notify you have to make sure that the systemd libraries are installed. On Debian systems you can do this using:

sudo apt-get update
sudo apt-get install libsystemd-dev

After this you also need to make sure that the compiler can find the header libraries. This is automatic on most systems, but if you are using a remote build machine on NetBeans you need to let it find the system tools again to update its libraries. Select the Services tab and select the Tools Collection of the server you want to update. Finally select Properties and click the Restore Defaults button which rescans the server for the tool chain and the configuration including the header files.

You also have to add the library file so that the linker can find it. Either use NetBeans to add the library file systemd or add -lsystemd to the compiler options.

With these modifications we can change our demonstration service to use a watchdog.

The C program is:

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <systemd/sd-daemon.h>
int main(int argc, char** argv) {
    while (1) {
        printf("Hello systemd world \n");
        fflush(NULL);
        sleep(rand()%7+1);
        sd_notify(0, "WATCHDOG=1");
    };
    return (EXIT_SUCCESS);
}

Now the program prints to the journal after a random pause between 1 and 7. If we set the timeout to 6 seconds:

WatchdogSec=6

then there is a one in seven chance of a timeout:

[Unit]
Description=My Hello Service
[Service]
Type=simple
ExecStart=/home/pi/myservice
Restart=always
RestartSec=10
KillMode=process
WatchdogSec=6
[Install]
WantedBy=multi-user.target

After making these changes you need to do to reload the unit file and the service is:

sudo systemctl daemon-reload
sudo systemctl restart  myService

You can monitor what is happening using:

sudo journalctl -f -a -umyService

Sooner or later you will see something like:

Mar 24 17:17:36 MyServer myservice[15370]: Hello systemd world
Mar 24 17:17:39 MyServer myservice[15370]: Hello systemd world
Mar 24 17:17:45 MyServer myservice[15370]: Hello systemd world
Mar 24 17:17:51 MyServer systemd[1]: myService.service: Watchdog timeout (limit 6s)!
Mar 24 17:17:51 MyServer systemd[1]: myService.service: Killing process 15370 (myservice) with signal SIGABRT.
Mar 24 17:17:51 MyServer systemd[1]: myService.service: Main process exited, code=killed, status=6/ABRT
Mar 24 17:17:51 MyServer systemd[1]: myService.service: Unit entered failed state.
Mar 24 17:17:51 MyServer systemd[1]: myService.service: Failed with result 'watchdog'.
Mar 24 17:18:01 MyServer systemd[1]: myService.service: Service hold-off time over, scheduling restart.
Mar 24 17:18:01 MyServer systemd[1]: Stopped My Hello Service.
Mar 24 17:18:01 MyServer systemd[1]: Started My Hello Service.
Mar 24 17:18:01 MyServer myservice[15394]: Hello systemd world
Mar 24 17:18:03 MyServer myservice[15394]: Hello systemd world

Notice that after the timeout systemd waits for the restart hold-off of ten seconds before restarting the service.

It is usually a good idea to use sd_notify to send systemd a message that your service has started:

sd_notify (0, "READY=1");

Not included in this extract

• Periodic Execution

Summary

• Linux permissions control which users can read, write or execute a file depending on their identity and group membership.

• Compilers generally automatically set the executable they produce to executable, but you still cannot run the program unless you give a path as well as its file name. This is because Linux does not search for programs in the current directory for security reasons. The simplest path is ./ which means "this directory".

• Systemd is rapidly becoming the standard way to run a program on startup.

• It is worth learning to use systemctl to create units needed to run programs. It is also used to install, start, stop and restart programs.

• Systemd maintains a central journal where all services log their status.

• You can specify restart and watchdog functions to make sure that your program is always running.

• Systemd is also a replacement for cron and can be used to run a program at a given time or periodically.

Now available as a paperback or ebook from Amazon.

Applying C For The IoT With Linux

1. C,IoT, POSIX & LINUX
2. Kernel Mode, User Mode & Syscall
   
3. Execution, Permissions & Systemd
   Extract Running Programs With Systemd
4. Signals & Exceptions
   Extract  Signals
5. Integer Arithmetic
   Extract: Basic Arithmetic As Bit Operations
   Extract: BCD Arithmetic  ***NEW
6. Fixed Point
   Extract: Simple Fixed Point Arithmetic
7. Floating Point 
8. File Descriptors
   Extract: Simple File Descriptors 
   Extract: Pipes 
9. The Pseudo-File System
   Extract: The Pseudo File System
   Extract: Memory Mapped Files 
10. Graphics
    Extract: framebuffer
11. Sockets
    Extract: Sockets The Client
    Extract: Socket Server
12. Threading
    Extract:  Pthreads
    Extract:  Condition Variables
    Extract:  Deadline Scheduling
13. Cores Atomics & Memory Management
    Extract: Applying C - Cores 
14. Interupts & Polling
    Extract: Interrupts & Polling 
15. Assembler
    Extract: Assembler

Also see the companion book: Fundamental C

<ASIN:1871962609>

<ASIN:1871962617>

Related Articles

Remote C/C++ Development With NetBeans

Raspberry Pi And The IoT In C

Getting Started With C/C++ On The Micro:bit

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info