systemctl daemon-reload and Puppet

Update: Puppet 6.1 adds support for automatically calling systemctl daemon-reload when required, making the technique described in this post redundant.

I manage my machines at home (and at work) with Puppet, a configuration management tool. Sometimes one of my manifests needs to change a systemd service’s configuration – for example, if the upstream package didn’t ship with a unit file, or to override some settings in a .service.d directory. systemctl daemon-reload needs to be executed after changing configuration in the /etc/systemd directory, but Puppet doesn’t have built-in support for this.

Our initial solution at work was to create a systemd::daemon_reload class. It contains a single exec resource that runs systemctl daemon-reload when the class receives a refresh notification:

class systemd::daemon_reload {

  exec { '/usr/bin/systemctl daemon-reload':
    refreshonly => true,
  }

}

We then set up notification relationships from:

For example, we’d do something like the following to create a custom unit file for fcgiwrap:

class fcgiwrap {

  include ::systemd::daemon_reload

  file { '/etc/systemd/system/fcgiwrap.service':
    # ...
    notify => Class['systemd::daemon_reload'],
    # ...
  }

  service { 'fcgiwrap':
    # ...
    subscribe => Class['systemd::daemon_reload'],
    # ...
  }

}

This creates the following dependency graph, with dashed lines representing a notification dependency:

Dependency graph for a single service with notification dependencies

This solution works nicely if you only have a single service. If fcgiwrap.service is updated, a notification is sent to the systemd::daemon-reload class, which invokes systemctl daemon-reload. In turn, a notification is sent to the fcgiwrap service, which is restarted.

However, it doesn’t work so well if you introduce more than one service. Here’s what the dependency graph looks like if we add a custom unit file for stunnel in the same way:

Dependency graph for multiple services with notification dependencies

If any unit file is edited the notification dependencies cause every service to be restarted. This isn’t ideal if you have lots of services in a production environment where you want to minimize downtime caused by configuration changes.

I recently came up with a better solution to this problem by using a mixture of normal order-only dependencies and notification dependencies:

The changes to the code are minimal:

class fcgiwrap {

  include ::systemd::daemon_reload

  file { '/etc/systemd/system/fcgiwrap.service':
    # ...
    notify => [
      Class['systemd::daemon_reload'],
      Service['fcgiwrap'],
    ],
    # ...
  }

  service { 'fcgiwrap':
    # ...
    require => Class['systemd::daemon_reload'],
    # ...
  }

}

This produces the following dependency graph, with dashed lines representing notification dependencies and solid lines representing order-only dependencies:

Dependency graph for a single service with a mixture of notification and order-only dependencies

If fcgiwrap.service is updated, notifications are sent to both systemd::daemon_reload and the fcgiwrap service resource, so systemctl daemon-reload will be invoked and Puppet will restart the fcgiwrap service.

The order-only dependency additionally ensures that systemctl daemon-reload is invoked before Puppet restarts fcgiwrap.

In the case of a single service there’s no real difference from the original solution. However, here’s what the dependency graph looks like when we add an additional custom unit file for stunnel:

Dependency graph for multiple services with a mixture of notification and order-only dependencies

Because the notification dependencies on systemd::daemon_reload have been replaced with order-only dependencies, only the corresponding service is restarted when a single unit file is changed.

Another neat side effect of this particular solution is that systemctl daemon-reload is only executed at most once per each Puppet run – which wouldn’t be the case with the more obvious solution of adding one systemctl daemon-reload exec resource per unit file/service combination.