logrotate succeeds when manually run as root, but fails with "Read-only file system" when run by logrotate.service

I have a (formerly working) logrotate setup for OpenResty that I took from a previous Ubuntu 18.04 installation. Yet the logrotate.service fails with this error now ...

error: error renaming /usr/local/openresty/nginx/logs/access.log.60.zst
to /usr/local/openresty/nginx/logs/access.log.61.zst: Read-only file system

... and I have difficulties understanding why. The new machines are running Ubuntu 20.04, but I don't see why that should make a difference in this case.

First, here is the config:

$ cat /etc/logrotate.d/custom-openresty
/usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
{ daily rotate 60 maxsize 1G missingok notifempty compress compresscmd /usr/bin/zstd uncompresscmd /usr/bin/unzstd compressoptions -9 --long -T1 compressext .zst delaycompress sharedscripts postrotate test ! -f /usr/local/openresty/nginx/logs/nginx.pid || kill -USR1 `cat /usr/local/openresty/nginx/logs/nginx.pid` endscript
}

/etc/logrotate.conf is unchanged and looks like this:

# see "man logrotate" for details
# rotate log files weekly
weekly
# use the adm group by default, since this is the owning group
# of /var/log/syslog.
su root adm
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# use date as a suffix of the rotated file
#dateext
# uncomment this if you want your log files compressed
#compress
# packages drop log rotation information into this directory
include /etc/logrotate.d
# system-specific logs may be also be configured here.

Here is the state of my files (I would expect to get access.log.1 after logrotate ran):

$ ls -alhg /usr/local/openresty/nginx/logs/
total 10G
drwxr-xr-x 2 root 4.0K Sep 16 20:04 .
drwxr-xr-x 18 root 4.0K Sep 16 19:42 ..
-rw-r--r-- 1 root 10G Sep 16 19:56 access.log
-rw-r--r-- 1 root 5.5K Sep 16 19:56 error.log

However, the logrotate.service fails with this error:

~$ systemctl status logrotate.service
● logrotate.service - Rotate log files Loaded: loaded (/lib/systemd/system/logrotate.service; static; vendor preset: enabled) Active: failed (Result: exit-code) since Wed 2020-09-16 20:11:32 UTC; 4min 32s ago
TriggeredBy: ● logrotate.timer Docs: man:logrotate(8) man:logrotate.conf(5) Process: 27403 ExecStart=/usr/sbin/logrotate /etc/logrotate.conf (code=exited, status=1/FAILURE) Main PID: 27403 (code=exited, status=1/FAILURE)
$ sudo journalctl --unit logrotate.service
Sep 16 20:11:32 fetcher-scheduler systemd[1]: Starting Rotate log files...
Sep 16 20:11:32 fetcher-scheduler logrotate[27403]: error: error renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst: Read-only file system
Sep 16 20:11:32 fetcher-scheduler systemd[1]: logrotate.service: Main process exited, code=exited, status=1/FAILURE
Sep 16 20:11:32 fetcher-scheduler systemd[1]: logrotate.service: Failed with result 'exit-code'.
Sep 16 20:11:32 fetcher-scheduler systemd[1]: Failed to start Rotate log files.

When I run it (without systemd) in debug mode as root, I get the following output:

# logrotate -v -d /etc/logrotate.d/custom-openresty
reading config file /etc/logrotate.d/custom-openresty
compress_prog is now /usr/bin/zstd
uncompress_prog is now /usr/bin/unzstd
compress_options is now -9 --long -T1
compress_ext is now .zst
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state
Creating new state
...
Creating new state
Handling 1 logs
rotating pattern: /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log after 1 days (60 rotations)
empty log files are not rotated, log files >= 1073741824 are rotated earlier, old logs are removed
considering log /usr/local/openresty/nginx/logs/access.log Now: 2020-09-16 20:24 Last rotated at 2020-09-16 20:11 log needs rotating
considering log /usr/local/openresty/nginx/logs/error.log Now: 2020-09-16 20:24 Last rotated at 2020-09-16 19:00 log does not need rotating (log has been rotated at 2020-9-16 19:0, that is not day ago yet)
rotating log /usr/local/openresty/nginx/logs/access.log, log->rotateCount is 60
dateext suffix '-20200916'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
previous log /usr/local/openresty/nginx/logs/access.log.1 does not exist
renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst (rotatecount 60, logstart 1, i 60),
renaming /usr/local/openresty/nginx/logs/access.log.59.zst to /usr/local/openresty/nginx/logs/access.log.60.zst (rotatecount 60, logstart 1, i 59),
...
/logs/access.log.3.zst (rotatecount 60, logstart 1, i 2),
renaming /usr/local/openresty/nginx/logs/access.log.1.zst to /usr/local/openresty/nginx/logs/access.log.2.zst (rotatecount 60, logstart 1, i 1),
renaming /usr/local/openresty/nginx/logs/access.log.0.zst to /usr/local/openresty/nginx/logs/access.log.1.zst (rotatecount 60, logstart 1, i 0),
log /usr/local/openresty/nginx/logs/access.log.61.zst doesn't exist -- won't try to dispose of it
renaming /usr/local/openresty/nginx/logs/access.log to /usr/local/openresty/nginx/logs/access.log.1
running postrotate script
running script with arg /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log
: " test ! -f /usr/local/openresty/nginx/logs/nginx.pid || kill -USR1 `cat /usr/local/openresty/nginx/logs/nginx.pid`
"

To me, everything looks normal. If I run it, it also works:

# logrotate -v /etc/logrotate.d/custom-openresty
reading config file /etc/logrotate.d/custom-openresty
compress_prog is now /usr/bin/zstd
uncompress_prog is now /usr/bin/unzstd
compress_options is now -9 --long -T1
compress_ext is now .zst
Reading state from file: /var/lib/logrotate/status
Allocating hash table for state file, size 64 entries
Creating new state
...
Creating new state
Handling 1 logs
rotating pattern: /usr/local/openresty/nginx/logs/access.log
/usr/local/openresty/nginx/logs/error.log after 1 days (60 rotations)
empty log files are not rotated, log files >= 1073741824 are rotated earlier, old logs are removed
considering log /usr/local/openresty/nginx/logs/access.log Now: 2020-09-16 20:26 Last rotated at 2020-09-16 20:11 log needs rotating
considering log /usr/local/openresty/nginx/logs/error.log Now: 2020-09-16 20:26 Last rotated at 2020-09-16 19:00 log does not need rotating (log has been rotated at 2020-9-16 19:0, that is not day ago yet)
rotating log /usr/local/openresty/nginx/logs/access.log, log->rotateCount is 60
dateext suffix '-20200916'
glob pattern '-[0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9]'
previous log /usr/local/openresty/nginx/logs/access.log.1 does not exist
renaming /usr/local/openresty/nginx/logs/access.log.60.zst to /usr/local/openresty/nginx/logs/access.log.61.zst (rotatecount 60, logstart 1, i 60),
old log /usr/local/openresty/nginx/logs/access.log.60.zst does not exist
...
old log /usr/local/openresty/nginx/logs/access.log.2.zst does not exist
renaming /usr/local/openresty/nginx/logs/access.log.1.zst to /usr/local/openresty/nginx/logs/access.log.2.zst (rotatecount 60, logstart 1, i 1),
old log /usr/local/openresty/nginx/logs/access.log.1.zst does not exist
renaming /usr/local/openresty/nginx/logs/access.log.0.zst to /usr/local/openresty/nginx/logs/access.log.1.zst (rotatecount 60, logstart 1, i 0),
old log /usr/local/openresty/nginx/logs/access.log.0.zst does not exist
log /usr/local/openresty/nginx/logs/access.log.61.zst doesn't exist -- won't try to dispose of it
renaming /usr/local/openresty/nginx/logs/access.log to /usr/local/openresty/nginx/logs/access.log.1
running postrotate script

No error and in the end access.log.1 is created as expected:

# ls -algh /usr/local/openresty/nginx/logs/
total 10G
drwxr-xr-x 2 root 4.0K Sep 16 20:26 .
drwxr-xr-x 18 root 4.0K Sep 16 19:42 ..
-rw-r--r-- 1 root 10G Sep 16 19:56 access.log.1
-rw-r--r-- 1 root 5.5K Sep 16 19:56 error.log

Note that it said ...

log /usr/local/openresty/nginx/logs/access.log.61.zst
doesn't exist -- won't try to dispose of it

... instead of error renaming ... Read-only file system (as logrotate.service does)

Why does it work when running manually as root, but fails when it is executed by the logrotate.service?

I did not make any changes to logrotate.service. For completeness, here is the unit file:

$ systemctl cat logrotate.service
# /lib/systemd/system/logrotate.service
[Unit]
Description=Rotate log files
Documentation=man:logrotate(8) man:logrotate.conf(5)
ConditionACPower=true
[Service]
Type=oneshot
ExecStart=/usr/sbin/logrotate /etc/logrotate.conf
# performance options
Nice=19
IOSchedulingClass=best-effort
IOSchedulingPriority=7
# hardening options
# details:
# no ProtectHome for userdir logs
# no PrivateNetwork for mail deliviery
# no ProtectKernelTunables for working SELinux with systemd older than 235
# no MemoryDenyWriteExecute for gzip on i686
PrivateDevices=true
PrivateTmp=true
ProtectControlGroups=true
ProtectKernelModules=true
ProtectSystem=full
RestrictRealtime=true

Now I'm running out of options. Any help in troubleshooting the problem is much appreciated.

2 Answers

I think, I found it. The logrotate file is not the problem.

Instead, it is caused by the hardening features in the systemd unit file. After I disabled the ProtectSystem=full hardening options, it worked. The reason is that logrotate has to operate on the /usr directory in my case, which is readonly if the option is enabled.

From the documentation on ProtectSystem=

Takes a boolean argument or the special values "full" or "strict". If true, mounts the /usr and the boot loader directories (/boot and /efi) read-only for processes invoked by this unit. If set to "full", the /etc directory is mounted read-only, too. If set to "strict" the entire file system hierarchy is mounted read-only, except for the API file system subtrees /dev, /proc and /sys (protect these directories using PrivateDevices=, ProtectKernelTunables=, ProtectControlGroups=). This setting ensures that any modification of the vendor-supplied operating system (and optionally its configuration, and local mounts) is prohibited for the service. It is recommended to enable this setting for all long-running services, unless they are involved with system updates or need to modify the operating system in other ways. If this option is used, ReadWritePaths= may be used to exclude specific directories from being made read-only. This setting is implied if DynamicUser= is set. This setting cannot ensure protection in all cases. In general it has the same limitations as ReadOnlyPaths=, see below. Defaults to off.

To fix it properly, I added now the following line to /lib/systemd/system/logrotate.service:

ReadWritePaths=/usr/local/openresty/nginx/logs

Then ProtectSystem=full will mount everything as readonly except the directory where the logs are. Now, the readonly error is gone:

# systemctl daemon-reload && systemctl start logrotate
(completed successfully without output)

I assume, I never hit the problem on Ubuntu 18.04 because the hardening options were not yet the defaults. Ubuntu 20.04 has it now, but I don't know in what Ubuntu release they were first introduced.

Nice find to the same problem i was seeing. Editing nginx.conf to log to different directory (eg. the common /var/log/nginx) is an alternative that also works.