looprm - remove oldest files to maintain free disk space
looprm [OPTION]... DIR SPACE
|
DIR |
target directory |
|||
|
SPACE |
minimal requested free space in bytes |
(supports suffixes K/M/G/T/P/E for KiB/MiB/GiB/TiB/PiB/EiB)
looprm removes the oldest files under directory DIR unless and until the free space on the underlying filesystem matches or exceeds SPACE. It operates in one-shot mode and is expected to be invoked on each new file creation event to implement the loop recording feature for video/audio recorders and data loggers.
If the free space is lower than SPACE, looprm non-recursively scans DIR for regular files (which may be name-filtered with the -b and -e options) and then removes the oldest ones (according to file modify time) until the free space grows to the requested value.
The number of files removed during a single invocation is limited, which allows efficient operation even if DIR contains a large number of matching files, as this limit allows looprm to sort only the limited number of oldest ones. The limit can and should be adjusted with the -m option according to the particular use case (see USAGE below). Without the -m option, a default limit of 16 files is used.
If the -z option is given, looprm also removes all matching empty files (except freshly created ones to avoid removing a newly created file before the first data are written to it).
Removed files are truncated to zero length before deletion to assure freeing of occupied disk space even if the files are open by other processes.
Options without arguments may be concatenated after a single hyphen. Space between options and mandatory arguments may be omitted. Optional arguments (indicated by square brackets) must directly follow options without any separating space.
Program Information
|
-h |
Output usage information. |
|||
|
-l |
Output built-in limits. |
Target File Selection
|
-b B |
Remove only files with names beginning with B. |
|||
|
-e E |
Remove only files with names ending in E. |
Operation
|
-d D |
Warn if execution time exceeds D seconds. | ||
|
-m M |
Remove at most M files (default: 16, maximum: 16777216); not applied to empty files removed with the -z option. The value of M should be set slightly above the ratio between maximal and minimal possible file size (see USAGE below). | ||
|
-n |
No-action dry run (log only, no files removed). | ||
|
-p |
Consider free space for privileged users. | ||
|
-z[A] |
Remove also all empty files older than A seconds (default: 60s when A is omitted). As the presence of empty files (which may be created as a result of free space exhaustion when unexpected files are created on the target filesystem, or SPACE and/or -m M are set too low) is disruptive for looprm due to not incrementing free space on removal, use of this option is recommended to enable removal of such empty files during directory scan (see USAGE below). |
Logging
|
-q |
Disable logging to standard error. | ||
|
-s F |
Log to syslog with facility F (implies -q). | ||
|
-v |
Increase log verbosity (use twice for debug output). |
To achieve optimal performance, SPACE should be set to a low multiple of the maximal possible file size created by the recording process. The maximal number of removed files (-m M) should be set to a minimal value required to assure freeing of sufficient disk space for such maximal file size, i.e., slightly above the ratio between maximal and minimal possible file size.
For example, in case of a data logger or recorder that creates a series of files with a constant size of 10 MiB, the value of SPACE could be set as low as 20 MiB to keep minimal free space sufficient for creating a new file, and the maximal number of removed files can be set to a low value of a few files (let’s say 8) using the -m option. Due to the nearly constant file size, the loop recording process will need to remove only the single oldest file to free enough space for the newly created one in the vast majority of cases, and even if a small file ocassionaly appears due to, e.g., a restart or power loss, the limit of -m 8 will hopefully allow looprm to remove enough files to meet the space goal, while being small enough to allow very effective operation even on a directory with a large number of files.
In the case of files with variable sizes (e.g., a motion-detection camera creating files of sizes between 0.5 and 10 MiB depending on the amount of detected motion), the maximal number of removed files needs to be raised to a higher value to reflect the fact that it may be necessary to remove 20 or even slightly more old files in order to free enough disk space for a single new file of maximal possible size. A limit of -m 32 could be sufficient for this particular example.
It’s also recommended to use the -z option, as without it, looprm treats empty files as other regular files and counts these into the limit specified with the -m option, which may amplify the negative impact of the original incident that caused the empty files to be created (without -z, a series of empty files created during a space exhaustion incident may cause another space exhaustion incident when these files become the oldest ones to remove).
To safeguard against removal of just-created files before the first data write, emty files newer than a minimal age of 60 seconds are not removed with -z. This minimal age may be adjusted with the optional argument A to the -z option.
In cases where looprm may get invoked in short intervals (e.g., a motion detection camera that creates one file per minute when continuous motion is detected, but may also create one file per 10 seconds when interrupted motion happens), it’s also recommended to use the -d D option with the value of D set to the maximal expected execution time of looprm in seconds (e.g., -d 10 in the above example). This option will cause looprm to log a warning message if it runs longer than expected, which allows to detect and avoid the possibility of multiple parallel executions of looprm due to frequent invocations.
looprm requires the underlying filesystem to return accurate and up-to-date free space information through the statvfs syscall immediately after file truncation and removal. Current version was tested on ext3/ext4 filesystems, which seem to comply with this requirement.
In addition, the filesystem must not have block count inconsistencies, as these may cause the filesystem to behave like a full one (i.e., allowing to create new files but failing to write any data to these due to space exhaustion), while the reported free block count claims that free blocks are available. To avoid such situations (which may completely block the loop recording process by causing looprm to stop removing files), use of a journaling filesystem and boot-time fsck is strongly recommended.
To achieve maximal disk utilization (by setting SPACE as low as possible to keep minimal free space) and maximal efficiency (by using low value for -m M to minimize CPU and memory usage), a dedicated partition should be used for the loop recording process.
As looprm depends on timestamps to identify the oldest files to remove, the system time needs to avoid moving backwards. On systems without hardware RTC, usage of fake-hwclock is recommended, while replacing its hourly time saving cronjob with a more frequent one may be a good idea as well:
rm
/etc/cron.hourly/fake-hwclock
echo -e ’*/5 * * * *\troot\t/sbin/fake-hwclock
save’ > /etc/cron.d/fake-hwclock
looprm uses log levels derived from syslog, ranging from debug to emergency. The default log level is notice. At this level, looprm logs one message on each removed file, as well as all warning and errors.
If a single -v option is given, the log level is changed to info, which adds a few statistic messages on filesystem usage, removed files and execution time. Giving this option twice changes log level to debug.
Errors encountered while parsing command line options given before the -s option are logged to standard error, even if the -s option is given later. As some calling programs (e.g., motion) silently discard error messages from invoked commands, it’s recommended to place the -s option as the first one when using syslog.
To implement loop video recording with motion using looprm for file removal, invoke looprm using the on_movie_start script option in motion.conf (replace EXT with the extension of recorded video files, DIR with full path to the target directory and SPACE with required free space, and adjust arguments to -d and -m according to your needs: the argument to -d should match the event_gap value which controls the minimal interval at which motion creates new files, while the argument to -m depends on the ratio between maximal and minimal possible file size as described above):
on_movie_start /usr/bin/looprm -suser -d10 -m128 -z -e.EXT DIR SPACE
The time complexity (a.k.a. CPU usage) and space complexity (a.k.a. memory usage) of looprm depend on:
|
N |
number of files under target directory | ||
|
M |
maximal number of removed files (specified with the -m M option) |
looprm first scans the target directory with worst case time complexity O(M*N) when reading directory returns files ordered as newest to oldest (best case complexity is O(N) when the order is oldest to newest), and then removes up to M files with time complexity O(M).
As the time required for filesystem operations (especially file removal) also grows with N, the time complexity of the removal phase is actualy O(M*fs(N)), with fs(N) expressing the dependency of file removal time on increasing N.
Space complexity is O(M) (or less when the actual number of target files is lower than M).
Exit status is 0 if the space goal was met, 1 if files were removed but the space goal was not met, and 2 in trouble (errors encoutered or space goal not met while no target files found).
Copyright (c)
2025 Vic B <vic@4ever.vip>.
License GPLv3: GNU GPL version 3
<https://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and
redistribute it.
There is NO WARRANTY, to the extent permitted by
law.