Logrotating single-issue files

[Nicolas Mailhot]

[re-sending as subscribed user ]

Hi,

I'm battling with a few apps that create uniquely-named traces/logs in
a logdir (typically using PID / date or a sequence number as a unique
identifier). I naively though it might be possible to use logrotate to
compress old traces and remove them after a while.

The problem is of course once the files have been rotated once nothing
will recreate the original files so they'll be stuck in first rotation
forever (of course one might tell logrotate to create empty new files
after compression but that means filling the disk with empty files -
plus gziped empty files are *not* zero-sized anymore).

I've been trying various workarounds such as creating empty foo.x.gz
files in prerotate and removing them in postrotate but it seems
prerotate is not even entered when pattern files are not present. So I'm
stuck - it seems logrotate can not handle this case and I'll have to
cook up my own cron script.

Anyone got an idea ? Is the modified SuSE logrotate some people have
been talking about able to cope with this ?

Cheers,

[Harry Putnam]

Post by Nicolas Mailhot
The problem is of course once the files have been rotated once nothing
will recreate the original files so they'll be stuck in first rotation
forever (of course one might tell logrotate to create empty new files
after compression but that means filling the disk with empty files -
plus gziped empty files are *not* zero-sized anymore).

Investigate the `create', `size' and compress specs in man logrotate.conf

Will something like this in /etc/lograte.conf work?

/home/reader/t/var/log/*.log {
create 0600 reader reader
size=1000k
rotate 10
compress
}

[Nicolas Mailhot]

Post by Harry Putnam

Post by Nicolas Mailhot
The problem is of course once the files have been rotated once nothing
will recreate the original files so they'll be stuck in first rotation
forever (of course one might tell logrotate to create empty new files
after compression but that means filling the disk with empty files -
plus gziped empty files are *not* zero-sized anymore).

Investigate the `create', `size' and compress specs in man logrotate.conf
Will something like this in /etc/lograte.conf work?

Size won't help - the size switch means some traces won't ever be
rotated (so instead of having all traces stuck at rotation 1 some of
them will be stuck at rotation 0). And lots of very small files do fill
a partition after a while.

The core problem seems to be you can not tell logrotate to rotate a log
inconditionally, it'll only do it if other versions of the file exist so
to treat a log/trace you have to accept $rotation number of this file
will exist on the disk forever.

When you add the compress+create+ifempty switch in the balance your
$rotation file will each take 31k of disk.

Cheers,

[Jos Vos]

Post by Nicolas Mailhot
The problem is of course once the files have been rotated once nothing
will recreate the original files so they'll be stuck in first rotation
forever (of course one might tell logrotate to create empty new files
after compression but that means filling the disk with empty files -
plus gziped empty files are *not* zero-sized anymore).

Well, yes, but at first sight creating new empty files seems to be
the best solution. Furthermore, IIRC you can define a shell script
as "compresscmd" in your logrotate config, that could be like this
(all untested, RTFM, I don't know the exact interface):

if [ -s $1 ]; then
gzip $1
else
mv $1 $1.gz
fi

Cheers,

[Nicolas Mailhot]

Post by Jos Vos

Post by Nicolas Mailhot
The problem is of course once the files have been rotated once nothing
will recreate the original files so they'll be stuck in first rotation
forever (of course one might tell logrotate to create empty new files
after compression but that means filling the disk with empty files -
plus gziped empty files are *not* zero-sized anymore).

Well, yes, but at first sight creating new empty files seems to be
the best solution. Furthermore, IIRC you can define a shell script
as "compresscmd" in your logrotate config, that could be like this
if [ -s $1 ]; then
gzip $1
else
mv $1 $1.gz
fi

Thank you for the idea, if got this awful hack to work now :

"/var/log/mydir" {
rotate 4
weekly
nomissingok
create
ifempty
compress
compresscmd /usr/bin/rotatecomp
compressext .bz2
sharedscripts
postrotate
/usr/bin/rotateclean /var/log/mydir
endscript
}

With /usr/bin/rotatecomp :
if ! [ "$#" -eq 2 -a -f "$2" ] ; then
echo "$0: Usage $(basename $0) option file" >&2
exit 1
fi

[ -s "$2" ] && bzip2 $1 $2 || mv $2 $2.bz2

And /usr/bin/rotateclean :
if ! [ "$#" -eq 1 -a -d "$1" ] ; then
echo "$0: Usage $(basename $0) directory" >&2
exit 1
fi

# Remove old logrotate leftovers
for file in $(find "$1" -type f | grep -E "\.[0-9]*\.bz2$" |\
sed "s+^\(.*\)\.\([0-9]*\)\.bz2$+\1+g") ; do
[ $(du -c $file* | tail -1 | sed "s+total$++g") -eq 0 ] && rm -f
$file*
done

Now this is all really ugly and I'd probably got as good a solution
writing a non-logrotate cron. Which proves logrotate is seriously
lacking when used in this (very simple and common) case:(.

Cheers,

[Harry Putnam]

Post by Jos Vos
if [ -s $1 ]; then
gzip $1
else
mv $1 $1.gz
fi

Will something like this work? The scripting below in /etc/logrotate
would: Put all the lines from each trace.*, prefaced by filename in
BIGTRACE.file. Then it deletes the trace* files.

The next log entry checks BIGTRACE.file for size then rotates and
compresses if necessary. You can test it by creating the files and dirs
then placing this code in some.file and running: `logrotate -v
some.file'

/home/reader/t/var/test/trace.* {
size=2
nocreate
rotate 2
firstaction
awk '{print FILENAME,$0}' /home/reader/t/var/test/trace* >> /home/reader/t/va
r/test/BIGTRACE.file
endscript
lastaction
rm -f /home/reader/t/var/test/trace*
endscript
}
/home/reader/t/var/test/BIGTRACE.file {
nocreate
rotate 2
compress
size=500k
}

[Nicolas Mailhot]

Post by Harry Putnam

Post by Jos Vos
if [ -s $1 ]; then
gzip $1
else
mv $1 $1.gz
fi

Will something like this work? The scripting below in /etc/logrotate
would: Put all the lines from each trace.*, prefaced by filename in
BIGTRACE.file. Then it deletes the trace* files.

Unfortunately, you do not really want to concatenate the traces since
the app utils expect them to be broken into bits

(plus the nice thing about individual files is it's much easier to see
where an "event" starts and ends - linear logs is really bad for
parallel stuff)

Cheers,

[Keith Lofstrom]

Post by Nicolas Mailhot
I'm battling with a few apps that create uniquely-named traces/logs in
a logdir (typically using PID / date or a sequence number as a unique
identifier). I naively though it might be possible to use logrotate to
compress old traces and remove them after a while.

...

Post by Nicolas Mailhot
Anyone got an idea ? Is the modified SuSE logrotate some people have
been talking about able to cope with this ?

The SUSE modified logrotate takes an original and appends one date extension
(YYYYMMDD) then tosses it after a while. Not what you are dealing with.

Logrotate is probably not the proper tool. You may be able to use a pair
of "find ... exec ..." commands in a shell script called by cron to do
what you want. While you might build some ugly scripts and call them
from logrotate, the next maintainer will have no idea where the behavior
is coming from - logrotate is expected to do only certain things.


For example:
-----------------------------------------------------------------
#!/bin/bash
# /etc/cron.daily/myapp-log-maintainer

# compress all unused myapp log files of type A older than 2 days
find -atime 48 /var/log/myapp/logA* -exec gzip '{}' ';'

# delete all unused myapp log files of type A older than 2 weeks
# (plus the above 2 days)
find -atime 336 /var/log/myapp/logA*gz -exec rm -f '{}' ';'

#all done
exit 0
-----------------------------------------------------------------

This is untested, and I often get the -exec part wrong, use at your
own risk. I also tend to code the file paths as variables (for example,
MYAPPLOG=/var/log/myapp) and give full paths for executables in my own
shell scripts so I do not have to rely on possibly hacked PATHs, so
the example script should probably be modified those ways. You should
also put a README in the log/myapp directory telling future maintainers
what the heck is going on. You probably also want to log the fact
that this script was run in /var/log/messages.

You probably do not want to compress right away (hence the -atime). You
might actually be running the app making the log when cron starts up.
But you may be happy with "-atime 1" or something, where the log file
can get compressed only an hour after you stop actively looking at it.
Remember, you may still be running myapp when this script runs.

Lastly, if you have a nightly backup procedure called by cron.daily, you
probably want to call this first. And if you are using an rsync-based
disk backup procedure like I am, you do not want to modify the files at
all, or it causes just another copy (albeit compressed) to happen. Just
delete them when they get old to keep your directories uncluttered.

Keith

[Nicolas Mailhot]

Post by Keith Lofstrom

Post by Nicolas Mailhot
I'm battling with a few apps that create uniquely-named traces/logs in
a logdir (typically using PID / date or a sequence number as a unique
identifier). I naively though it might be possible to use logrotate to
compress old traces and remove them after a while.

...

Post by Nicolas Mailhot
Anyone got an idea ? Is the modified SuSE logrotate some people have
been talking about able to cope with this ?

The SUSE modified logrotate takes an original and appends one date extension
(YYYYMMDD) then tosses it after a while. Not what you are dealing with.
Logrotate is probably not the proper tool. You may be able to use a pair
of "find ... exec ..." commands in a shell script called by cron to do
what you want. While you might build some ugly scripts and call them
from logrotate, the next maintainer will have no idea where the behavior
is coming from - logrotate is expected to do only certain things.

I was seriously considering this way, right. I got logrotate to work but
it's way too ugly.

The downside of it is when you write custom scripts nobody enhances them
for you and the next user won't know where to find them :( (plus you
don't have the nice logrotate config interface). It's really sad the
logrotate authors seem to have forgotten/ignored this case. I started
because I app crashed a system filling up the disks with unhandled
traces, and I've already identified two other apps which behave the same
way:(

Regards,