vi backup files considered harmful

Mark this one down as another in the long list of "duh" — once you realise what is going on!

Bug report comes in about a long running daemon that has stopped logging. lsof reports the log file is now named logfile~ and further more is deleted! This happens after a system upgrade scenario, so of course I go off digging through a multitude of scripts and what-not to find the culprit...

Have you got it yet?

Try this...

# lsof | grep syslogd | grep messages
syslogd    1376        root   15w      REG        3,1    99851    4605625 /var/log/messages
# cd /var/log/
# vi messages (and save the file)
root@jj:/var/log# lsof | grep syslogd | grep messages
syslogd    1376        root   15w      REG        3,1    99851    4605625 /var/log/messages~ (deleted)

vi is very careful and renames your existing file, so that if anything goes wrong when writing the new version you can get something back. It's a shame the daemon doesn't know about this! The kernel is happy to deal with the rename, but when the backup file is unlinked you're out of luck. Confusingly to a casual inspection your log file looks like it's there ... just that nothing is going into it. (oh, and if you tried that, you might like to restart syslogd now :)

Moral of the story -- overcome that finger-memory and never use vi on a live file; you're asking for trouble!