Unprofessional Protip:
Keep a 1GB file full of random numbers on your #Linux server's disk so that when you neglect it and let the drive get full, you have 1GB you can delete right away to get things less broken.
@Alex
There's often those reserved blocks tunefs -m keeps to one side to play with (unless there isn't, obvs). But I have genuinely known people who keep lists of log files to sacrifice in this very eventuality.
@emmatonkin I run a personal Asterisk server, and until I had the good sense to set up fail2ban, I had a cronjob that would wipe out most of the oldest rotated log files, since it would be literally gigabytes of failed logins.
The job was modeled on what I'd have to do every time the drive got full and my phones stopped working.
In conclusion I guess my version of the unprofessional advice would also include 1) never allow loads of different things to log into a single logfile and 2) when deploying remotely on limited hardware avoid things that don't deflate gracefully (cough that sounds like *mongodb*) 😃
@Alex
In our academic remote hardware deployments in homes, despite everything, often the first or even the only remote sign of a new and exotic failure mode would be logfile usage suddenly accelerating like mad. We ended up with Ansible scripts to detect and characterise these issues ... and if the only way to keep it running during that process were to sacrifice less-significant low-hanging log/datafiles, we'd take that route rather than lose data :)