@emmatonkin I run a personal Asterisk server, and until I had the good sense to set up fail2ban, I had a cronjob that would wipe out most of the oldest rotated log files, since it would be literally gigabytes of failed logins.
The job was modeled on what I'd have to do every time the drive got full and my phones stopped working.
@Alex
In our academic remote hardware deployments in homes, despite everything, often the first or even the only remote sign of a new and exotic failure mode would be logfile usage suddenly accelerating like mad. We ended up with Ansible scripts to detect and characterise these issues ... and if the only way to keep it running during that process were to sacrifice less-significant low-hanging log/datafiles, we'd take that route rather than lose data :)
In conclusion I guess my version of the unprofessional advice would also include 1) never allow loads of different things to log into a single logfile and 2) when deploying remotely on limited hardware avoid things that don't deflate gracefully (cough that sounds like *mongodb*) 😃
@Alex would have saved my ass more than once. Servers where even command auto completion doesn't work because you don't have enough space left are hard to clean.
@Alex that is actually a good tip and used that on the past. By today standards a system can afford 1-2G dummy file unless special cases.
@robertomurta I use this trick regularly when I'm dealing with a VPS that's basically scratch space. Servers that don't do any damage when they break beyond annoying me personally.
It's definitely not in-production advice.
@Alex this is used professionally more often than you may think
@xoder I have seen it myself 😬
@Alex & you can claim that it's an encrypted folder for which you don't recall the password if you're visited by the police
@Alex sounds like a version of safe fork that we wrote and used in CS101 so that we didn't fork bomb the Linux nodes
@Alex
There's often those reserved blocks tunefs -m keeps to one side to play with (unless there isn't, obvs). But I have genuinely known people who keep lists of log files to sacrifice in this very eventuality.