Using Crontab to Keep Processes Running

Linux Feb 25, 2020

Though I haven't really been doing anything else interesting lately, other than looking for a new job, uninteresting busy times and all that fluff have hampered my ability to post anything meaningful (I will get around to all that crazy shit I plan on putting on here eventually as I probably keep saying). But in amongst all the ups and downs of life, this site kept going offline and requiring me to manually restart it. I'm not entirely sure why it keeps doing this (I mean, I have suspicions but I'm not going to bother going into them here). As such if you've visited and got a '503 Service Unavailable' error at any point then now you know why.

I don't actively monitor the site (I should really) for errors like this, all I do is occasionally check to make sure it's still running and if not, restart it. And yes I expect there's been a multitude of things I could have done to prevent this, even automating it all, but I haven't. Well, not until today at any rate; it did it again at some point recently and I have no idea how many hours or even days it was offline for this time, so I decided to fix the bloody issue by slaving over the crontab.

The solution I came up with is actually more than just using the crontab to keep a process running, it actually now checks to see if the daemon that runs all the programs behind the scenes is actually running itself. And if it isn't, to run it with all the previous processes that were running before the server unceremoniously killed them all for no good fucking reason. If it checks and see the daemon is already running it doesn't do anything. It does this check every 15 minutes, so if you ever come across the site with the 503 error again just wait until one of the quarter points of the hour and it should restart it.

Meanwhile if you actually want to use this method for yourself on a Linux server then here's an example crontab line for you:

*/15 * * * *  /usr/bin/pgrep processname > /dev/null || /path/to/app -arguments

The first part of course is the timing schedule the cronjob should run at and mine triggers at every 15 minutes (you can use Crontab Guru to figure out what to put here if you need something different). Then I use pgrep to find the process name in the list of running processes which normally would output the PID if it was running, but instead we dump that into the black hole that is /dev/null because we don't need it. Meanwhile if pgrep returns anything it is considered to be 'true' so I then use the 'OR' logical operator || to separate the next command, that way it stops if everything before it was set to true. This prevents the cronjob from trying to start the processes even if they're already running. Then to finish off you enter the path to your app along with any of the arguments you need to pass to it if any.

By the way, if you're running something like a Python script you might also need to include the path to python before the path to the script in the above example, so keep that in mind if it doesn't work initially.