screen for ghetto servers and startup scripts.
GNU screen is a good little tool for server administration, or running things on your own remote machines. It's even good for running things locally.
I hope this is useful for people who want to run scripts every time they login, or reboot... and who need interactive access to those scripts. Or useful for those people who are already using screen, but would like to make their setup a bit better:
Running things as daemons is cool... but if you'd also like interactive control occasionally, running things with screen is useful.
Most servers have screen, watch and crontab(osx is lacking watch though) - including most linux distros, *bsd, osx, windows(with cygwin). Most OSes also have their own style init scripts(scripts to run things at boot or logon). So this screen, watch, crontab combination is ok if you want to use multiple different types of computers - but reuse your scripts. If you want something robust, and good this isn't for you.
Scripting sessions
You can make a shell script to script your screen sessions. So you don't need to do them manually each time you login. This can save you a *lot* of time, and can make you less afraid of a reboot :).
-- /home/me/bin/my_screens.sh --
Finding screen sessions easily: you can run these commands:
This will let you connect to that screen from any shell. It detatches any session that's open. This is good as you can easily remember things with just the short names you choose, eg 'app1' 'proxy' etc. Normally you have to do 'screen -ls' look at the output, find the session you want to connect to, then finally 'screen -r random_sessionname'.
At Reboot: Then you can add this to your crontab with crontab -e (be careful not to use -r!!! which is right next to e on a qwerty keyboard). Or your crontab web interface with your hosting account( for example cpanel/whm, webmin, plex etc). Note that each user has a crontab. So to run your apps as different users, just login and change each ones crontab(or use sudo, or 'su username -l -c' from the roots crontab).
Restarting: Of course your app might crash or something... Look at the first one... that one has a watch -n 2... which means "run this process, wait for it to finish, then start it again after 2 seconds." It's kind of like a ghetto daemontools. Not as good as something like daemontools... but good enough for some purposes.
Monitoring: You can have a separate tool monitoring your scripts if you like... then if your app has frozen, or is overloaded... just send it some term, then kill signals... and watch will start another one up when it dies. Consider each script a 'runtime test'. If the 'runtime test' fails, then kill the app and restart it. The app could be a ssh proxy for your vpn - in which case the 'runtime test' would see if you can ping the network, and if not kill the ssh connection... so it restarts. A webserver runtime test might see if it can do a GET request... if not kill the server. A ghetto monitoring system for sure... but simple.
Logging: It's pretty easy to add some stderr and stdout redirection with > /home/me/logs/app1stdout >> /home/me/logs/app1stderr. This way you can have ghetto logging too.
Resource control: You can use ulimit in your scripts to limit how many resources your server can use. Then if it uses too much, it will die and be restarted in two seconds. Say you think your python web server should never *ever* take up 500MB of ram, then run it from a .sh file, and put ulimit -m 500000 before it. See ulimit -a for a list of things you can limit. Ghetto-quick resource control. Similarly you can use nice and ionice to make things behave more nicely :).
Debugging: screen doesn't give you an error message with -d -m. So you can either look in your logs or try out your command first with "screen cmd". eg "screen python". You can try out your @reboot command, not just by rebooting but by setting it to run in 10 minutes from now. See cron help pages for how to do that. You might want to use a sh -l in front of your command so it's a 'login shell'. This will setup your paths and environment variables like your login. Or setup explicitly for each app/script which paths and environment variables they need.
Again this isn't for everyone... but some ideas here might be useful for your own ghetto screen usage for servers and startup scripts.
I hope this is useful for people who want to run scripts every time they login, or reboot... and who need interactive access to those scripts. Or useful for those people who are already using screen, but would like to make their setup a bit better:
- scripting sessions, rather than doing them manually at each login or reboot,
- finding your screen sessions more easily.
- restarting scripts at reboot,
- monitoring,
- logging,
- resource control
Running things as daemons is cool... but if you'd also like interactive control occasionally, running things with screen is useful.
Most servers have screen, watch and crontab(osx is lacking watch though) - including most linux distros, *bsd, osx, windows(with cygwin). Most OSes also have their own style init scripts(scripts to run things at boot or logon). So this screen, watch, crontab combination is ok if you want to use multiple different types of computers - but reuse your scripts. If you want something robust, and good this isn't for you.
Scripting sessions
You can make a shell script to script your screen sessions. So you don't need to do them manually each time you login. This can save you a *lot* of time, and can make you less afraid of a reboot :).
-- /home/me/bin/my_screens.sh --
#!/bin/bash
# start up your 'app1' to be restarted 2 seconds after it dies, every time it dies.
screen -d -m -S app1 -t "my web app one descriptive name" /opt/local/bin/watch -n 2 /home/me/someapp1/run_app.py > /home/me/logs/app1stdout >> /home/me/logs/app1stderr
# start running an application.
screen -d -m -S proxy -t "proxy server" /home/me/someapp2/run_app.py
# connect to my my server setup with ssh keys.
screen -d -m -S server1 -t "my server" ssh myserver.example.com
# monitoring run time test of my app1 every 67 seconds
screen -d -m -S monitor1 -t "my monitor script" /opt/local/bin/watch -n 67 /home/me/bin/app1_monitor.pyFinding screen sessions easily: you can run these commands:
screen -d -r app1
screen -d -r server1
screen -d -r proxyThis will let you connect to that screen from any shell. It detatches any session that's open. This is good as you can easily remember things with just the short names you choose, eg 'app1' 'proxy' etc. Normally you have to do 'screen -ls' look at the output, find the session you want to connect to, then finally 'screen -r random_sessionname'.
At Reboot: Then you can add this to your crontab with crontab -e (be careful not to use -r!!! which is right next to e on a qwerty keyboard). Or your crontab web interface with your hosting account( for example cpanel/whm, webmin, plex etc). Note that each user has a crontab. So to run your apps as different users, just login and change each ones crontab(or use sudo, or 'su username -l -c' from the roots crontab).
@reboot /home/me/bin/my_screens.sh
# this line below can be used from a root account to run as the user 'me'.
@reboot su me -l -c /home/me/bin/my_screens.shRestarting: Of course your app might crash or something... Look at the first one... that one has a watch -n 2... which means "run this process, wait for it to finish, then start it again after 2 seconds." It's kind of like a ghetto daemontools. Not as good as something like daemontools... but good enough for some purposes.
Monitoring: You can have a separate tool monitoring your scripts if you like... then if your app has frozen, or is overloaded... just send it some term, then kill signals... and watch will start another one up when it dies. Consider each script a 'runtime test'. If the 'runtime test' fails, then kill the app and restart it. The app could be a ssh proxy for your vpn - in which case the 'runtime test' would see if you can ping the network, and if not kill the ssh connection... so it restarts. A webserver runtime test might see if it can do a GET request... if not kill the server. A ghetto monitoring system for sure... but simple.
Logging: It's pretty easy to add some stderr and stdout redirection with > /home/me/logs/app1stdout >> /home/me/logs/app1stderr. This way you can have ghetto logging too.
Resource control: You can use ulimit in your scripts to limit how many resources your server can use. Then if it uses too much, it will die and be restarted in two seconds. Say you think your python web server should never *ever* take up 500MB of ram, then run it from a .sh file, and put ulimit -m 500000 before it. See ulimit -a for a list of things you can limit. Ghetto-quick resource control. Similarly you can use nice and ionice to make things behave more nicely :).
Debugging: screen doesn't give you an error message with -d -m. So you can either look in your logs or try out your command first with "screen cmd". eg "screen python". You can try out your @reboot command, not just by rebooting but by setting it to run in 10 minutes from now. See cron help pages for how to do that. You might want to use a sh -l in front of your command so it's a 'login shell'. This will setup your paths and environment variables like your login. Or setup explicitly for each app/script which paths and environment variables they need.
Again this isn't for everyone... but some ideas here might be useful for your own ghetto screen usage for servers and startup scripts.
Comments