Production Environment Tips & Tricks

This page is collection of tips & tricks about our production environment.

Running rake/rails commands

We are running the rails application as the user wwwrun. So whatever you want to do you should also do this as this user to avoid creating files/running services with the wrong permissions. For this just prepend

run_in_api rails console

List service status

We have our own systemd target that defines all the services we need. You can look at the status of all of them with

systemctl list-dependencies obs-api-support.target

Service Logs

There are a couple of log files

• Ruby on Rails: /srv/www/obs/api/log/production.log

• Ruby on Rails calling the backend: /srv/www/obs/api/log/backend_access.log

• clockworkd: /srv/www/obs/api/log/clockworkd.clock.output

• searchd: /srv/www/obs/api/log/production.searchd.log and /srv/www/obs/api/log/production.searchd.query.log

• Apache: /srv/www/obs/api/log/apache_access.log and /srv/www/obs/api/log/error.log

• Postfix: /var/log/mail

• All systemd units: journalctl -u $SERVICE

  Filter messages within a time range (either timestamp or placeholders like "yesterday"):

  journalctl -u $SERVICE --since now|today|yesterday|tomorrow --until YYYY-MM-DD HH:MM:SS

Bash History

We are recording shell commands with timestamps in /root/.bash_history. You can look at it with the history command. Please note that the history is not sorted in chronological order. You can look at this in chronological order with

history | tr --squeeze-repeats " " | cut -d " " -f 3- | sort

Pry History

pry which we use as rails console has a nice history feature. You can also look at the history file:

/var/lib/wwwrun/.local/share/pry/pry_history

Who is/was logged in?

journalctl -u sshd --utc -o json --since "6 hours ago" | ruby /root/who.rb

Toggle admin rights for a user

run_in_api rake user:toggle_admin_rights theusername

Poor mans IP based analytics

Grep the top 15 IPs requesting Webui::PackageController#view_file

grep 'controller=Webui::PackageController action=view_file' log/production.log |awk 'match($0,/\yhost=(\S+)/, arr) {print arr[1]}'|sort |uniq -c|sort |tail -n 15