Creating your own file format definition

Request-log-analyzer support custom log file formats that can easily be specified using a simple file format. This can be used to gear the parser to your particular application and the get additional information and more detailed reports out of your log files.

Running request-log-analyzer with your own format file

Run request-log-analyzer with the --format (or -f) argument to specify your own log format file:

$ request-log-analyzer production.log --format my_format.rb

Request-log-analyzer expects the file my_format.rb to define the class MyFormat, as a subclass of RequestLogAnalyzer::FileFormat::Base or another FileFormat subclass.

Writing your own log file format

The contents of my custom file format definition my_format.rb should look something like this:

class MyFormat < RequestLogAnalyzer::FileFormat::Base
  # Define line types
  line_definition :my_line_type do |line|
    line.header = "Header"
    line.regexp = /My line looks something like: (.+)/
    line.captures << { :name => :looks_like, :type => :string }
    line.footer = "Footer"
  end

  line_definition :another_line_type do |line| ...

  # define the summary report
  report do |analyze|
    analyze.frequency :looks_like, :title => "Top 20 of looks like phrases", :amount=> 20
  end
end

A custom file format example

For Floorplanner.com, we log some additional information for every request: the current locale, because we support multiple languages, and what user currently is logged in and what account type he has. The log file contains information like this for every request:

Using locale: en-US
Logged in as: [email protected], account_type: pro

Parsing and analyzing this information can can be very insightful, so I create a custom file format. Because I want to parse the standard Rails log lines as well, I use RequestLogAnalyzer::FileFormat::Rails as base class. The report should by extended by frequency tables for locales and account types. The contents of my format file floorplanner.rb:

class Floorplanner < RequestLogAnalyzer::FileFormat::Rails

  line_definition :current_locale do |line|
    line.regexp = /Using locale: ([A-z-]+)/
    line.captures << { :name => :locale, :type => :string }
  end

  line_definition :current_user do |line|
    line.regexp = /Logged in as: (.+), account_type: (\w+)/
    line.captures << { :name => :email, :type => :string } \
                  << { :name => :account_type, :type => :string }
  end

  # Append the default Rails report with some frequency tables
  report(:append) do |analyze|
    analyze.frequency :locale, :title => "Request locales", :line_type => :current_locale
    analyze.frequency :account_type, :title => "Account types", :line_type => :current_user
  end
end

I can now use this file format by including the --format command line option:

request-log-analyzer --format floorplanner.rb production.log

Another custom file format example

Lets say you log a line every time a particular type of event happens, and that event may happen multiple times per request. For example, every time you call a thrift service you log something like:

Completed thrift call to fetch_user in 20ms

or

Completed thrift call to fetch_post in 16ms

Where ‘fetch_user’ and ‘fetch_post’ are the names of the services. You want to augment your analysis of rails logs to include analysis of which services were called, and how long they took. Your custom file format might look like:

class ThriftUsageFormat < RequestLogAnalyzer::FileFormat::Rails
   line_definition :thrift_call do |line|
      line.regexp = /Completed thrift call to ([\w\.]+) in (\d+)ms/
      line.captures = [ { :name => :service_name, :type => :string },
                        { :name => :thrift_call_duration, :type => :duration, :unit => :msec } ]
   end

   report(:append) do |analyze|
     analyze.frequency :service_name, :title => "Number of thrift calls", :line_type => :thrift_call, :multiple => true
     analyze.duration :thrift_call_duration, :title => "Duration of thrift calls", :category => :service_name, :multiple => true
   end
end

Using filters

Bacause the account_type information is now parsed, I can use this information as a filter. For instance, I can create a performance report in which only requests by pro-account holders are considered:

request-log-analyzer --format floorplanner.rb production.log --select account_type pro

Using the request database

See Creating a request database for more information about request databases in general. The newly defined lines will be included in the database as well when one is created.

request-log-analyzer --format floorplanner.rb production.log --database requests.db

On this database, I can now run a query like:

SELECT locale, COUNT(locale) FROM current_locale_lines GROUP BY locale;