Implement Structured Logging with Axiom Support

[loftwah]

We need to improve our application's logging capabilities by implementing structured logging. This will make our logs more searchable, analyzable, and machine-readable. We'll use Axiom as our primary logging service, but also provide an option for non-Axiom structured logging.

Tasks:

1. Research and choose a Ruby gem for structured logging (e.g., lograge, semantic_logger)

2. Add the chosen gem to the project's Gemfile

3. Configure the logging gem in the Rails application

4. Implement Axiom integration:

   • Add Axiom client library to the project
   • Configure Axiom credentials and endpoint
   • Set up log forwarding to Axiom

5. Create a configuration option to toggle between Axiom and non-Axiom logging

6. Implement custom log formatters for both Axiom and non-Axiom outputs

7. Add context to logs (e.g., request ID, user ID, environment)

8. Update existing log calls throughout the application to use the new structured format

9. Implement error handling and fallback mechanisms for logging failures

10. Create documentation for the new logging system, including:

    • How to configure logging (Axiom and non-Axiom)
    • How to add custom fields to logs
    • Best practices for logging in the application

11. Add logging statements for key events and metrics in the application

12. Set up log rotation and archiving for non-Axiom logs

13. Create sample queries for common log analysis tasks in Axiom

14. Test logging in various environments (development, staging, production)

This enhancement will greatly improve our ability to monitor, debug, and analyze our application's behavior and performance.