feat: add console logging for LLM guardrail events

[greysonlalonde]

Summary

• Add console formatter for LLM guardrail events
• Fix event emission order to ensure started event fires before execution
• Add exception handling to prevent incomplete event streams

Changes

• Implement guardrail event formatting in console formatter
• Display retry count for guardrail completed events
• Move LLMGuardrailStartedEvent emission before guardrail execution to capture all attempts
• Wrap guardrail execution in try-except to ensure completed event is always emitted even on failure

[joaomdmoura]

Disclaimer: This review was made by a crew of AI Agents.

Code Review Comment for PR #3105

Overview

This pull request introduces console logging functionality for memory events and LLM guardrail events in the CrewAI framework. Two primary files are modified: event_listener.py and console_formatter.py, each enhancing the observability of event handling.

src/crewai/utilities/events/event_listener.py

Positive Aspects

• The event handling structure is well-implemented and demonstrates a clear separation of concerns for different event types.
• Error handling is consistent, which aids in maintaining the robustness of the event management system.

Recommendations:

1. Avoid Code Duplication in Guardrail Name Resolution:

   • Current Implementation: The logic for determining guardrail_name appears multiple times.

   • Suggestion: Extract this logic into a helper function to promote DRY (Don't Repeat Yourself) principles:

     def resolve_guardrail_name(guardrail):
         if isinstance(guardrail, str):
             return guardrail[:50] + "..." if len(guardrail) > 50 else guardrail
         return getattr(guardrail, "__name__", None) or \
                getattr(guardrail, "__class__.__name__", "Guardrail")
     
     guardrail_name = resolve_guardrail_name(event.guardrail)

2. Enhance Event Handler Documentation:

   • Adding comprehensive docstrings to each event handler will improve maintainability and facilitate knowledge transfer amongst team members. For instance, enhance the following method:

     @crewai_event_bus.on(LLMGuardrailStartedEvent)
     def on_llm_guardrail_started(source, event: LLMGuardrailStartedEvent):
         """Handle the start of LLM guardrail evaluation.
     
         Args:
             source: Event source
             event: LLMGuardrailStartedEvent containing guardrail details
         """

src/crewai/utilities/events/utils/console_formatter.py

Positive Aspects

• The formatting logic is well-structured with consistent color-coding to denote different states of events.
• Good error handling practices are established in display functions.

Recommendations:

1. Refactor Redundant Status Creation Logic:

   • Current Issue: The pattern for creating status content appears several times with minor variations.

   • Suggestion: Create a dedicated method to handle guardrail statuses:

     def create_guardrail_status(self, state, name, color, retry_count, error=None):
         """Create standardized guardrail status content."""
         status_map = {
             'started': ('Evaluating', '🔄'),
             'success': ('Validated', '✅'),
             'failed': ('Failed', '❌')
         }
     
         status_text, icon = status_map[state]
         return self.create_status_content(
             f"Guardrail {status_text}",
             name,
             color,
             Status=f'{icon} {status_text}',
             Attempts=f'{retry_count + 1}',
             Error=str(error) if error else None
         )

2. Improve Error Message Handling:

   • Ensure that error messages are appropriately formatted and truncated. Here’s a potential enhancement:

     def format_error_message(error):
         """Format error message with proper truncation and sanitization."""
         if not error:
             return "Unknown error"
         return (str(error)[:100] + '...') if len(str(error)) > 100 else str(error)

General Recommendations:

• Introduce Comprehensive Type Hints: Explicit type hints throughout the code will reduce ambiguity and improve code comprehension and type safety.

• Define Constants for Magic Values: Extract hard-coded values into meaningful constants to enhance clarity and maintainability.

  class ConsoleFormatter:
      MAX_GUARDRAIL_NAME_LENGTH = 50
      DEFAULT_GUARDRAIL_NAME = "Guardrail"
      COLORS = {
          'success': 'green',
          'warning': 'yellow',
          'error': 'red'
      }

• Implement Unit Tests: Add unit tests for the new functionality, especially for edge cases and error handling, to ensure reliability.

• Documentation: Providing comprehensive module-level documentation, capturing the event handling flow and the use cases for formatting, will enhance the onboarding process for new developers.

Overall, while the changes in this PR significantly improve the functionality for logging events, the above recommendations will further enhance maintainability, readability, and robustness, contributing positively to the codebase's health and future scalability.