Prompt injection detection in goose

[dorien-koelemeijer]

What Is Prompt Injection?

Prompt injection is a security vulnerability where malicious commands are embedded in content that goose processes. These attacks can trick goose into running dangerous system commands, accessing sensitive files, or executing unintended actions on your system.
These threats can come from:

• External recipes downloaded from untrusted sources
• Documents or files containing hidden commands
• Web content that goose processes
• Innocent-looking sources that seem legitimate but contain hidden malicious instructions
  Example: You download a helpful-looking recipe for "organising project files" that actually contains hidden commands to access your sensitive data.

How Goose Protects You

Goose will include built-in security scanning that analyses tool calls before they execute from the next release onwards. Our security system:

• Pattern-based detection: Identifies suspicious commands, file paths, and injection techniques
• Risk assessment: Categorises threats as low, medium, high, or critical
• User control: Always asks for your approval when potential threats are detected

What You'll Experience

When goose detects a potentially malicious tool call, you'll see:

• Security alert: A warning explaining what was detected
• Risk level: How serious the potential threat is (Low/Medium/High/Critical)
• Approval dialogue: Options to “Allow Once” or “Deny” this type of action

How To Enable Prompt Injection Protection

To enable security protection in goose, simply update your goose config to include the following (this config will change slightly in the next release):

security:
  enabled: true
  threshold: 0.7

Or enable in the UI (available soon):

The threshold (0.0 to 1.0) controls how sensitive the security scanning is:

• Lower values (0.3-0.5): More sensitive - catches more potential threats but may flag legitimate actions
• Higher values (0.7-0.9): Less sensitive - only flags high-confidence threats, fewer false positives
• Recommended: 0.7 - Good balance between security and usability

What’s Next

This is the first iteration of goose's security protection, using pattern-based detection. We're actively working on adding AI model-based scanning for more sophisticated prompt injection detection in future releases.

[taniandjerry]

Thank you for sharing this important discussion topic here! Security is always an important concern.