nginx add robots.txt

[ZhaiSoul]

Given that Frigate is primarily used in home environments where privacy is a significant concern, should we consider adding a robots.txt file through nginx to prevent search engines from indexing the content? While this may not completely eliminate privacy risks, it should at least help avoid having the content indexed.

[NickM-27]

I took a look at HomeAssistant repo and they don't have one either, this seems like a non-concern to me personally especially since I believe in most cases users are:

1. Not exposing to the open web (either not available or using VPN / tailscale)
2. Use something like cloudflare which enforces this separately
3. None of these things and also just exposing Frigate to the open web with no protection

Seems like something that users who prefer it can add if they'd like, either inside Frigate or at their own hosting level

[ZhaiSoul]

Yes, I'm specifically addressing the third scenario. Adding a robots.txt is merely a defensive security measure, and it won't cause any negative side effects.

[hawkeye217]

robots.txt is just fundamentally a polite request to web crawlers and search engines, not a security mechanism. It has no enforcement capability - malicious actors can completely ignore it. It's also publicly accessible, potentially revealing directory structure or endpoints that one might try to hide, and relies on voluntary compliance from web crawlers.

I don't see a need to implement it, especially if Home Assistant doesn't even do so.

[ZhaiSoul]

I don't see a need to implement it, especially if Home Assistant doesn't even do so.

https://github.com/home-assistant/frontend/blob/98c4e34a230e54d6e4110a3fb0921a958272c06b/public/robots.txt

Actually, Home Assistant's frontend already has a robots.txt configured - you just need to set a simple Disallow: / to block all crawler access.

[ZhaiSoul]

I believe this really warrants our attention. Unprotected Frigate instances can be directly found through public search engines, and their owners may not even realize their surveillance feeds have been exposed.

At the minimum, we should ensure regular search engines can't crawl these pages or add them to their indexes. For users who have port 5000 open, we might explore other solutions for security detection in the future - but for now, we need to do what's fundamentally necessary.

Moreover, projects like Home Assistant already implement robots.txt to prevent search engine crawling, and this doesn't expose their directory structures. It simply requires adding one line: Disallow: / .

[hawkeye217]

I'm fine with adding this, but I will continue to caution this should never be considered a security measure for protecting a user's Frigate instance. The robots.txt file is simply a polite request that malicious actors will completely ignore, and since it's publicly accessible, it may actually reveal that you're running a Frigate instance. It provides zero protection against unauthorized access attempts, brute force attacks, or API exploitation. Real security requires strong authentication, HTTPS/TLS encryption, network-level protection through VPN or firewall rules, regular security updates, and proper reverse proxy configuration with additional authentication layers. While the robots.txt file serves a legitimate privacy purpose by keeping a Frigate instance out of search results, we can't let it create a false sense of security. If a user's Frigate instance is exposed to the internet, they must implement proper security controls beyond just asking crawlers to stay away.

[ZhaiSoul]

Yes, I understand that. However, information leakage caused by search engine crawling is the easiest attack vector to exploit - anyone with basic search engine skills can carry out such attacks. While this measure can't completely prevent data breaches, it should serve as a fundamental safeguard. I'll be submitting additional security-focused PRs in the future.

[hawkeye217]

As we've said in the past, we always suggest users open a discussion before spending time on major changes - which would include any architectural and security modifications - to ensure any changes align with the direction of the project.