Document apparmor

[tresf]

I'm trying to fix our deployment script however @liushuyu it appears that by default apparmor is enabled, but it's not obvious at all what the settings are or how to fix it.

This post describes it best: https://www.reddit.com/r/openSUSE/comments/kanlom/phpfpm_is_blocked_by_apparmor/

Quoting:

I saw no indication of AppArmor being an issue or having any involvement, and spent about 3 hours trying various FastCGI snippet options and PHP-FPM connection options, changing folder ownerships/permissions, and looking at logs. nginx was reporting that FastCGI was reporting permission denied, and nothing more specific than that. Not even 777'ing the entire folder fixed it.

One of the commenters says that they wasted 12 hours on this problem. I believe it.

For now, I'm using:

# TOGGLE OFF
sudo aa-complain /etc/apparmor.d/php-fpm
# TOGGLE ON
sudo aa-enforce /etc/apparmor.d/php-fpm

... but what does this even do? The above article says to use sudo aa-logprof to "Allow or deny whatever AppArmor would have blocked", but that seems a bit presumptuous without reviewing them.

I think reviewing syslog is the way:

 sudo tail -f /var/log/syslog |grep audit

... but the events seem to vanish after complain has been toggled on and off. How do I actually allow these? Is there a procedure that isn't so esoteric?

[tresf]

I just had to deploy a PR five times changing to complain and back to enforce to dance around this apparmor nonsense. @liushuyu any help here would be greatly appreciated. This is making it much harder to make website changes.

[liushuyu]

I'm trying to fix our deployment script however @liushuyu it appears that by default apparmor is enabled, but it's not obvious at all what the settings are or how to fix it.

The local settings are at /etc/apparmor.d/local/. I have already created a php-fpm rule there. A system upgrade will override any rule file outside of /etc/apparmor.d/local/ next time.

So AppArmor is enabled for security isolation purposes, as we have encountered directory traversal exploits on the server before.

This post describes it best: https://www.reddit.com/r/openSUSE/comments/kanlom/phpfpm_is_blocked_by_apparmor/

Quoting:

I saw no indication of AppArmor being an issue or having any involvement, and spent about 3 hours trying various FastCGI snippet options and PHP-FPM connection options, changing folder ownerships/permissions, and looking at logs. nginx was reporting that FastCGI was reporting permission denied, and nothing more specific than that. Not even 777'ing the entire folder fixed it.

One of the commenters says that they wasted 12 hours on this problem. I believe it.

For now, I'm using:

TOGGLE OFF

sudo aa-complain /etc/apparmor.d/php-fpm

TOGGLE ON

sudo aa-enforce /etc/apparmor.d/php-fpm

... but what does this even do? The above article says to use sudo aa-logprof to "Allow or deny whatever AppArmor would have blocked", but that seems a bit presumptuous without reviewing them.

aa-complain will switch AppArmor to complaint mode, which disables the rule enforcement but prints any rule violations to the logs.
sudo aa-logprof will prompt you to select what rules to add to the ruleset, so you will still review them if I remember correctly.

I think reviewing syslog is the way:

sudo tail -f /var/log/syslog |grep audit

... but the events seem to vanish after complain has been toggled on and off. How do I actually allow these? Is there a procedure that isn't so esoteric?

On a modern Linux system, you can use journalctl -xe to see the saved logs. Also, AppArmor is a kernel module, it will also print its complaints to dmesg as well.

[tresf]

This breaks automatic deployments but going one-by-one through logprof is the wrong way. It took 5 deployments for this nonsense to deploy an update. There must be a way for these to live together in harmony that doesn't involve us monitoring for what each process does and whitelisting it.

[liushuyu]

This breaks automatic deployments but going one-by-one through logprof is the wrong way. It took 5 deployments for this nonsense to deploy an update. There must be a way for these to live together in harmony that doesn't involve us monitoring for what each process does and whitelisting it.

I think turning on complaint mode the first time and deploying it once should give logprof enough information to generate the ruleset. From my experience, the ruleset generated by logprof is usually very restrictive and still requires some human knowledge to relax the rules (e.g., which file should be allowed to be written).

[tresf]

I think turning on complaint mode the first time and deploying it once should give logprof enough information to generate the ruleset. From my experience, the ruleset generated by logprof is usually very restrictive and still requires some human knowledge to relax the rules (e.g., which file should be allowed to be written).

It didn't. I've already stated that twice.

[tresf]

In regards to the security of this system, git-deploy can upload arbitrary code so I'm not sure how this apparmor helps. It doesn't prevent git-deploy from doing anything specific, it just prevents it from doing anything at all. It's not securing the git-deploy product, just making us jump through hoops for it to work as designed by running it over and over until we have an idea of all the system calls it makes and hope that nothing changes the next time we deploy.

[liushuyu]

I think turning on complaint mode the first time and deploying it once should give logprof enough information to generate the ruleset. From my experience, the ruleset generated by logprof is usually very restrictive and still requires some human knowledge to relax the rules (e.g., which file should be allowed to be written).

It didn't. I've already stated that twice.

I see. So the issue is still logprof did not generate the correct ruleset?

... but the events seem to vanish after complain has been toggled on and off. How do I actually allow these? Is there a procedure that isn't so esoteric?

Also, you don't have to turn off complaint mode to generate the AppArmor rulesets.

I know this is problematic, as with SELinux (used in Fedora/RHEL), these MAC (Mandatory Access Control) systems are very much like enterprise software

[liushuyu]

In regards to the security of this system, git-deploy can upload arbitrary code so I'm not sure how this apparmor helps. It doesn't prevent git-deploy from doing anything specific, it just prevents it from doing anything at all. It's not securing the git-deploy product, just making us jump through hoops for it to work as designed by running it over and over until we have an idea of all the system calls it makes and hope that nothing changes the next time we deploy.

I can agree, but we still have to improve the situation somehow, isn't it? We still review the source repository from which it downloads the code, so this is more of preventing unknown access.

The initial thought was to secure the php-fpm daemon from going rogue (as PHP still has a bad reputation for being difficult to write secure code).
But it does seem like we have some special use cases. I hope we can find a better solution for safely deploying the entire thing. I was working on turning the entire site into a static website and made some progress on that front, but it might not be finished anytime soon due to obstacles in moving the translations.

[tresf]

Also, you don't have to turn off complaint mode to generate the AppArmor rulesets.

This is objectively wrong. Without it, the deployment is in an undefined state, so you'd have to deploy for every apparmor "branch" that gets in your way as you fall to the ground.

I see. So the issue is still logprof did not generate the correct ruleset?

How would I even know? I put it in complain, run aa-logprof, answer all the questions and the next deployment fails and I have to do it again. I think it's working now, but I thought it was working before and clearly I was wrong.

[liushuyu]

Also, you don't have to turn off complaint mode to generate the AppArmor rulesets.

This is objectively wrong. Without it, the deployment is in an undefined state, so you'd have to deploy for every apparmor "branch" that gets in your way as you fall to the ground.

Got it.

I see. So the issue is still logprof did not generate the correct ruleset?

How would I even know? I put it in complain, run aa-logprof, answer all the questions and the next deployment fails and I have to do it again. I think it's working now, but I thought it was working before and clearly I was wrong.

I see. So our course of action is to turn off ArmArmor completely? I think that's fine by me as well.

[tresf]

So our course of action is to turn off ArmArmor completely? I think that's fine by me as well.

If you're comfortable with fixing it for git-deploy, I'm fine with that solution, but in its current state, we don't have a document explaining how to get each service we offer to work with AppArmor.

The issue that I have is this:

We still review the source repository from which it downloads the code, so this is more of preventing unknown access.

The "arbitrary code" I'm referring to is anything in our source tree that can be executed by the nginx user (e.g. php scripts, cron entries) but we have an expectation that that cannot happen unless our actions password is leaked. I am all for a second layer of security, but I just think we need to treat it like a breaking change and make sure we have a reproducible environment, document the steps, and then we can have project steps to reference when we find out we missed something or an updated package or code snippet behaves differently than before. e.g. a living wiki doc.

[tresf]

Upgraded phpBB and hit more apparmor exceptions. Had to put it into complain mode, but leaving an FYI here as the ruleset wasn't saved back into this repo.