[Bug] Issues encountered while running an HTTP server with Gramine-SGX.

[LeoneChen]

Hello, I'm running nginx and lighttpd over Gramine-SGX to host an HTTP server. I'm facing the following bugs, and point out the code that needs to be patched (may be helpful to other users):

nginx

Shared memory and file offset issues between parent and child processes. so we need to patch nginx to make it work.

1. accept_mutex not work

Nginx uses accept_mutex to prevent the thundering herd problem, where multiple worker processes simultaneously try to handle new requests, sometimes causing performance issues. Related codes as below:

1. Shared memory is used to create ngx_accept_mutex.

static ngx_int_t
ngx_event_module_init(ngx_cycle_t *cycle)
{
    if (ngx_shm_alloc(&shm) != NGX_OK) {
        return NGX_ERROR;
    }

    shared = shm.addr;

    ngx_accept_mutex_ptr = (ngx_atomic_t *) shared;
    ngx_accept_mutex.spin = (ngx_uint_t) -1;

    if (ngx_shmtx_create(&ngx_accept_mutex, (ngx_shmtx_sh_t *) shared,
                         cycle->lock_file.data)
        != NGX_OK)
    {
        return NGX_ERROR;
    }
}

2. Nginx uses accept_mutex to prevent worker race to serve.

void
ngx_process_events_and_timers(ngx_cycle_t *cycle)
{
    if (ngx_use_accept_mutex) {
        if (ngx_accept_disabled > 0) {
            ngx_accept_disabled--;

        } else {
            if (ngx_trylock_accept_mutex(cycle) == NGX_ERROR) {
                return;
            }

            if (ngx_accept_mutex_held) {
                flags |= NGX_POST_EVENTS;

            } else {
                if (timer == NGX_TIMER_INFINITE
                    || timer > ngx_accept_mutex_delay)
                {
                    timer = ngx_accept_mutex_delay;
                }
            }
        }
    }
    /* here to process request */
    if (ngx_accept_mutex_held) {
        ngx_shmtx_unlock(&ngx_accept_mutex);
    }
}

2. Connection limit not work

Worker processes share connection statistics via shared memory. However, due to Gramine's limitations with shared memory, the number of connections might exceed the configured limit.

Each process can ensure the connection limit isn't exceeded, but we lose the overall limit control.

(ngx_stream_limit_conn_handler has same problem as ngx_http_limit_conn_handler)

static ngx_int_t
ngx_http_limit_conn_handler(ngx_http_request_t *r)
{
    for (i = 0; i < lccf->limits.nelts; i++) {
        node = ngx_http_limit_conn_lookup(&ctx->sh->rbtree, &key, hash);
        if (node == NULL) {
            /* Share connection statistics using shared memory. */
            node = ngx_slab_alloc_locked(ctx->shpool, n);
        } else {
            /* Process when the number of connections exceeds the limit  */
            if ((ngx_uint_t) lc->conn >= limits[i].conn) {

                ngx_shmtx_unlock(&ctx->shpool->mutex);

                ngx_log_error(lccf->log_level, r->connection->log, 0,
                              "limiting connections%s by zone \"%V\"",
                              lccf->dry_run ? ", dry run," : "",
                              &limits[i].shm_zone->shm.name);

                ngx_http_limit_conn_cleanup_all(r->pool);

                r->main->limit_conn_status = NGX_HTTP_LIMIT_CONN_REJECTED;

                return lccf->status_code;
            }

            lc->conn++;
        }
    }
}

3. Cache limit not work

Workers should share cache limit, but it fail to do so.

static ngx_msec_t
ngx_http_file_cache_manager(void *data)
{
    for ( ;; ) {
        ngx_shmtx_lock(&cache->shpool->mutex);

        size = cache->sh->size;
        count = cache->sh->count;
        watermark = cache->sh->watermark;

        ngx_shmtx_unlock(&cache->shpool->mutex);

        ngx_log_debug3(NGX_LOG_DEBUG_HTTP, ngx_cycle->log, 0,
                       "http file cache size: %O c:%ui w:%i",
                       size, count, (ngx_int_t) watermark);

        if (size < cache->max_size && count < watermark) {
            if (!cache->min_free) {
                break;
            }

            free = ngx_fs_available(cache->path->name.data);

            ngx_log_debug1(NGX_LOG_DEBUG_HTTP, ngx_cycle->log, 0,
                           "http file cache free: %O", free);

            if (free > cache->min_free) {
                break;
            }
        }
    }
}

Attacker can hide log

In nginx, after the parent process opens a log file, it forks worker processes that should share a file offset (or called cursor/pos) for coordinated writing.

But this offset is isolated by Enclave in Gramine-SGX, workers worked as it had exclusive access to the log file, so we can see worker B‘s log overwrite worker A's log, due to file offset can not be correctly shared.

2025/08/14 13:38:38 [debug] 1#0: bind() 0.0.0.0:80 #6 
2025/08/14 13:38:38 [notice] 1#0: using the "epoll" event method
2025/08/14 13:38:38 [debug] 1#0: counter: 0000000004EF5080, 1
2025/08/14 13:38:38 [notice] 1#0: nginx/1.27.4
2025/08/14 13:38:38 [notice] 1#0: built by clang 18.1.8 (git@github.com:LeoneChen/llvm-project.git 50d2edd53bfb24337d312bb44d491fc4a6bb5bc5)
2025/08/14 13:38:38 [notice] 1#0: OS: Linux 3.10.0
2025/08/14 13:38:38 [notice] 1#0: getrlimit(RLIMIT_NOFILE): 900:65536
2025/08/14 13:38:38 [debug] 1#0: write: 7, 000000000678B650, 2, 0
2025/08/14 13:38:38 [debug] 1#0: setproctitle: "nginx: master process /home/leone/SGXDiff/install-dir/"
2025/08/14 13:38:38 [notice] 1#0: start worker processes
2025/08/14 13:38:38 [debug] 1#0: channel 3:7
2025/08/14 13:38:38 [notice] 1#0: start worker process 2
2025/08/14 13:38:38 [debug] 1#0: channel 8:9
2025/08/14 13:38:38 [debug] 3#0: add cleanup: 0000000004E2025/02025/08/14 13:38:38 [debug] 3#0: malloc: 0000000008507EB0:8
2025/08/14 13:38:38 [debug] 3#0: notify eventfd: 11
2025/08/14 13:38:38 [debug] 3#0: testing the EPOLLRDHUP flag: success
2025/08/14 13:38:38 [debug] 3#0: malloc: 00000000084F4C90:6144
2025/08/14 13:38:38 [debug] 3#0: malloc: 0000000004EFC440:114688
2025/08/14 13:38:38 [debug] 3#0: malloc: 0000000004F18450:49152
2025/08/14 13:38:38 [debug] 3#0: malloc: 0000000004F24460:49152
2025/08/14 13:38:38 [debug] 3#0: epoll add event: fd:9 op:1 ev:00002001
2025/08/14 13:38:38 [debug] 3#0: setproctitle: "nginx: worker process"
2025/08/14 13:38:38 [debug] 3#0: worker cycle
2025/08/14 13:38:38 [debug] 3#0: accept mutex locked
2025/08/14 13:38:32025/08/14 13:38:54 [debug] 2#0: epoll: fd:6 ev:0001 d:0000000004EFC440
##### We can see here worker B's log overwrites worker A's log. #####
2025/08/14 13:38:54 [debug] 2#0: post event 0000000004F18450
0: epoll: fd:6 ev:0001 d:0000000004EFC440

lighttpd

1. ETag mismatch

Gramine-SGX uses the file path to calculate st_ino, because it is difficult for userspace to get inode information.

But in lighttpd, it uses st_ino to calculate ETag. When a file is renamed in lighttpd, st_ino and ETag may be different from the original file, thus invalidating the cache.

st_ino is used in many places in lighttpd, so we need to patch them all.

void
http_etag_create (buffer * const etag, const struct stat * const st, const int flags)
{
    if (flags & ETAG_USE_INODE)
        x[len++] = (uint64_t)st->st_ino;

    http_etag_remix(etag, (char *)x, len << 3);
}

[kailun-qin]

@LeoneChen:
Thank you for the detailed report!

May I know how you found these issues? Were you using our Nginx/lighttpd CI-Examples? Any specific Nginx/lighttpd configuration did you use when you ran into these issues?

[LeoneChen]

Hi @kailun-qin:

May I know how you found these issues?

I instrument the Nginx and Gramine to trace the shared memory/file metadata operations, and perform an offline analysis with a Python script. I will publish these tools in the near future, and share a link to this issue (I'm systematically summarizing differences and analyzing affected code patterns to help automate detection, I think this is helpful for Gramine's ecosystem).

Were you using our Nginx/lighttpd CI-Examples?

Almost same as the manifest file in CI-Examples/nginx. insecure__use_cmdline_argv helps me to adjust when testing, loader.env.{AFL_MAP_DUMP_PATH/LLVM_PROFILE_FILE/SLSAN_LOG_PREFIX} helps me to perform some analysis.

libos.entrypoint = "{{ install_dir }}/sbin/nginx"

loader.insecure__use_cmdline_argv = true
loader.log_level = "{{ log_level }}"
loader.env.LD_LIBRARY_PATH = "/lib:{{ arch_libdir }}"
loader.env.AFL_MAP_DUMP_PATH = "/log/gramine.map"
loader.env.LLVM_PROFILE_FILE = "/log/gramine%9m.profraw"
loader.env.SLSAN_LOG_PREFIX = "/log/"

# Gramine by default reports the root user (uid = 0) to applications. This default behavior forces
# Nginx to attempt to drop privileges and assume some non-privileged user ID via setuid(), which is
# meaningless in Gramine. Instead, we set up a dummy user (uid = 1000); this is irrelevant for
# consequent Nginx execution anyway.
loader.uid = 1000
loader.gid = 1000

sys.enable_sigterm_injection = true

fs.mounts = [
  { path = "/lib", uri = "file:{{ gramine.runtimedir() }}" },
  { path = "{{ arch_libdir }}", uri = "file:{{ arch_libdir }}" },
  { path = "/log/", uri = "file:log/" },
  { path = "/tmp/", uri = "file:/tmp/" },
  { path = "{{ install_dir }}", uri = "file:{{ install_dir }}" },
]

sgx.debug = true
sgx.edmm_enable = {{ 'true' if env.get('EDMM', '0') == '1' else 'false' }}
sgx.enclave_size = "512M"
sgx.max_threads = {{ '1' if env.get('EDMM', '0') == '1' else '4' }}

# `use_exinfo = true` is needed because the application may trigger lazy allocation of pages
# (through exception handling) when EDMM is enabled
sgx.use_exinfo = {{ 'true' if env.get('EDMM', '0') == '1' else 'false' }}

sgx.trusted_files = [
  "file:{{ gramine.runtimedir() }}/",
  "file:{{ arch_libdir }}/",
  "file:{{ install_dir }}/",
]

# For simplicity, Nginx logs are not protected. In production, you must either disable logs in Nginx
# or mount them as "encrypted" files.
sgx.allowed_files = [
  "file:log/",
  "file:/tmp/",
  "file:{{ install_dir }}/logs/",
  "file:{{ install_dir }}/cache/",
  "file:{{ install_dir }}/conf/",
]

Any specific Nginx/lighttpd configuration did you use when you ran into these issues?

I add lines below to Nginx configuration file when dynamic testing.

daemon off;
worker_processes  2;
error_log  logs/error.log  debug;
http {
    # shm not support
    limit_conn_zone $binary_remote_addr zone=addr:10m;
    server {
        limit_conn addr 10;
    }
}

Cache limit in Nginx and ETag mismatch in Lighttpd currently are found by static analysis.

[gstrauss]

lighttpd

1. ETag mismatch

Gramine-SGX uses the file path to calculate st_ino, because it is difficult for userspace to get inode information.

But in lighttpd, it uses st_ino to calculate ETag. When a file is renamed in lighttpd, st_ino and ETag may be different from the original file, thus invalidating the cache.

st_ino is used in many places in lighttpd, so we need to patch them all.

@LeoneChen what do you mean?
What you describe is "working as intended" in lighttpd. If you do not want lighttpd to include st_ino in the ETag calculation, then configure lighttpd.conf with etag.use-inode = "disable" (https://wiki.lighttpd.net/Etag_use-inodeDetails)

[LeoneChen]

@gstrauss
The point is that no one was aware of it until it was pointed out.

What you say is a direct countermeasures for ETag. However, st_ino is not only used to calculate ETag in lighttpd.