NSFS Simple | Nooba container crashes with ERROR] CONSOLE:: PANIC: process uncaughtException TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string

[RobertFloor]

Environment info

• NooBaa Version: 5.17.5 But also all 5.17 and 5.18 version
• Platform: Docker Container

Actual behavior

1. When calling the S3 bucket we get this error and our container crashes

noobaa-1  | Mar-28 15:20:48.760 [nsfs/7] [ERROR] CONSOLE:: PANIC: process uncaughtException TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
noobaa-1  |     at Object.join (node:path:1203:7)
noobaa-1  |     at new ConfigFS (/root/node_modules/noobaa-core/src/sdk/config_fs.js:38:39)
noobaa-1  |     at new AccountSpaceFS (/root/node_modules/noobaa-core/src/sdk/accountspace_fs.js:49:26)
noobaa-1  |     at new NsfsAccountSDK (/root/node_modules/noobaa-core/src/cmd/nsfs.js:228:28)
noobaa-1  |     at init_request_sdk (/root/node_modules/noobaa-core/src/cmd/nsfs.js:365:35)
noobaa-1  |     at Server.endpoint_request_handler (/root/node_modules/noobaa-core/src/endpoint/endpoint.js:286:9)
noobaa-1  |     at Server.emit (node:events:519:28)
noobaa-1  |     at Server.emit (node:domain:488:12)
noobaa-1  |     at parserOnIncoming (node:_http_server:1140:12)
noobaa-1  |     at HTTPParser.parserOnHeadersComplete (node:_http_common:119:17)

Expected behavior

1. Should respond with a no access message, like this

curl -k http://localhost:6001
<?xml version="1.0" encoding="UTF-8"?><Error><Code>AccessDenied</Code><Message>Access Denied</Message><Resource>/</Resource><RequestId>m8sxm30t-87aqce-twf</RequestId></Error>%  

Steps to reproduce

1. Build a docker file like this

FROM noobaa/noobaa-core:5.17.5

EXPOSE 6001

USER root

COPY ./resources/noobaa.sh /

RUN sed -i 's/force_md5_etag: false/force_md5_etag: true/' -- /root/node_modules/noobaa-core/src/cmd/nsfs.js

CMD ["/bin/bash", "noobaa.sh"]

This is the script copies into the container

#!/bin/bash
echo 'starting noobaa'

/usr/local/bin/node /root/node_modules/noobaa-core/src/cmd/nsfs.js /tmp --simple true --access_key=mykey --secret_key=mysecret --backend 'NFSv4'

Then Run

docker build -t nooba-local .  
docker run  --rm -p 6001:6001   nooba-local  

#And check with a simple curl command
curl -k http://localhost:6001 

More information - Screenshots / Logs / Other output

❯ docker run  --rm -p 6001:6001 -p 6443:6443  nooba-local
starting noobaa
Error accrued while getting the data from /etc/noobaa.conf.d/config_dir_redirect: Error: ENOENT: no such file or directory, open '/etc/noobaa.conf.d/config_dir_redirect'
config.get_config_root - could not find custom config_root, will use the default config_root  /etc/noobaa.conf.d
load_nsfs_nc_config.setting config.NSFS_NC_CONF_DIR /etc/noobaa.conf.d
config.load_nsfs_nc_config could not find config.json... skipping
Mar-28 15:24:14.025 [nsfs/7] [EVENT]{"timestamp":"2025-03-28T15:24:14.020Z","host":"d4f3dd7a01d7","event":{"code":"noobaa_gpfslib_missing","message":"Noobaa GPFS library file is missing","description":"Noobaa GPFS library file is missing","entity_type":"NODE","event_type":"STATE_CHANGE","scope":"NODE","severity":"ERROR","state":"DEGRADED","arguments":{},"pid":7}}
nsfs: setting up ... { fs_root: '/tmp', http_port: 6001, https_port: 6443, https_port_sts: -1, https_port_iam: -1, metrics_port: 7004, backend: 'NFSv4', forks: 0, access_key: SENSITIVE-bfeb78ddbd2f6b88, secret_key: SENSITIVE-7b6f7690ae2a5ecd, uid: 0, gid: 0, nsfs_config_root: undefined }
Mar-28 15:24:14.089 [nsfs/7]    [L0] core.endpoint.endpoint:: endpoint: replacing old umask:  22 with new umask:  0
Mar-28 15:24:14.089 [nsfs/7]    [L0] core.endpoint.endpoint:: Configured Virtual Hosts: []
Mar-28 15:24:14.090 [nsfs/7]    [L0] core.endpoint.endpoint:: Configured Location Info: { region: '' }
Mar-28 15:24:14.091 [nsfs/7]  [WARN] CONSOLE:: config.load_nsfs_nc_config could not find config.json... skipping
Mar-28 15:24:14.091 [nsfs/7]   [LOG] CONSOLE:: read_rand_seed: reading 32 bytes from /dev/urandom ...
Mar-28 15:24:14.092 [nsfs/7]   [LOG] CONSOLE:: read_rand_seed: got 32 bytes from /dev/urandom, total 32 ...
Mar-28 15:24:14.092 [nsfs/7]   [LOG] CONSOLE:: read_rand_seed: closing fd ...
Mar-28 15:24:14.093 [nsfs/7]   [LOG] CONSOLE:: init_rand_seed: seeding with 32 bytes
Mar-28 15:24:14.093 [nsfs/7]   [LOG] CONSOLE:: init_rand_seed: done
Mar-28 15:24:14.113 [nsfs/7]    [L0] core.util.ssl_utils:: SSL certificate loaded from dir /etc/s3-secret
Mar-28 15:24:14.116 [nsfs/7]    [L0] core.util.ssl_utils:: SSL certificate not found in dir /etc/sts-secret
Mar-28 15:24:14.116 [nsfs/7]  [WARN] core.util.ssl_utils:: Generating self-signed SSL certificate for STS
Mar-28 15:24:14.310 [nsfs/7]  [WARN] core.util.ssl_utils:: Certificate folder  /etc/sts-secret  does not exist. New certificate won't be loaded.
Mar-28 15:24:14.311 [nsfs/7]    [L0] core.endpoint.endpoint:: Starting S3 HTTP 6001
Mar-28 15:24:14.312 [nsfs/7]    [L0] core.endpoint.endpoint:: Started S3 HTTP successfully
Mar-28 15:24:14.312 [nsfs/7]    [L0] core.endpoint.endpoint:: Starting S3 HTTPS 6443
Mar-28 15:24:14.312 [nsfs/7]    [L0] core.endpoint.endpoint:: Started S3 HTTPS successfully
Mar-28 15:24:14.312 [nsfs/7]    [L0] core.endpoint.endpoint:: Starting metrics server 7004
Mar-28 15:24:14.313 [nsfs/7]    [L0] core.server.analytic_services.prometheus_reporting:: Starting prometheus metrics server on HTTP port 7004
Mar-28 15:24:14.313 [nsfs/7]    [L0] core.endpoint.endpoint:: Started metrics server successfully
Mar-28 15:24:14.313 [nsfs/7]    [L0] core.util.background_scheduler:: Registering semaphore_monitor bg worker
Mar-28 15:24:14.313 [nsfs/7]   [LOG] core.util.background_scheduler:: run_background_worker: INIT semaphore_monitor
Mar-28 15:24:14.313 [nsfs/7] [EVENT]{"timestamp":"2025-03-28T15:24:14.313Z","host":"d4f3dd7a01d7","event":{"code":"noobaa_started","message":"Noobaa started","description":"Noobaa started running","entity_type":"NODE","event_type":"STATE_CHANGE","scope":"NODE","severity":"INFO","state":"HEALTHY","pid":7}}
Mar-28 15:24:14.313 [nsfs/7]    [L0] core.endpoint.endpoint:: Endpoint monitor started
Mar-28 15:24:14.313 [nsfs/7]   [LOG] CONSOLE:: nsfs: listening on 'https://localhost:6443'
Mar-28 15:24:18.490 [nsfs/7] [ERROR] CONSOLE:: PANIC: process uncaughtException TypeError [ERR_INVALID_ARG_TYPE]: The "path" argument must be of type string. Received undefined
    at Object.join (node:path:1203:7)
    at new ConfigFS (/root/node_modules/noobaa-core/src/sdk/config_fs.js:38:39)
    at new AccountSpaceFS (/root/node_modules/noobaa-core/src/sdk/accountspace_fs.js:49:26)
    at new NsfsAccountSDK (/root/node_modules/noobaa-core/src/cmd/nsfs.js:228:28)
    at init_request_sdk (/root/node_modules/noobaa-core/src/cmd/nsfs.js:365:35)
    at Server.endpoint_request_handler (/root/node_modules/noobaa-core/src/endpoint/endpoint.js:286:9)
    at Server.emit (node:events:519:28)
    at Server.emit (node:domain:488:12)
    at parserOnIncoming (node:_http_server:1140:12)
    at HTTPParser.parserOnHeadersComplete (node:_http_common:119:17)

[romayalon]

Hey @RobertFloor
Thanks for opening this issue, we are aware that NSFS simple mode had some issues with the latest NooBaa NC changes.
Until fixing this issue, you can try running without --simple flag, which is also a pretty simple solution for deploying NooBaa.

[RobertFloor]

Hi thanks for the answer

If I change the startup script to not use simple mode (the default)

#!/bin/bash
echo 'starting noobaa'

/usr/local/bin/node /root/node_modules/noobaa-core/src/cmd/nsfs.js /tmp  --access_key=mykey --secret_key=mysecret --backend 'NFSv4'

I get in version noobaa/noobaa-core:5.18.1

❯ docker run -p 6001:6001 nooba-local
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
starting noobaa
Error accrued while getting the data from /etc/noobaa.conf.d/config_dir_redirect: Error: ENOENT: no such file or directory, open '/etc/noobaa.conf.d/config_dir_redirect'
config.get_config_root - could not find custom config_root, will use the default config_root  /etc/noobaa.conf.d
load_nsfs_nc_config.setting config.NSFS_NC_CONF_DIR /etc/noobaa.conf.d
config.load_nsfs_nc_config could not find config.json... skipping
Mar-30 10:54:42.963 [nsfs/7] [EVENT]{"timestamp":"2025-03-30T10:54:42.960Z","host":"3505222ee243","event":{"code":"noobaa_gpfslib_missing","message":"Noobaa GPFS library file is missing","description":"Noobaa GPFS library file is missing","entity_type":"NODE","event_type":"STATE_CHANGE","scope":"NODE","severity":"ERROR","state":"DEGRADED","arguments":{},"pid":7}}
Error: Access key and IAM dir cannot be used together

Help:

    "nsfs" is a noobaa-core command runs a local S3 endpoint on top of a filesystem.
    Each sub directory of the root filesystem represents an S3 bucket.
    Objects data and meta-data is stored and retrieved from the files.
    For more information refer to the noobaa docs.

Usage:

    node src/cmd/nsfs <root-path> [options...]

Arguments:

    <root-path>      Set the root of the filesystem where each subdir is a bucket.

Options:

    --http_port <port>          (default 6001)      Set the S3 endpoint listening HTTP port to serve.
    --https_port <port>         (default 6443)      Set the S3 endpoint listening HTTPS port to serve.
    --https_port_sts <port>     (default -1)        Set the S3 endpoint listening HTTPS port for STS.
    --https_port_iam <port>     (default -1)        Set the endpoint listening HTTPS port for IAM.
    --http_metrics_port <port>      (default 7004)    Set the metrics listening HTTP port for prometheus.
    --https_metrics_port <port>     (default 9443)    Set the metrics listening HTTPS port for prometheus.
    --forks <n>                     (default none)  Forks spread incoming requests (config.ENDPOINT_FORKS used if flag is not provided).
    --debug <level>                 (default 0)     Increase debug level.

    ## single user mode

    --simple <boolean>   (default false)        Starts a single user/fs mode
    --uid <uid>          (default as process)   Send requests to the Filesystem with uid.
    --gid <gid>          (default as process)   Send requests to the Filesystem with gid.
    --access_key <key>      (default none)      Authenticate incoming requests for this access key only (default is no auth).
    --secret_key <key>      (default none)      The secret key pair for the access key.

Do I need to delete a IAM directory?

[romayalon]

@RobertFloor when using the "non-simple" mode you will need to create an account and you will receive a pair of access key and secret key, so you should remove the all the other flags as well - /tmp --access_key=mykey --secret_key=mysecret --backend 'NFSv4'

[RobertFloor]

Hi we just would like to use it as a standalone test container without any account. Therefore we would prefer the simple mode.

[RobertFloor]

Is there a way we could script the account creation in a "non-simple" mode?

[romayalon]

@RobertFloor Yes, there are a few options -

1. The stable method - scripting the execution of creating an account via the CLI - https://github.com/noobaa/noobaa-core/blob/master/docs/NooBaaNonContainerized/NooBaaCLI.md#add-account
2. The new method but still not fully GA'd - via NooBaa's IAM server - https://github.com/noobaa/noobaa-core/blob/master/docs/design/iam.md

Currently, we don't have a REST API for creating accounts but we might want to support it in the future.

[RobertFloor]

Thanks for the information, I was busy with other projects, but now I have time for this. I will keep you updated.

[RobertFloor]

@romayalon is the nooba-cli installed by default in the container noobaa/noobaa-core:5.18.3? I cannot find it in the container?
Ok Edit I found this solution

adduser test
/usr/local/bin/node /root/node_modules/noobaa-core/src/cmd/manage_nsfs.js account add --name test --user test --allow_bucket_creation --access_key aaaaaaaaaaaaaaaaaaaa --secret_key aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
/usr/local/bin/node /root/node_modules/noobaa-core/src/cmd/nsfs /tmp --config_root   /etc/noobaa.conf.d/

So the only problem is that our old test access_key and secret_key in our test environment are

--access_key=mykey --secret_key=mysecret

However this seems impossible, while it was possible in the simple mode. Is it possible to disable this requirement, it is only a test setup?

{
  "error": {
    "code": "InvalidAccountAccessKeyFlag",
    "message": "Account access key length must be 20, and must contain only alpha-numeric chars"
  }
}

[romayalon]

Hey @RobertFloor
Do you see the noobaa-cli file in the following location? - /usr/local/bin/noobaa-cli
How did you install NooBaa? via rpm?
Unfortunately, non AWS compatible access key/secret key are not supported via noobaa-cli.

[RobertFloor]

Hi @romayalon

Thanks for the help! I am nearly there
I am using the docker image from docker hub noobaa/noobaa-core:5.18.3

I do these steps to configure an account

adduser test
/usr/local/bin/node /root/node_modules/noobaa-core/src/cmd/manage_nsfs.js account add --name test2 --user test2 --allow_bucket_creation --access_key aaaaaaaaaaaaaaaaaaaa --secret_key aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa
/usr/local/bin/node /root/node_modules/noobaa-core/src/cmd/nsfs  /tmp/nooba/ 

And I use s3cmd to test the connection

s3cmd --configure

Test access with supplied credentials? [Y/n] Y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to '/home/xxx/.s3cfg'

❯ s3cmd  mb s3://test
ERROR: S3 error: 403 (InvalidAccessKeyId): The AWS Access Key Id you provided does not exist in our records.

And the noobaa container says

ay-26 10:39:01.964 [nsfs/7] [WARN] core.sdk.config_fs:: get_config_data: with config_file_path /etc/noobaa.conf.d/buckets/test.json got an error [Error: No such file or directory] { code: 'ENOENT', context: 'Readfile _path=/etc/noobaa.conf.d/buckets/test.json ' }


May-26 10:39:01.964 [nsfs/7] [EVENT]{"timestamp":"2025-05-26T10:39:01.964Z","host":"31aa9d1f8b76","event":{"code":"noobaa_no_such_bucket","message":"Bucket not found, value: test","description":"Bucket not found, error: Error: No such file or directory","entity_type":"NODE","event_type":"ERROR","scope":"NODE","severity":"ERROR","state":"HEALTHY","arguments":{"bucket_name":"test"},"pid":7}}


May-26 10:39:01.964 [nsfs/7] [ERROR] core.endpoint.s3.s3_rest:: S3 ERROR <?xml version="1.0" encoding="UTF-8"?><Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist.</Message><Resource>/test/?location</Resource><RequestId>mb4yg66h-b459x5-15ni</RequestId></Error> GET /test/?location {"host":"localhost:6443","accept-encoding":"identity","content-length":"0","x-amz-date":"20250526T103901Z","authorization":"AWS4-HMAC-SHA256 Credential=aaaaaaaaaaaaaaaaaaaa/20250526/us-east-1/s3/aws4_request,SignedHeaders=host;x-amz-content-sha256;x-amz-date,Signature=1701c3e5640626f398047b4b16595dd9c99790a38f33b27983b210ec80a67c3e","x-amz-content-sha256":"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"} Error: No such file or directory - context: Readfile _path=/etc/noobaa.conf.d/buckets/test.json


May-26 10:39:01.965 [nsfs/7] [L0] core.sdk.bucketspace_fs:: BucketSpaceFS.read_bucket_sdk_info: bucket name test


May-26 10:39:01.965 [nsfs/7] [WARN] core.sdk.config_fs:: get_config_data: with config_file_path /etc/noobaa.conf.d/buckets/test.json got an error [Error: No such file or directory] { code: 'ENOENT', context: 'Readfile _path=/etc/noobaa.conf.d/buckets/test.json ' }


May-26 10:39:01.966 [nsfs/7] [EVENT]{"timestamp":"2025-05-26T10:39:01.965Z","host":"31aa9d1f8b76","event":{"code":"noobaa_no_such_bucket","message":"Bucket not found, value: test","description":"Bucket not found, error: Error: No such file or directory","entity_type":"NODE","event_type":"ERROR","scope":"NODE","severity":"ERROR","state":"HEALTHY","arguments":{"bucket_name":"test"},"pid":7}}


May-26 10:39:01.966 [nsfs/7] [WARN] core.endpoint.s3.s3_bucket_logging:: send_bucket_op_logs of bucket test got an error: [Error: No such file or directory] { code: 'ENOENT', context: 'Readfile _path=/etc/noobaa.conf.d/buckets/test.json ', rpc_code: 'NO_SUCH_BUCKET' }

[romayalon]

Hi @RobertFloor
The logs are showing NoSuchBucket error on GET bucket operation, I don't think it's on the make bucket request.
Do you see InvalidAccessKeyId in the logs?

[RobertFloor]

Yes I do see that in the Client logs, the strange thing is that s3cmd configure works. I will try was cli