r=3 object store fails to accept large files #1908
Description
Observed behavior
Can not put files larger than a few hundred MBs to r=3 object store. Not sure if it is a natscli issue though...
Expected behavior
Either can put reliably, or disallow creating r=3 object stores?
Server and client version
server main (nats-io/nats-server@74a7d88)
# nats --version
v0.2.4
Host environment
aws cluster deployed with:
# faber create-cluster abozhenko-test --aws=3 --aws-instance-type="c7a.2xlarge" --disk-size=30 -v
Config with mostly defaults:
$ cat /etc/nats.conf
port: 4222
client_advertise: IP:4222
debug: true
trace: false
monitor_port: 8222
prof_port: 65432
server_name: "faber-aws-alexbozhenko-abozhenko-test-07a3a0-2"
server_tags: ["cloud:aws", "region:us-east-1"]
cluster {
port: 6222
name: faberCluster
routes = [
# 3 node cluster IPs here
]
}
gateway: {
name: faberCluster,
port: 7222,
connect_retries: 5
gateways: [
{
name: faberCluster,
urls: [
# IPs here
]
},
]
}
jetstream {
store_dir: /data/nats
}
leafnodes {
port: 7422
}
accounts: {
SYS: {
users: [
{user: sys, password: sys}
]
},
}
system_account: SYS
pid_file: /tmp/nats.pid
Steps to reproduce
dd if=/dev/random of=file bs=1M count=1024
nats obj add test3 --replicas=3
When I try to put this large file to a r=3 object store, I get:
# nats obj put test3 file
50.17% ║███████████████████████░░░░░░░░░░░░░░░░░░░░░░░║ [514 MiB in 2.001s; 257 MiB/s]
nats: error: nats: stalled with too many outstanding async published messages
or
# nats obj put test3 file
51.98% ║███████████████████████▓░░░░░░░░░░░░░░░░░░░░░░║ [532 MiB in 2.251s; 236 MiB/s]
nats: error: nats: API error: code=429 err_code=10167 description=too many requests
or simply exit status 1, without indication what was happening:
# nats obj put test3 file
47.90% ║██████████████████████░░░░░░░░░░░░░░░░░░░░░░░░║ [490 MiB in 2.001s; 245 MiB/s]
47.90% ║██████████████████████░░░░░░░░░░░░░░░░░░░░░░░░║ [490 MiB in 2.054s; 239 MiB/s]
[❌ ERROR]
Also, when I dropped file size to 500MB, I got both done and error:
# nats obj put test3 file -f
done! [500 MiB in 911ms; 499 MiB/s]
nats: error: nats: API error: code=429 err_code=10167 description=too many requests