🐛 Bug Report: Response stops during answering (external inference engine) #1863
Description
📜 Description
We are running an external inference engine for performance reasons (we need DeepSeek-R1 in a confidential setting) using ik_llama.cpp a, more performant and feature-rich, fork of llama.cpp. ik_llama.cpp does perform as expected with it's internal server as well as OpenWebUI. It is a large model (350GB) that is currently being run without GPU acceleration so the processing speed is rather slow.
We deployed DocsGPT for initial testing purposes by using the following command:
docker compose --env-file .env -f deployment/docker-compose.yaml up
However after we send a promt to DocsGPT it stops processing and still abruptly stops during the thinking-process (please see attached screenshot for reference).
Please also see attached logs (of docker-compose up [without -d])
👟 Reproduction steps
Start ik_llama.cpp with large model, start DocsGPT (using docker-compose)
👍 Expected behavior
DocsGPT should wait for answer to be processed, thinking process and answer to finish.
👎 Actual Behavior with Screenshots
💻 Operating system
Linux
What browsers are you seeing the problem on?
Firefox
🤖 What development environment are you experiencing this bug on?
Docker
🔒 Did you set the correct environment variables in the right path? List the environment variable names (not values please!)
API_KEY
LLM_NAME
LLM_PROVIDER
OPENAI_BASE_URL
EMBEDDINGS_NAME
VITE_API_STREAMING
📃 Provide any additional context for the Bug.
No response
📖 Relevant log output
Attaching to backend-1, frontend-1, mongo-1, redis-1, worker-1
redis-1 | 1:C 26 Jun 2025 08:25:58.768 # oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo
redis-1 | 1:C 26 Jun 2025 08:25:58.768 # Redis version=6.2.18, bits=64, commit=00000000, modified=0, pid=1, just started
redis-1 | 1:C 26 Jun 2025 08:25:58.768 # Warning: no config file specified, using the default config. In order to specify a config file use redis-server /path/to/redis.conf
redis-1 | 1:M 26 Jun 2025 08:25:58.768 * monotonic clock: POSIX clock_gettime
redis-1 | 1:M 26 Jun 2025 08:25:58.769 * Running mode=standalone, port=6379.
redis-1 | 1:M 26 Jun 2025 08:25:58.769 # Server initialized
redis-1 | 1:M 26 Jun 2025 08:25:58.769 # WARNING Memory overcommit must be enabled! Without it, a background save or replication may fail under low memory condition. Being disabled, it can can also cause failures without low memory condition, see https://github.com/jemalloc/jemalloc/issues/1328. To fix this issue add 'vm.overcommit_memory = 1' to /etc/sysctl.conf and then reboot or run the command 'sysctl vm.overcommit_memory=1' for this to take effect.
redis-1 | 1:M 26 Jun 2025 08:25:58.769 * Loading RDB produced by version 6.2.18
redis-1 | 1:M 26 Jun 2025 08:25:58.769 * RDB age 366 seconds
redis-1 | 1:M 26 Jun 2025 08:25:58.769 * RDB memory usage when created 0.78 Mb
redis-1 | 1:M 26 Jun 2025 08:25:58.769 # Done loading RDB, keys loaded: 2, keys expired: 0.
redis-1 | 1:M 26 Jun 2025 08:25:58.770 * DB loaded from disk: 0.000 seconds
redis-1 | 1:M 26 Jun 2025 08:25:58.770 * Ready to accept connections
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.779+00:00"},"s":"I", "c":"NETWORK", "id":4915701, "ctx":"-","msg":"Initialized wire specification","attr":{"spec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":17},"outgoing":{"minWireVersion":6,"maxWireVersion":17},"isInternalClient":true}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.780+00:00"},"s":"I", "c":"CONTROL", "id":23285, "ctx":"-","msg":"Automatically disabling TLS 1.0, to force-enable TLS 1.0 specify --sslDisabledProtocols 'none'"}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.781+00:00"},"s":"I", "c":"NETWORK", "id":4648601, "ctx":"main","msg":"Implicit TCP FastOpen unavailable. If TCP FastOpen is required, set tcpFastOpenServer, tcpFastOpenClient, and tcpFastOpenQueueSize."}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.782+00:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationDonorService","namespace":"config.tenantMigrationDonors"}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.782+00:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"TenantMigrationRecipientService","namespace":"config.tenantMigrationRecipients"}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.782+00:00"},"s":"I", "c":"REPL", "id":5123008, "ctx":"main","msg":"Successfully registered PrimaryOnlyService","attr":{"service":"ShardSplitDonorService","namespace":"config.tenantSplitDonors"}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.782+00:00"},"s":"I", "c":"CONTROL", "id":5945603, "ctx":"main","msg":"Multi threading initialized"}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.783+00:00"},"s":"I", "c":"CONTROL", "id":4615611, "ctx":"initandlisten","msg":"MongoDB starting","attr":{"pid":1,"port":27017,"dbPath":"/data/db","architecture":"64-bit","host":"3e3ab422d759"}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.783+00:00"},"s":"I", "c":"CONTROL", "id":23403, "ctx":"initandlisten","msg":"Build Info","attr":{"buildInfo":{"version":"6.0.24","gitVersion":"1b052b94a23863fd12be97aaa4e4b1d96456e5cc","openSSLVersion":"OpenSSL 3.0.2 15 Mar 2022","modules":[],"allocator":"tcmalloc","environment":{"distmod":"ubuntu2204","distarch":"x86_64","target_arch":"x86_64"}}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.783+00:00"},"s":"I", "c":"CONTROL", "id":51765, "ctx":"initandlisten","msg":"Operating System","attr":{"os":{"name":"Ubuntu","version":"22.04"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.783+00:00"},"s":"I", "c":"CONTROL", "id":21951, "ctx":"initandlisten","msg":"Options set by command line","attr":{"options":{"net":{"bindIp":"*"}}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.784+00:00"},"s":"I", "c":"STORAGE", "id":22270, "ctx":"initandlisten","msg":"Storage engine to use detected by data files","attr":{"dbpath":"/data/db","storageEngine":"wiredTiger"}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.784+00:00"},"s":"I", "c":"STORAGE", "id":22315, "ctx":"initandlisten","msg":"Opening WiredTiger","attr":{"config":"create,cache_size=257109M,session_max=33000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,remove=true,path=journal,compressor=snappy),builtin_extension_config=(zstd=(compression_level=6)),file_manager=(close_idle_time=600,close_scan_interval=10,close_handle_minimum=2000),statistics_log=(wait=0),json_output=(error,message),verbose=[recovery_progress:1,checkpoint_progress:1,compact_progress:1,backup:0,checkpoint:0,compact:0,evict:0,history_store:0,recovery:0,rts:0,salvage:0,tiered:0,timestamp:0,transaction:0,verify:0,log:0],"}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.915+00:00"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926358,"ts_usec":914995,"thread":"1:0x7f0764620cc0","session_name":"txn-recover","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"Recovering log 37 through 38"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.938+00:00"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926358,"ts_usec":938089,"thread":"1:0x7f0764620cc0","session_name":"txn-recover","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"Recovering log 38 through 38"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:58.994+00:00"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926358,"ts_usec":994697,"thread":"1:0x7f0764620cc0","session_name":"txn-recover","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"Main recovery loop: starting at 37/20352 to 38/256"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.045+00:00"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926359,"ts_usec":45058,"thread":"1:0x7f0764620cc0","session_name":"txn-recover","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"Recovering log 37 through 38"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.079+00:00"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926359,"ts_usec":79813,"thread":"1:0x7f0764620cc0","session_name":"txn-recover","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"Recovering log 38 through 38"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.105+00:00"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926359,"ts_usec":105828,"thread":"1:0x7f0764620cc0","session_name":"txn-recover","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"recovery log replay has successfully finished and ran for 191 milliseconds"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.105+00:00"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926359,"ts_usec":105929,"thread":"1:0x7f0764620cc0","session_name":"txn-recover","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"Set global recovery timestamp: (0, 0)"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.105+00:00"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926359,"ts_usec":105941,"thread":"1:0x7f0764620cc0","session_name":"txn-recover","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"Set global oldest timestamp: (0, 0)"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.106+00:00"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926359,"ts_usec":106375,"thread":"1:0x7f0764620cc0","session_name":"txn-recover","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"recovery rollback to stable has successfully finished and ran for 0 milliseconds"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.107+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926359,"ts_usec":107010,"thread":"1:0x7f0764620cc0","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 1, snapshot max: 1 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.117+00:00"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926359,"ts_usec":117271,"thread":"1:0x7f0764620cc0","session_name":"txn-recover","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"recovery checkpoint has successfully finished and ran for 10 milliseconds"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.117+00:00"},"s":"I", "c":"WTRECOV", "id":22430, "ctx":"initandlisten","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926359,"ts_usec":117425,"thread":"1:0x7f0764620cc0","session_name":"txn-recover","category":"WT_VERB_RECOVERY_PROGRESS","category_id":30,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"recovery was completed successfully and took 202ms, including 191ms for the log replay, 0ms for the rollback to stable, and 10ms for the checkpoint."}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.121+00:00"},"s":"I", "c":"STORAGE", "id":4795906, "ctx":"initandlisten","msg":"WiredTiger opened","attr":{"durationMillis":337}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.121+00:00"},"s":"I", "c":"RECOVERY", "id":23987, "ctx":"initandlisten","msg":"WiredTiger recoveryTimestamp","attr":{"recoveryTimestamp":{"$timestamp":{"t":0,"i":0}}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.130+00:00"},"s":"W", "c":"CONTROL", "id":22120, "ctx":"initandlisten","msg":"Access control is not enabled for the database. Read and write access to data and configuration is unrestricted","tags":["startupWarnings"]}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.130+00:00"},"s":"W", "c":"CONTROL", "id":22178, "ctx":"initandlisten","msg":"/sys/kernel/mm/transparent_hugepage/enabled is 'always'. We suggest setting it to 'never' in this binary version","tags":["startupWarnings"]}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.130+00:00"},"s":"W", "c":"CONTROL", "id":5123300, "ctx":"initandlisten","msg":"vm.max_map_count is too low","attr":{"currentValue":65530,"recommendedMinimum":2000000,"maxConns":1000000},"tags":["startupWarnings"]}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.131+00:00"},"s":"I", "c":"NETWORK", "id":4915702, "ctx":"initandlisten","msg":"Updated wire specification","attr":{"oldSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"incomingInternalClient":{"minWireVersion":0,"maxWireVersion":17},"outgoing":{"minWireVersion":6,"maxWireVersion":17},"isInternalClient":true},"newSpec":{"incomingExternalClient":{"minWireVersion":0,"maxWireVersion":17},"incomingInternalClient":{"minWireVersion":17,"maxWireVersion":17},"outgoing":{"minWireVersion":17,"maxWireVersion":17},"isInternalClient":true}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.131+00:00"},"s":"I", "c":"REPL", "id":5853300, "ctx":"initandlisten","msg":"current featureCompatibilityVersion value","attr":{"featureCompatibilityVersion":"6.0","context":"startup"}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.131+00:00"},"s":"I", "c":"STORAGE", "id":5071100, "ctx":"initandlisten","msg":"Clearing temp directory"}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.135+00:00"},"s":"I", "c":"CONTROL", "id":20536, "ctx":"initandlisten","msg":"Flow Control is enabled on this deployment"}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.136+00:00"},"s":"I", "c":"FTDC", "id":20625, "ctx":"initandlisten","msg":"Initializing full-time diagnostic data capture","attr":{"dataDirectory":"/data/db/diagnostic.data"}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.138+00:00"},"s":"I", "c":"REPL", "id":6015317, "ctx":"initandlisten","msg":"Setting new configuration state","attr":{"newState":"ConfigReplicationDisabled","oldState":"ConfigPreStart"}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.138+00:00"},"s":"I", "c":"STORAGE", "id":22262, "ctx":"initandlisten","msg":"Timestamp monitor starting"}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.140+00:00"},"s":"I", "c":"NETWORK", "id":23015, "ctx":"listener","msg":"Listening on","attr":{"address":"/tmp/mongodb-27017.sock"}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.140+00:00"},"s":"I", "c":"NETWORK", "id":23015, "ctx":"listener","msg":"Listening on","attr":{"address":"0.0.0.0"}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.140+00:00"},"s":"I", "c":"NETWORK", "id":23016, "ctx":"listener","msg":"Waiting for connections","attr":{"port":27017,"ssl":"off"}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.140+00:00"},"s":"I", "c":"CONTROL", "id":8423403, "ctx":"initandlisten","msg":"mongod startup complete","attr":{"Summary of time elapsed":{"Startup from clean shutdown?":true,"Statistics":{"Transport layer setup":"0 ms","Run initial syncer crash recovery":"0 ms","Create storage engine lock file in the data directory":"0 ms","Create storage engine lock file in the data directory":"0 ms","Get metadata describing storage engine":"0 ms","Get metadata describing storage engine":"0 ms","Validate options in metadata against current startup options":"0 ms","Validate options in metadata against current startup options":"0 ms","Create storage engine":"0 ms","Create storage engine":"338 ms","Write current PID to file":"0 ms","Write current PID to file":"0 ms","Initialize FCV before rebuilding indexes":"0 ms","Initialize FCV before rebuilding indexes":"1 ms","Drop abandoned idents and get back indexes that need to be rebuilt or builds that need to be restarted":"0 ms","Drop abandoned idents and get back indexes that need to be rebuilt or builds that need to be restarted":"0 ms","Rebuild indexes for collections":"0 ms","Rebuild indexes for collections":"0 ms","Build user and roles graph":"0 ms","Set up the background thread pool responsible for waiting for opTimes to be majority committed":"0 ms","Start up the replication coordinator":"1 ms","Start transport layer":"0 ms","_initAndListen total elapsed time":"357 ms"}}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.428+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.18.0.4:33006","uuid":"fd9c9832-1796-4eb5-9b86-01cd89d5ea93","connectionId":1,"connectionCount":1}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.428+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn1","msg":"client metadata","attr":{"remote":"172.18.0.4:33006","client":"conn1","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo|c","version":"4.11.3"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"5.14.0-570.22.1.el9_6.x86_64"},"platform":"CPython 3.12.3.final.0","env":{"container":{"runtime":"docker"}}}}}
frontend-1 |
frontend-1 | > [email protected] dev
frontend-1 | > vite --host
frontend-1 |
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.432+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.18.0.4:33008","uuid":"b43e0cc5-47d3-46b8-9bd5-3970638ab329","connectionId":2,"connectionCount":2}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.433+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn2","msg":"client metadata","attr":{"remote":"172.18.0.4:33008","client":"conn2","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo|c","version":"4.11.3"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"5.14.0-570.22.1.el9_6.x86_64"},"platform":"CPython 3.12.3.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.547+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.18.0.5:36624","uuid":"f3c4118d-ebba-4481-bcbc-7740880cdd73","connectionId":3,"connectionCount":3}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.547+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn3","msg":"client metadata","attr":{"remote":"172.18.0.5:36624","client":"conn3","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo|c","version":"4.11.3"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"5.14.0-570.22.1.el9_6.x86_64"},"platform":"CPython 3.12.3.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.549+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.18.0.5:36636","uuid":"c04b9871-195e-49af-bea1-54b6f5646fd6","connectionId":4,"connectionCount":4}}
mongo-1 | {"t":{"$date":"2025-06-26T08:25:59.549+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn4","msg":"client metadata","attr":{"remote":"172.18.0.5:36636","client":"conn4","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo|c","version":"4.11.3"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"5.14.0-570.22.1.el9_6.x86_64"},"platform":"CPython 3.12.3.final.0","env":{"container":{"runtime":"docker"}}}}}
frontend-1 |
frontend-1 | VITE v6.3.5 ready in 148 ms
frontend-1 |
frontend-1 | ➜ Local: http://localhost:5173/
frontend-1 | ➜ Network: http://172.18.0.6:5173/
backend-1 | [2025-06-26 08:26:04,373] WARNING in user_agent: USER_AGENT environment variable not set, consider setting it to identify your requests.
mongo-1 | {"t":{"$date":"2025-06-26T08:26:04.380+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.18.0.4:52542","uuid":"e22c5d72-7854-43d5-8093-e9dbcf845f44","connectionId":5,"connectionCount":5}}
mongo-1 | {"t":{"$date":"2025-06-26T08:26:04.380+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn5","msg":"client metadata","attr":{"remote":"172.18.0.4:52542","client":"conn5","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo|c","version":"4.11.3"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"5.14.0-570.22.1.el9_6.x86_64"},"platform":"CPython 3.12.3.final.0","env":{"container":{"runtime":"docker"}}}}}
worker-1 | [2025-06-26 08:26:04,546] WARNING in user_agent: USER_AGENT environment variable not set, consider setting it to identify your requests.
mongo-1 | {"t":{"$date":"2025-06-26T08:26:04.552+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.18.0.5:49726","uuid":"dfbc6e0b-8528-461f-a253-e5aeddd28e3b","connectionId":6,"connectionCount":6}}
mongo-1 | {"t":{"$date":"2025-06-26T08:26:04.552+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn6","msg":"client metadata","attr":{"remote":"172.18.0.5:49726","client":"conn6","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo|c","version":"4.11.3"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"5.14.0-570.22.1.el9_6.x86_64"},"platform":"CPython 3.12.3.final.0","env":{"container":{"runtime":"docker"}}}}}
worker-1 | /venv/lib/python3.12/site-packages/celery/platforms.py:829: SecurityWarning: You're running the worker with superuser privileges: this is
worker-1 | absolutely not recommended!
worker-1 |
worker-1 | Please specify a different user using the --uid option.
worker-1 |
worker-1 | User information: uid=0 euid=0 gid=0 egid=0
worker-1 |
worker-1 | warnings.warn(SecurityWarning(ROOT_DISCOURAGED.format(
worker-1 |
worker-1 | -------------- celery@cd59318ceb40 v5.4.0 (opalescent)
worker-1 | --- ***** -----
worker-1 | -- ******* ---- Linux-5.14.0-570.22.1.el9_6.x86_64-x86_64-with-glibc2.39 2025-06-26 08:26:04
worker-1 | - *** --- * ---
worker-1 | - ** ---------- [config]
worker-1 | - ** ---------- .> app: application.celery_init:0x7ff146da9670
worker-1 | - ** ---------- .> transport: redis://redis:6379/0
worker-1 | - ** ---------- .> results: redis://redis:6379/1
worker-1 | - *** --- * --- .> concurrency: 64 (prefork)
worker-1 | -- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
worker-1 | --- ***** -----
worker-1 | -------------- [queues]
worker-1 | .> celery exchange=celery(direct) key=celery
worker-1 |
worker-1 |
worker-1 | [tasks]
worker-1 | . application.api.user.tasks.ingest
worker-1 | . application.api.user.tasks.ingest_remote
worker-1 | . application.api.user.tasks.process_agent_webhook
worker-1 | . application.api.user.tasks.schedule_syncs
worker-1 | . application.api.user.tasks.store_attachment
worker-1 |
worker-1 | [2025-06-26 08:26:21,148] WARNING in warnings: /venv/lib/python3.12/site-packages/celery/worker/consumer/consumer.py:508: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
worker-1 | whether broker connection retries are made during startup in Celery 6.0 and above.
worker-1 | If you wish to retain the existing behavior for retrying connections on startup,
worker-1 | you should set broker_connection_retry_on_startup to True.
worker-1 | warnings.warn(
worker-1 |
worker-1 | [2025-06-26 08:26:21,153] INFO in connection: Connected to redis://redis:6379/0
worker-1 | [2025-06-26 08:26:21,154] WARNING in warnings: /venv/lib/python3.12/site-packages/celery/worker/consumer/consumer.py:508: CPendingDeprecationWarning: The broker_connection_retry configuration setting will no longer determine
worker-1 | whether broker connection retries are made during startup in Celery 6.0 and above.
worker-1 | If you wish to retain the existing behavior for retrying connections on startup,
worker-1 | you should set broker_connection_retry_on_startup to True.
worker-1 | warnings.warn(
worker-1 |
worker-1 | [2025-06-26 08:26:21,156] INFO in mingle: mingle: searching for neighbors
worker-1 | [2025-06-26 08:26:22,163] INFO in mingle: mingle: all alone
frontend-1 | Browserslist: browsers data (caniuse-lite) is 6 months old. Please run:
frontend-1 | npx update-browserslist-db@latest
frontend-1 | Why you should do it regularly: https://github.com/browserslist/update-db#readme
backend-1 | /venv/lib/python3.12/site-packages/pymongo/_csot.py:119: UserWarning: MongoClient opened before fork. May not be entirely fork-safe, proceed with caution. See PyMongo's documentation for details: https://dochub.mongodb.org/core/pymongo-fork-deadlock
backend-1 | return func(self, *args, **kwargs)
mongo-1 | {"t":{"$date":"2025-06-26T08:26:33.561+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.18.0.4:56970","uuid":"ba086e8c-b481-49c1-b3cf-9664ca2a4b1e","connectionId":7,"connectionCount":7}}
mongo-1 | {"t":{"$date":"2025-06-26T08:26:33.561+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn7","msg":"client metadata","attr":{"remote":"172.18.0.4:56970","client":"conn7","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo|c","version":"4.11.3"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"5.14.0-570.22.1.el9_6.x86_64"},"platform":"CPython 3.12.3.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:26:33.563+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.18.0.4:56978","uuid":"a61951c8-2adc-41c6-aa7e-fc4b925beb60","connectionId":8,"connectionCount":8}}
mongo-1 | {"t":{"$date":"2025-06-26T08:26:33.564+00:00"},"s":"I", "c":"NETWORK", "id":22943, "ctx":"listener","msg":"Connection accepted","attr":{"remote":"172.18.0.4:56984","uuid":"d3430654-33da-4012-9283-802c47be9b12","connectionId":9,"connectionCount":9}}
mongo-1 | {"t":{"$date":"2025-06-26T08:26:33.564+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn8","msg":"client metadata","attr":{"remote":"172.18.0.4:56978","client":"conn8","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo|c","version":"4.11.3"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"5.14.0-570.22.1.el9_6.x86_64"},"platform":"CPython 3.12.3.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:26:33.564+00:00"},"s":"I", "c":"NETWORK", "id":51800, "ctx":"conn9","msg":"client metadata","attr":{"remote":"172.18.0.4:56984","client":"conn9","negotiatedCompressors":[],"doc":{"driver":{"name":"PyMongo|c","version":"4.11.3"},"os":{"type":"Linux","name":"Linux","architecture":"x86_64","version":"5.14.0-570.22.1.el9_6.x86_64"},"platform":"CPython 3.12.3.final.0","env":{"container":{"runtime":"docker"}}}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:26:59.131+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926419,"ts_usec":131935,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 3, snapshot max: 3 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
backend-1 | [2025-06-26 08:27:02,818] INFO in routes: /stream - request_data: {'question': 'What is DocsGPT?', 'history': '[{"prompt":"What is DocsGPT?"}]', 'conversation_id': None, 'prompt_id': 'default', 'chunks': '2', 'token_limit': 2000, 'isNoneDoc': True, 'save_conversation': True}, source: {}, attachments: 0
backend-1 | [2025-06-26 08:27:02,885] INFO in logging: Starting activity: stream - 48c90530-0054-49df-9bbb-d863af31ad89 - User: local
backend-1 | [2025-06-26 08:27:03,171] INFO in _client: HTTP Request: POST http://host.docker.internal:8000/v1/chat/completions "HTTP/1.1 200 OK"
mongo-1 | {"t":{"$date":"2025-06-26T08:27:59.146+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926479,"ts_usec":146858,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 6, snapshot max: 6 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:28:59.157+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926539,"ts_usec":157900,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 8, snapshot max: 8 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:29:03.695+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn9","msg":"Connection ended","attr":{"remote":"172.18.0.4:56984","uuid":"d3430654-33da-4012-9283-802c47be9b12","connectionId":9,"connectionCount":8}}
mongo-1 | {"t":{"$date":"2025-06-26T08:29:04.646+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn8","msg":"Connection ended","attr":{"remote":"172.18.0.4:56978","uuid":"a61951c8-2adc-41c6-aa7e-fc4b925beb60","connectionId":8,"connectionCount":7}}
mongo-1 | {"t":{"$date":"2025-06-26T08:29:04.647+00:00"},"s":"I", "c":"-", "id":20883, "ctx":"conn7","msg":"Interrupted operation as its client disconnected","attr":{"opId":2425}}
mongo-1 | {"t":{"$date":"2025-06-26T08:29:04.647+00:00"},"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn7","msg":"Connection ended","attr":{"remote":"172.18.0.4:56970","uuid":"ba086e8c-b481-49c1-b3cf-9664ca2a4b1e","connectionId":7,"connectionCount":6}}
mongo-1 | {"t":{"$date":"2025-06-26T08:29:59.165+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926599,"ts_usec":165617,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 11, snapshot max: 11 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:30:59.178+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926659,"ts_usec":178911,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 17, snapshot max: 17 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:31:59.190+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926719,"ts_usec":190864,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 20, snapshot max: 20 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:32:59.199+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926779,"ts_usec":199862,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 22, snapshot max: 22 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:33:59.206+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926839,"ts_usec":206084,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 24, snapshot max: 24 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:34:59.215+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926899,"ts_usec":215613,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 26, snapshot max: 26 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:35:59.222+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750926959,"ts_usec":222869,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 28, snapshot max: 28 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:36:59.230+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750927019,"ts_usec":230033,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 30, snapshot max: 30 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:37:59.235+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750927079,"ts_usec":235802,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 32, snapshot max: 32 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:38:59.242+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750927139,"ts_usec":242397,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 34, snapshot max: 34 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:39:59.249+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750927199,"ts_usec":249534,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 36, snapshot max: 36 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:40:59.257+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750927259,"ts_usec":257337,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 38, snapshot max: 38 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}
mongo-1 | {"t":{"$date":"2025-06-26T08:41:59.263+00:00"},"s":"I", "c":"WTCHKPT", "id":22430, "ctx":"Checkpointer","msg":"WiredTiger message","attr":{"message":{"ts_sec":1750927319,"ts_usec":263268,"thread":"1:0x7f075c60f640","session_name":"WT_SESSION.checkpoint","category":"WT_VERB_CHECKPOINT_PROGRESS","category_id":6,"verbose_level":"DEBUG","verbose_level_id":1,"msg":"saving checkpoint snapshot min: 40, snapshot max: 40 snapshot count: 0, oldest timestamp: (0, 0) , meta checkpoint timestamp: (0, 0) base write gen: 3211"}}}👀 Have you spent some time to check if this bug has been raised before?
- I checked and didn't find similar issue
🔗 Are you willing to submit PR?
None
🧑⚖️ Code of Conduct
- I agree to follow this project's Code of Conduct