Incosistent temporary locations used in store, log message give not much indication #3671
Description
Describe the bug
The temporary directories involved in the store operation is inconsistent. In some cases, the /tmp is used, which in case of size-limited tmpfs, can exhaust available space. In some other cases, the temporary directory is created in the server's config-directory, right now (during a store being processed serverside) I am observing a ~/.codechecker/tmpXXXXXX directory containing blame/, content_hashes.json, reports/ and root/. But other contexts, such as the client-side operation before store itself, and the serverside unpacking of the result, uses /tmp.
I hit a case where using a 4 GiB limited tmpfs /tmp I ended up receiving a ~9 GiB ZIP, which could not be unpacked into /tmp and the following exception was thrown. I did not hit the issue clientside because my client environment was a Docker container, where the inside /tmp was on persistent (and thus "infinitely big") storage. Were it also in RAM, I would have expected a clientside failure to happen too.
[INFO 2022-05-20 14:42] - [openssl_openssl-3.0.0-alpha7_Tidy-Master-BugproneUnusedReturnValue-OFF] Unzip storage file done... (duration: 10.04 sec)
[ERROR 2022-05-20 14:42] - Failed to store results: [Errno 28] No space left on device
Traceback (most recent call last):
File "/home/w/opt/python3-env/lib/python3.6/site-packages/codechecker_server/api/mass_store_run.py", line 1054, in store
zip_size = unzip(self.__b64zip, zip_dir)
File "/home/w/opt/python3-env/lib/python3.6/site-packages/codechecker_server/api/mass_store_run.py", line 82, in unzip
zip_file.write(zlib.decompress(base64.b64decode(b64zip)))
File "/home/w/opt/python3-env/lib/python3.6/tempfile.py", line 624, in func_wrapper
return func(*args, **kwargs)
OSError: [Errno 28] No space left on device
Traceback (most recent call last):
File "/home/w/opt/python3-env/lib/python3.6/site-packages/codechecker_server/api/report_server.py", line 132, in wrapper
res = func(*args, **kwargs)
File "/home/w/opt/python3-env/lib/python3.6/site-packages/codechecker_server/profiler.py", line 74, in wrapper
return release_wrapper(*args, **kwargs)
File "/home/w/opt/python3-env/lib/python3.6/site-packages/codechecker_server/profiler.py", line 66, in release_wrapper
res = function(*args, **kwargs)
File "/home/w/opt/python3-env/lib/python3.6/site-packages/codechecker_server/api/report_server.py", line 2888, in massStoreRun
return m.store()
File "/home/w/opt/python3-env/lib/python3.6/site-packages/codechecker_server/api/mass_store_run.py", line 1054, in store
zip_size = unzip(self.__b64zip, zip_dir)
File "/home/w/opt/python3-env/lib/python3.6/site-packages/codechecker_server/api/mass_store_run.py", line 82, in unzip
zip_file.write(zlib.decompress(base64.b64decode(b64zip)))
File "/home/w/opt/python3-env/lib/python3.6/tempfile.py", line 624, in func_wrapper
return func(*args, **kwargs)
OSError: [Errno 28] No space left on device
[WARNING 2022-05-20 14:42] - massStoreRun:
[Errno 28] No space left on device
First, it should be noted that the size log entry is only printed at a successful store. So the No space left on device is a cryptic warning, because it does not explain where things were being exported and how much space was exhausted. While the Info loglevel prints a lot of verbose details about the store processing, the @massStoreRun access log entry is not annotated with the size of the request.
(And it says done and a short time duration before showing the error!)
Furthermore, I think this should be made at least consistent. According to the FHS, the /tmp MIGHT be cleared between executions, and thus mounted into RAM (and thus limited in size compared to a non-volatile storage medium). (Notably, /var/tmp MUST be persistent, and /run MUST be volatile.) I would say that a custom temporary directory within the server's own data directory is also persistent storage.
In general, /tmp things should be small or short-lived. And the server is perfectly capable of processing 9 GiB of data, so this is not a configuration issue... after I remounted with a larger size limit, the store finished in ~12 minutes. With RAM size being hundreds of gigabytes nowadays (for proper developer machines, anyway), it is common practice to have most of the temporary things mounted into RAM. (While not the default on Debian, I believe it is the default suggested by Arch.)
All in all, I believe the default location for these temporary unzips should be in /var/tmp instead of /tmp, and this should be consistent for all such operations. (Or in a temporary location together with the server's data.)
And while the storage itself is persistent, CodeChecker should make sure that temporaries are deleted when needed, e.g. when the server crashes/starts, etc.
CodeChecker version
v6.19.1, installed from PyPI
Desktop (please complete the following information)
- OS: Linux