Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Artifact downloading from Hydra buffers the whole artifact into RAM before sending it #1357

Open
delroth opened this issue Feb 5, 2024 · 4 comments
Open

Artifact downloading from Hydra buffers the whole artifact into RAM before sending it #1357

delroth opened this issue Feb 5, 2024 · 4 comments
Labels
bug

Comments

@delroth
Copy link
Contributor

delroth commented Feb 5, 2024

hydra/src/lib/Hydra/Controller/Build.pm

Line 239 in c1bd50a

$c->stash->{'plain'} = { data => grab(cmd => ["nix", "--experimental-features", "nix-command",

The way Hydra artifacts downloads works is that it runs "nix store cat", reads the whole contents into RAM, and then sends it as HTTP response. This has multiple issues for larger files (e.g. disk images):

  • Very long time to first byte, which could cause some clients to time out.
  • Very high RAM usage by the Hydra web workers.
  • Significant decrease in throughput, since the transfer time happens ~ 2x.

We noticed it a few weeks ago on #infra:nixos.org but then this got rediscovered again today on #hydra:nixos.org so I thought it might be worth filing a bug to actually track this.
@delroth delroth added the bug label Feb 5, 2024
@delroth
Copy link
Contributor Author

delroth commented Apr 12, 2024

This sometimes ends up hogging all workers on h.n.o, causing the website to become unavailable and channel scripts to fail.

Ma27 pushed a commit to Ma27/hydra that referenced this issue Aug 17, 2024
@71rd @yuyuyureka
Stream files from store instead of buffering them
459aa0a 
When an artifact is requested from hydra the output is first copied
from the nix store into memory and then sent as a response, delaying
the download and taking up significant amounts of memory.

As reported in NixOS#1357

Instead of calling a command and blocking while reading in the entire
output, this adds read_into_socket(). the function takes a
command, starting a subprocess with that command, returning a file
descriptor attached to stdout.
This file descriptor is then by responsebuilder of Catalyst to steam
the output directly
@mweinelt
Copy link
Member

mweinelt commented Jan 9, 2025

@Ma27 is the referenced commit working and worth testing on h.n.o?

@Ma27
Copy link
Member

Ma27 commented Jan 12, 2025

@mweinelt you'll also want https://git.lix.systems/lix-project/hydra/commit/3ee51dbe589458cc54ff753317bbc6db530bddc0.

mweinelt pushed a commit that referenced this issue Jan 13, 2025
@71rd @mweinelt
Stream files from store instead of buffering them
0ce7197 
When an artifact is requested from hydra the output is first copied
from the nix store into memory and then sent as a response, delaying
the download and taking up significant amounts of memory.

As reported in #1357

Instead of calling a command and blocking while reading in the entire
output, this adds read_into_socket(). the function takes a
command, starting a subprocess with that command, returning a file
descriptor attached to stdout.
This file descriptor is then by responsebuilder of Catalyst to steam
the output directly

(cherry picked from commit 459aa0a)
@mweinelt
Copy link
Member

Thanks, I picked both patches and deployed them.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug
Projects
Infra operational issues
Status: Todo
Milestone
No milestone
Development

No branches or pull requests
3 participants
@mweinelt @delroth @Ma27