Skip to content

Commit

Permalink
Stream files from store instead of buffering them
Browse files Browse the repository at this point in the history
When an artifact is requested from hydra the output is first copied
from the nix store into memory and then sent as a response, delaying
the download and taking up significant amounts of memory.

As reported in #1357

Instead of calling a command and blocking while reading in the entire
output, this adds read_into_socket(). the function takes a
command, starting a subprocess with that command, returning a file
descriptor attached to stdout.
This file descriptor is then by responsebuilder of Catalyst to steam
the output directly

(cherry picked from commit 459aa0a)
  • Loading branch information
71rd authored and mweinelt committed Jan 12, 2025
1 parent 4f09fb5 commit 0ce7197
Show file tree
Hide file tree
Showing 2 changed files with 13 additions and 1 deletion.
2 changes: 1 addition & 1 deletion src/lib/Hydra/Controller/Build.pm
Original file line number Diff line number Diff line change
Expand Up @@ -238,7 +238,7 @@ sub serveFile {
# XSS hole.
$c->response->header('Content-Security-Policy' => 'sandbox allow-scripts');

$c->stash->{'plain'} = { data => grab(cmd => ["nix", "--experimental-features", "nix-command",
$c->stash->{'plain'} = { data => readIntoSocket(cmd => ["nix", "--experimental-features", "nix-command",
"store", "cat", "--store", getStoreUri(), "$path"]) };

# Detect MIME type.
Expand Down
12 changes: 12 additions & 0 deletions src/lib/Hydra/Helper/Nix.pm
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ our @EXPORT = qw(
jobsetOverview
jobsetOverview_
pathIsInsidePrefix
readIntoSocket
readNixFile
registerRoot
restartBuilds
Expand Down Expand Up @@ -416,6 +417,17 @@ sub pathIsInsidePrefix {
return $cur;
}

sub readIntoSocket{
my (%args) = @_;
my $sock;

eval {
my $x= join(" ", @{$args{cmd}});
open($sock, "-|", $x) or die q(failed to open socket from command:\n $x);
};

return $sock;
}



Expand Down

0 comments on commit 0ce7197

Please sign in to comment.