Skip to content

Conversation

@ghuls
Copy link

@ghuls ghuls commented Dec 23, 2025

Add pipe function to read and write to pipes with the help of popen. This allows the following:

$ cat pipe_command_tar.codon
print("Read tar file list")
with pipe("tar tzvf codon-linux-x86_64.tar.gz", "r") as fh:
    i = 0
    for line in fh:
        if i == 15:
            break
        print(line, end="")
        i += 1

print("Gunzip tar file and compress with zstd")
with pipe("gzip -cd codon-linux-x86_64.tar.gz", "r") as in_fh:
    with pipe("zstd -c -9 - > codon-linux-x86_64.tar.zst", "w") as out_fh:
        chunk = in_fh.read(4096*4)
        while chunk:
            out_fh.write(chunk)
            chunk = in_fh.read(4096*4)

$ codon run pipe_command_tar.codon
Read tar file list
drwxr-xr-x root/root         0 2025-11-28 01:00 codon-deploy-linux-x86_64/
drwxr-xr-x root/root         0 2025-11-28 01:49 codon-deploy-linux-x86_64/python/
-rw-r--r-- root/root      2692 2025-11-28 00:32 codon-deploy-linux-x86_64/python/setup.py
-rw-r--r-- root/root        32 2025-11-28 00:32 codon-deploy-linux-x86_64/python/.gitignore
-rw-r--r-- root/root       184 2025-11-28 00:32 codon-deploy-linux-x86_64/python/README.md
-rw-r--r-- root/root        69 2025-11-28 00:32 codon-deploy-linux-x86_64/python/pyproject.toml
drwxr-xr-x root/root         0 2025-11-28 01:00 codon-deploy-linux-x86_64/python/codon/
-rw-r--r-- root/root       134 2025-11-28 00:37 codon-deploy-linux-x86_64/python/codon/version.py
-rw-r--r-- root/root     10317 2025-11-28 00:32 codon-deploy-linux-x86_64/python/codon/decorator.py
-rw-r--r-- root/root      2541 2025-11-28 00:32 codon-deploy-linux-x86_64/python/codon/jit.pyx
-rw-r--r-- root/root       636 2025-11-28 00:32 codon-deploy-linux-x86_64/python/codon/jit.pxd
-rw-r--r-- root/root       279 2025-11-28 00:32 codon-deploy-linux-x86_64/python/codon/__init__.py
-rw-r--r-- root/root        20 2025-11-28 00:32 codon-deploy-linux-x86_64/python/MANIFEST.in
drwxr-xr-x root/root         0 2025-11-28 01:00 codon-deploy-linux-x86_64/python/codon_jit.egg-info/
-rw-r--r-- root/root         6 2025-11-28 01:00 codon-deploy-linux-x86_64/python/codon_jit.egg-info/top_level.txt
Gunzip tar file and compress with zstd

There are a few downsides to using popen:

  • popen spawns a shell to execute the "command" and failures of finding the actual command are not 100% reliable to detect. The shell return exit code 127 if if can't find the command and exit code 126 if it is not an executable, but if for some reason the actual command would use the same exit codes, there is no way to communicate that.
  • popen only captures stdout and not stderr

So in the long run it is probably better to have a proper implementation that can also capture stderr (and avoids deadlocks when reading both streams) and also captures the real exit code instead of always spawning a shell.

Add `pipe` function to read and write to pipes with the help of `popen`.
This allows the following:

    $ cat pipe_command_tar.codon
    print("Read tar file list")
    with pipe("tar tzvf codon-linux-x86_64.tar.gz", "r") as fh:
        i = 0
        for line in fh:
            if i == 15:
                break
            print(line, end="")
            i += 1

    print("Gunzip tar file and compress with zstd")
    with pipe("gzip -cd codon-linux-x86_64.tar.gz", "r") as in_fh:
        with pipe("zstd -c -9 - > codon-linux-x86_64.tar.zst", "w") as out_fh:
            chunk = in_fh.read(4096*4)
            while chunk:
                out_fh.write(chunk)
                chunk = in_fh.read(4096*4)

    $ codon run pipe_command_tar.codon
    Read tar file list
    drwxr-xr-x root/root         0 2025-11-28 01:00 codon-deploy-linux-x86_64/
    drwxr-xr-x root/root         0 2025-11-28 01:49 codon-deploy-linux-x86_64/python/
    -rw-r--r-- root/root      2692 2025-11-28 00:32 codon-deploy-linux-x86_64/python/setup.py
    -rw-r--r-- root/root        32 2025-11-28 00:32 codon-deploy-linux-x86_64/python/.gitignore
    -rw-r--r-- root/root       184 2025-11-28 00:32 codon-deploy-linux-x86_64/python/README.md
    -rw-r--r-- root/root        69 2025-11-28 00:32 codon-deploy-linux-x86_64/python/pyproject.toml
    drwxr-xr-x root/root         0 2025-11-28 01:00 codon-deploy-linux-x86_64/python/codon/
    -rw-r--r-- root/root       134 2025-11-28 00:37 codon-deploy-linux-x86_64/python/codon/version.py
    -rw-r--r-- root/root     10317 2025-11-28 00:32 codon-deploy-linux-x86_64/python/codon/decorator.py
    -rw-r--r-- root/root      2541 2025-11-28 00:32 codon-deploy-linux-x86_64/python/codon/jit.pyx
    -rw-r--r-- root/root       636 2025-11-28 00:32 codon-deploy-linux-x86_64/python/codon/jit.pxd
    -rw-r--r-- root/root       279 2025-11-28 00:32 codon-deploy-linux-x86_64/python/codon/__init__.py
    -rw-r--r-- root/root        20 2025-11-28 00:32 codon-deploy-linux-x86_64/python/MANIFEST.in
    drwxr-xr-x root/root         0 2025-11-28 01:00 codon-deploy-linux-x86_64/python/codon_jit.egg-info/
    -rw-r--r-- root/root         6 2025-11-28 01:00 codon-deploy-linux-x86_64/python/codon_jit.egg-info/top_level.txt
    Gunzip tar file and compress with zstd

There are a few downsides to using popen:
  - popen spawns a shell to execute the "command" and failures of finding the actual command
    are not 100% reliable to detect. The shell return exit code 127 if if can't find the
    command and exit code 126 if it is not an executable, but if for some reason the actual
    command would use the same exit codes, there is no way to communicate that.
  - popen only captures stdout and not stderr

So in the long run it is probably better to have a proper implementation that can also capture
stderr (and avoids deadlocks when reading both streams) and also captures the real exit code
instead of always spawning a shell.
@cla-bot cla-bot bot added the cla-signed label Dec 23, 2025
@ghuls ghuls mentioned this pull request Dec 23, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant