Redirect stdout/stderr automatically

[darukutsu]

Is there a way to redirect stdout/stderr of every command which are being executed for purpose of let's say highlighting. Something in style of preexec but don't execute main command :D. Some sort of HOOK maybe like ADDHISTORY one where you can have custom transformed command saved. I wasn't sure if something like this exists. Some sort of global pipe mechanism. Certainly wasn't able to figure it out myself.

preexec() {
  $@ | bat -p --paging=never
  return 1 # <-- don't execute $@
}

If you run this it would print output 2 times which is not desired. (I know this isn't purpose of preexec, just example)

I want better alternative to https://github.com/solworktech/zaje since this projects creates aliases, stderr isn't always handled and has other issues.

This could create option for more advanced highlighters which already exists and create own regex like rules for colorizing certain parts of commands with comfort of not registering alias for every command or manually piping it to bat. Probably would need sort of mechanism to turn off when not needed for grep etc, but this part isn't important to me for now.

[akinomyoga]

ble.sh manages stdout/stderr of the executed commands separately from stdout/stderr of the line editor itself. The file descriptors for stdout/stderr of the executed commands are stored in the shell variables _ble_util_fd_cmd_stdout and _ble_util_fd_cmd_stderr. So, technically, you can redirect these file descriptors in preexec.

For example, if you want to redirect stdout/stderr when the executed command is ls, you can naively write it in the following way.

preexec() {
  exec {_ble_util_fd_cmd_stdout}>&-
  exec {_ble_util_fd_cmd_stderr}>&-
  if [[ $1 == 'ls' ]]; then
    builtin eval -- "exec $_ble_util_fd_cmd_stdout> >(cat -n) $_ble_util_fd_cmd_stderr>&$_ble_util_fd_cmd_stdout"
  else
    builtin eval -- "exec $_ble_util_fd_cmd_stdout> /dev/tty $_ble_util_fd_cmd_stderr>&$_ble_util_fd_cmd_stdout"
  fi
}

I showed the above example for illustrative purposes, but a more robust version would be this:

preexec() {
  if [[ $1 == 'ls' ]]; then
    ble/fd#alloc _ble_util_fd_cmd_stdout '> >(cat -n)' overwrite
  else
    ble/fd#alloc _ble_util_fd_cmd_stdout '> /dev/tty' overwrite
  fi
  ble/fd#alloc _ble_util_fd_cmd_stderr ">&$_ble_util_fd_cmd_stdout" overwrite
  ble/fd#add-cloexec "$_ble_util_fd_cmd_stdout"
  ble/fd#add-cloexec "$_ble_util_fd_cmd_stderr"
}

edit: I modified the above examples to explicitly redirect stdout/stderr to /dev/tty when redirection is not wanted.

exec {_ble_util_fd_cmd_stdxxx}>&- in the first example are needed as a workaround for conflicts between Bash's internal file-descriptor manipulation and ble.sh's internal uses of file descriptors. Actually, I think this is a bug of Bash, but Chet doesn't seem to fix it soon. ble/fd#alloc in the second example takes care of it and other bash bugs and pitfalls. ble/fd#add-cloexec adds CLOEXEC to the backup file descriptors _ble_util_fd_cmd_stdout and _ble_util_fd_cmd_stderr, so that the child processes do not hold `ble.sh''s internal file descriptors.

Note

The preexec setup you showed is probably merely an example, but in case, it's not a good idea to redirect every command unconditionally. For example, TUI terminal applications like less, vim, and emacs -nw wouldn't properly function when its output is connected to a pipe: e.g., vim | bat -p, emacs -nw | bat -p, and tmux | bat -p wouldn't work. Also, { sleep 10 & } | bat -p will block until the background process terminates because the shell waits for the termination of bat, and bat waits for any possible output from sleep. You should properly design the condition that you redirects stdout/stderr.

Note

The process substitution >(cat -n) in the above example will rewrite $! before the command is executed. This means that the following doesn't work as usual with the above trick:

$ background-command &
$ saved_pid=$!    # <-- here, $! contains pid of `cat -n` instead of pid of `background-command`

But this always happens in general when a prompt framework or another bash configuration uses process substitutions, so it's probably not a big problem. If the user wants to save the pid, the user is recommended to do it in a single command line in general:

$ background-command & saved_pid=$!

Note

Another thing is that the output of the process substitution >(cat -n) isn't synchronized as done in a pipe. This may cause another issue of overwriting the next prompt, etc.

One might consider synchronizing it by saving the pid of the process substitution and wait for it in POSTEXEC, but I think that would cause another issue: If the executed command creates background processes, the wait will block until all the background processes terminates, so the shell hangs effectively.

[akinomyoga]

Thinking about it again, we need to revert the file descriptors to the original one in postexec, so you also need to hook into POSTEXEC.

postexec() {
  ble/fd#alloc _ble_util_fd_cmd_stdout '>/dev/tty' overwrite
  ble/fd#alloc _ble_util_fd_cmd_stderr ">&$_ble_util_fd_cmd_stdout" overwrite
  ble/fd#add-cloexec "$_ble_util_fd_cmd_stdout"
  ble/fd#add-cloexec "$_ble_util_fd_cmd_stderr"
}
bleopt POSTEXEC!=postexec

edit: I think it is the best to explicitly redirect to /dev/tty in preexec when stdout/stderr are not intended to be replaced. I updated the original comment.

[akinomyoga]

If you want to decorate the outputs of every command without caring about the difference between TUI programs and programs simply outputting text, the best solution is probably to support it at the terminal level.

TUI programs require PTY, so if we want to process any programs at the shell level, one needs to process the program outputs through PTY. However, this is exactly what terminals do. So, processing the PTY output by the shell seems like nesting two levels of PTY. I think it is rather straightforward to hook into the terminal itself instead of nesting PTY.

[darukutsu]

Thanks for quick reply and your guidance, sorry for late response. I tried some things myself. I discovered some more alternatives such as grc which mostly does things I want. Regarding determining if application is TUI or CLI based or subshell, I wasn't able to solve this I tried listing information through stty and /dev/{pts,ptmx} for loaded dependencies, but come to conclusion manually excluding programs like you suggested is probably best solution. Also redirecting to /dev/tty mangles output for some applications such as pacman.

This is what I have for now in .bashrc

## Terminal Coloring
GRC_ALIASES=true
[[ -s "/etc/profile.d/grc.sh" ]] && source /etc/profile.d/grc.sh

preexec() {
  if [[ "$*" =~ --help|help|-h ]] && ! [[ "$*" =~ --colou?rs? ]]; then
    ble/fd#alloc _ble_util_fd_cmd_stdout ">/tmp/blesh-stdout" overwrite
    ble/fd#alloc _ble_util_fd_cmd_stderr ">&$_ble_util_fd_cmd_stdout" overwrite
    ble/fd#add-cloexec "$_ble_util_fd_cmd_stdout"
    ble/fd#add-cloexec "$_ble_util_fd_cmd_stderr"
  fi
}

postexec() {
  if [[ "$*" =~ --help|help|-h ]] && ! [[ "$*" =~ --colou?rs? ]]; then
    bat -pplhelp /tmp/blesh-stdout
    ble/fd#alloc _ble_util_fd_cmd_stdout '>/dev/tty' overwrite
    ble/fd#alloc _ble_util_fd_cmd_stderr ">&$_ble_util_fd_cmd_stdout" overwrite
    ble/fd#add-cloexec "$_ble_util_fd_cmd_stdout"
    ble/fd#add-cloexec "$_ble_util_fd_cmd_stderr"
    rm -f /tmp/blesh-stdout
  fi
}

blehook POSTEXEC!=postexec

What else I tried:

preexec() {
  pts_before=$(ps -t "$(tty)" | wc -l)

  for cmd in $(ls /usr/share/grc | cut -f2 -d'.'); do
    # grc does it's thing
    if [[ $1 = "$cmd" ]]; then
      ble/fd#alloc _ble_util_fd_cmd_stdout ">/dev/tty" overwrite
    fi
  done

  if [[ -z $_ble_util_fd_cmd_stdout ]]; then
    ble/fd#alloc _ble_util_fd_cmd_stdout ">/tmp/blesh-stdout" overwrite
  fi
  tty=$(tty)
  ( (
    sleep 0.3                         # approx. time needed for pts allocation, delays TUI apps by this time
    pts_after=$(ps -t "$tty" | wc -l) # FIX: every command creates entry in pts
    if [[ $pts_after -gt $((pts_before + 1)) ]]; then
      # creates stdout for new application
      ble/fd#alloc _ble_util_fd_cmd_stdout ">/dev/tty" overwrite
      ble/fd#alloc _ble_util_fd_cmd_stderr ">&$_ble_util_fd_cmd_stdout" overwrite
      ble/fd#add-cloexec "$_ble_util_fd_cmd_stdout"
      ble/fd#add-cloexec "$_ble_util_fd_cmd_stderr"
    fi
  ) &)
  ble/fd#alloc _ble_util_fd_cmd_stderr ">&$_ble_util_fd_cmd_stdout" overwrite
  ble/fd#add-cloexec "$_ble_util_fd_cmd_stdout"
  ble/fd#add-cloexec "$_ble_util_fd_cmd_stderr"
}

postexec() {
  if [[ -e /tmp/blesh-stdout ]]; then
    content_type=$(file -b /tmp/blesh-stdout | cut -f1 -d' ')
    content_type=${content_type:-empty}
  fi

  if [[ "$*" =~ --help|help|-h ]]; then
    bat -pplhelp /tmp/blesh-stdout
  elif [[ "$content_type" != "empty" ]]; then
    language=$(BAT_PAGER='' bat --list-languages | grep -i -m1 "$content_type" | cut -f1 -d':')
    bat -ppl"${language:-Plain Text}" /tmp/blesh-stdout
  fi
  ble/fd#alloc _ble_util_fd_cmd_stdout '>/dev/tty' overwrite
  ble/fd#alloc _ble_util_fd_cmd_stderr ">&$_ble_util_fd_cmd_stdout" overwrite
  ble/fd#add-cloexec "$_ble_util_fd_cmd_stdout"
  ble/fd#add-cloexec "$_ble_util_fd_cmd_stderr"
  rm -f /tmp/blesh-stdout
}

If I ever find a better solution will post it.