I was really confused how this install worked as the readme didn't really go into details.
It turns out that brew install chruby-fish also installs fish via brew and then unless you are using that version of fish chruby won't work.
I'm adding this for others who end up here in the future with the same issue as the issues was the first place I went looking for a solution.