int4 weight float8 rowwise activation workflow doesn't work with HF accelerate

repro: https://github.com/pytorch/ao/pull/3473
stack trace: https://gist.github.com/vkuzo/6b128681b628744d445c553cdeac8a85