Autocast support for half precision

[majing921201]

hi, I didn't found autocast support in the train step. Where we just convert model to bfloat16.

generative-recommenders/generative_recommenders/trainer/train.py

Line 263 in bb389f9

model = model.to(torch.bfloat16)

But I found some autocast related code in hstu main body, and autocast option is hard code to "None".

generative-recommenders/generative_recommenders/modeling/sequential/hstu.py

Line 621 in bb389f9

autocast_dtype=None,

User cannot pass autocast argument to model.
So for my understanding autocast is not supported now, may I know do you have a plan for autocast ? Thank you

[jiaqizhai]

Hi, the public experiments were run with autocast disabled given the datasets were small. In large-scale settings, enabling autocast would definitely be beneficial. It should be easy to flip that flag back on, but you probably want to integrate the custom triton kernels as well if efficiency is a main focus.

[majing921201]

Thank you for your quick response.
So for the configs provided in current repo: https://github.com/facebookresearch/generative-recommenders/tree/main/configs. They are all not large-scale scope, right ? Is there any specific definition for "large-scale" per hstu model ?

[jiaqizhai]

Hi - we discussed some possible industrial-scale configurations in Section 4.1.2 and Section 4.3.1 of the paper. For comparison, the Amazon Books dataset (one of the largest public datasets commonly used in prior work) consists of 694,897 users, 674,044 items, which is a few orders of magnitude smaller than the industrial-scale configurations we discussed/tested.