Rational Behind Month Encoding

[leanderloew]

Hey, I'm just curious to better understand the rational for why we use a month encoding (instead of say a day of the year encoding), and why we apply this encoding through gates in the network, instead of adding of concatenating the information? Thanks!

[robert-DL]

Hi Leander,

conditioning via AdaLN is pretty standard in many deep learning frameworks. I believe other methods are an option too. However, adding extra tokens for example throws up the question how to handle conditioning within the context of the SWIN Transformer where windowed attention is utilized (you want to give the information to all windows, so would you use an extra attention layer? This causes a massive overhead).

The month encoding is a more condensed way of providing "climatological" information I believe, compared to 365 different encodings, as the per day variability can be relatively high and thus day_of_the_year does not add more information.

[leanderloew]

Hey Robert,
thanks for getting back to me! I was not indeed aware that adaptive layer norm is the "standard" way to condition vision transformers, thanks for pointing that out to me! I was just adding the conditioning information to the embedding (no extra layer or parameters needed).
Regarding the month encoding, I still feel a bit worried that there will be a distribution shift happening from say the 30th of November to the first of december.