This repository was archived by the owner on Jul 24, 2024. It is now read-only.
This repository was archived by the owner on Jul 24, 2024. It is now read-only.
Why use bn in projection head? #205
Open
Description
Line 178 in 3ad6700
In this code, bn is used in every layer of projection head, which is not mentioned in the papers.
Moreover, there is no bias in middle layer, which is conflict with the comments.
Metadata
Metadata
Assignees
Labels
No labels