New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

请问训练albert_xxlarge时超参如何设置 #3

Open

bestbzw opened this issue Dec 1, 2020 · 1 comment

bestbzw commented Dec 1, 2020

您好，我在ReCo上用xx_large训练的时候发现模型的loss一直不下降，请问您的超参数是如何设置的？是否加了warmup，dropout等策略？

Owner

benywon commented Dec 13, 2020

使用LAMB优化器，lr设置小一点

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment