Skip to content
This repository has been archived by the owner on Oct 31, 2022. It is now read-only.

fix for bigger models (774M) #30

Open
wants to merge 66 commits into
base: finetuning
Choose a base branch
from
Open

fix for bigger models (774M) #30

wants to merge 66 commits into from

Commits on Feb 18, 2019

  1. update README

    @WuTheFWasThat
    WuTheFWasThat authored Feb 18, 2019
    Configuration menu
    Copy the full SHA
    5b64684 View commit details
    Browse the repository at this point in the history

Commits on Feb 19, 2019

  1. reorganize and add temp 0.7

    @WuTheFWasThat
    WuTheFWasThat committed Feb 19, 2019
    Configuration menu
    Copy the full SHA
    6dab221 View commit details
    Browse the repository at this point in the history

Commits on Feb 20, 2019

  1. add license

    @WuTheFWasThat
    WuTheFWasThat committed Feb 20, 2019
    Configuration menu
    Copy the full SHA
    aae26ab View commit details
    Browse the repository at this point in the history

  2. add conditional samples

    @WuTheFWasThat
    WuTheFWasThat committed Feb 20, 2019
    Configuration menu
    Copy the full SHA
    fc0ee6d View commit details
    Browse the repository at this point in the history

  3. separate out tensorflow install

    @WuTheFWasThat
    WuTheFWasThat committed Feb 20, 2019
    Configuration menu
    Copy the full SHA
    825aa3d View commit details
    Browse the repository at this point in the history

  4. shuffle headings

    @WuTheFWasThat
    WuTheFWasThat committed Feb 20, 2019
    Configuration menu
    Copy the full SHA
    92ce9f2 View commit details
    Browse the repository at this point in the history

  5. more warning

    @WuTheFWasThat
    WuTheFWasThat committed Feb 20, 2019
    Configuration menu
    Copy the full SHA
    bf43e73 View commit details
    Browse the repository at this point in the history

  6. instructinos mention git clone

    @WuTheFWasThat
    WuTheFWasThat committed Feb 20, 2019
    Configuration menu
    Copy the full SHA
    23ed990 View commit details
    Browse the repository at this point in the history

  7. Add a Dockerfile and document usage in README

    @madisonmay @WuTheFWasThat
    madisonmay authored and WuTheFWasThat committed Feb 20, 2019
    Configuration menu
    Copy the full SHA
    99af6d7 View commit details
    Browse the repository at this point in the history

  8. fixed unconditional sampling reproducibility issue

    @WuTheFWasThat
    Ignacio Lopez-Francos authored and WuTheFWasThat committed Feb 20, 2019
    Configuration menu
    Copy the full SHA
    2cf46d9 View commit details
    Browse the repository at this point in the history

  9. fixed seed arg to ensure reproducibility in conditional-samples model

    @WuTheFWasThat
    Ignacio Lopez-Francos authored and WuTheFWasThat committed Feb 20, 2019
    Configuration menu
    Copy the full SHA
    946facf View commit details
    Browse the repository at this point in the history

  10. update readme

    @WuTheFWasThat
    WuTheFWasThat committed Feb 20, 2019
    Configuration menu
    Copy the full SHA
    b6f943d View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2019

  1. add conditional samples with default settings

    @WuTheFWasThat
    WuTheFWasThat committed Feb 21, 2019
    Configuration menu
    Copy the full SHA
    a3aa7de View commit details
    Browse the repository at this point in the history

  2. add .gitattributes file to ensure files copied to docker container ha… 

    …ve LF line endings and all files stay unix on commit
    @WuTheFWasThat
    stephan orlowsky authored and WuTheFWasThat committed Feb 21, 2019
    Configuration menu
    Copy the full SHA
    68bf7a0 View commit details
    Browse the repository at this point in the history

  3. Minor: update readme 

    Add note about setting PYTHONIOENCODING=UTF-8 env var for running
examples
    @natemurthy @WuTheFWasThat
    natemurthy authored and WuTheFWasThat committed Feb 21, 2019
    Configuration menu
    Copy the full SHA
    c5b9c89 View commit details
    Browse the repository at this point in the history

Commits on Feb 27, 2019

  1. Minor: update readme 

    Example will `tee` stdout to `/tmp/samples` from conditional and
unconditional generation scripts.
    @natemurthy @WuTheFWasThat
    natemurthy authored and WuTheFWasThat committed Feb 27, 2019
    Configuration menu
    Copy the full SHA
    c314dda View commit details
    Browse the repository at this point in the history

  2. Add documentation for help flags (nshepperd#81) 

    add description for flags
    @ArmaanBhullar @WuTheFWasThat
    ArmaanBhullar authored and WuTheFWasThat committed Feb 27, 2019
    Configuration menu
    Copy the full SHA
    ed49f03 View commit details
    Browse the repository at this point in the history

  3. slight fix to batch size description

    @WuTheFWasThat
    WuTheFWasThat committed Feb 27, 2019
    Configuration menu
    Copy the full SHA
    9d1e704 View commit details
    Browse the repository at this point in the history

Commits on Feb 28, 2019

  1. updates

    @WuTheFWasThat
    WuTheFWasThat committed Feb 28, 2019
    Configuration menu
    Copy the full SHA
    0465394 View commit details
    Browse the repository at this point in the history

Commits on Mar 3, 2019

  1. Add finetuning code.

    Neil Shepperd committed Mar 3, 2019
    Configuration menu
    Copy the full SHA
    d1fc873 View commit details
    Browse the repository at this point in the history

  2. chmod +x

    Neil Shepperd committed Mar 3, 2019
    Configuration menu
    Copy the full SHA
    1fba31f View commit details
    Browse the repository at this point in the history

  3. Add finetuning instructions

    Neil Shepperd committed Mar 3, 2019
    Configuration menu
    Copy the full SHA
    dfca3cf View commit details
    Browse the repository at this point in the history

  4. Fix sample generation with batch_size greater than 1.

    Neil Shepperd committed Mar 3, 2019
    Configuration menu
    Copy the full SHA
    9423776 View commit details
    Browse the repository at this point in the history

Commits on Mar 4, 2019

  1. Python download script (nshepperd#89) 

    added python download script and modified requirements to add the modules needed. Tested in Windows Version 10.0.17134 Build 17134  and Ubuntu 18.04.1 LTS
    @webproduktion01 @WuTheFWasThat
    webproduktion01 authored and WuTheFWasThat committed Mar 4, 2019
    Configuration menu
    Copy the full SHA
    8eb6793 View commit details
    Browse the repository at this point in the history

  2. update download stuff

    @WuTheFWasThat
    WuTheFWasThat committed Mar 4, 2019
    Configuration menu
    Copy the full SHA
    ed0dedc View commit details
    Browse the repository at this point in the history

Commits on Mar 6, 2019

  1. update readme with usage caveats and calls for research 

    This write-up was loosely inspired in part by Mitchell et al.’s work on
[Model Cards for Model Reporting](https://arxiv.org/abs/1810.03993).
Adding such model usage sections could be good practice in general for
open source research projects with potentially broad applications.
    @WuTheFWasThat
    WuTheFWasThat committed Mar 6, 2019
    Configuration menu
    Copy the full SHA
    953530f View commit details
    Browse the repository at this point in the history

  2. add contributors md and move dev docs out

    @WuTheFWasThat
    WuTheFWasThat committed Mar 6, 2019
    Configuration menu
    Copy the full SHA
    79a246a View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2019

  1. fix for windows (thanks to chrothenbach)

    @WuTheFWasThat
    WuTheFWasThat committed Mar 7, 2019
    Configuration menu
    Copy the full SHA
    8637828 View commit details
    Browse the repository at this point in the history

Commits on Mar 18, 2019

  1. Add training script with Horovod support 

    This enables multi-GPU or distributed training using Horovod
    @tlkh
    tlkh authored Mar 18, 2019
    Configuration menu
    Copy the full SHA
    3e18729 View commit details
    Browse the repository at this point in the history

  2. Fix typo in train command in README

    @tlkh
    tlkh authored Mar 18, 2019
    Configuration menu
    Copy the full SHA
    ec16bad View commit details
    Browse the repository at this point in the history

  3. Added instructions for training using Horovod

    @tlkh
    tlkh authored Mar 18, 2019
    Configuration menu
    Copy the full SHA
    0bad9e4 View commit details
    Browse the repository at this point in the history

  4. Update CONTRIBUTORS.md

    @WuTheFWasThat
    WuTheFWasThat authored Mar 18, 2019
    Configuration menu
    Copy the full SHA
    d14501a View commit details
    Browse the repository at this point in the history

Commits on Mar 19, 2019

  1. Merge pull request nshepperd#2 from tlkh/finetuning 

    Added instructions and script for distributed training with Horovod
    @nshepperd
    nshepperd authored Mar 19, 2019
    Configuration menu
    Copy the full SHA
    ef62678 View commit details
    Browse the repository at this point in the history

  2. autoformat

    Neil Shepperd committed Mar 19, 2019
    Configuration menu
    Copy the full SHA
    c465071 View commit details
    Browse the repository at this point in the history

  3. Combine input text files with <|endoftext|> delimiter to ensure there… 

    … are enough contiguous tokens to sample from.

This can be disabled with --combine 0.
    Neil Shepperd committed Mar 19, 2019
    Configuration menu
    Copy the full SHA
    1e32b10 View commit details
    Browse the repository at this point in the history

Commits on Mar 20, 2019

  1. Write losses to summary file for tensorboard.

    Neil Shepperd committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    3a3ce65 View commit details
    Browse the repository at this point in the history

  2. Add learning rate as command line flag.

    Neil Shepperd committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    d5b387b View commit details
    Browse the repository at this point in the history

  3. Use argparse instead of fire in train.py.

    Neil Shepperd committed Mar 20, 2019
    Configuration menu
    Copy the full SHA
    b106d0a View commit details
    Browse the repository at this point in the history

Commits on Mar 21, 2019

  1. Fix encode.py

    Neil Shepperd committed Mar 21, 2019
    Configuration menu
    Copy the full SHA
    2044d13 View commit details
    Browse the repository at this point in the history

  2. Add gradient accumulation with default of 5 minibatches

    Neil Shepperd committed Mar 21, 2019
    Configuration menu
    Copy the full SHA
    a359a34 View commit details
    Browse the repository at this point in the history

Commits on Mar 25, 2019

  1. Merge remote-tracking branch 'origin/master' into finetuning

    Neil Shepperd committed Mar 25, 2019
    Configuration menu
    Copy the full SHA
    8738950 View commit details
    Browse the repository at this point in the history

Commits on May 2, 2019

  1. Turn off gradient accumulation by default, it shouldn't be needed.

    Neil Shepperd committed May 2, 2019
    Configuration menu
    Copy the full SHA
    eda8777 View commit details
    Browse the repository at this point in the history

Commits on May 3, 2019

  1. updates for 345M model

    @WuTheFWasThat
    WuTheFWasThat committed May 3, 2019
    Configuration menu
    Copy the full SHA
    0503b1b View commit details
    Browse the repository at this point in the history

  2. reference dataset

    @WuTheFWasThat
    WuTheFWasThat committed May 3, 2019
    Configuration menu
    Copy the full SHA
    b5ef71a View commit details
    Browse the repository at this point in the history

  3. remove samples

    @WuTheFWasThat
    WuTheFWasThat committed May 3, 2019
    Configuration menu
    Copy the full SHA
    dd75299 View commit details
    Browse the repository at this point in the history

Commits on May 4, 2019

  1. Add gradient checkpointing and another optimization necessary to allo… 

    …w training the 345M model.
    Neil Shepperd committed May 4, 2019
    Configuration menu
    Copy the full SHA
    47df6da View commit details
    Browse the repository at this point in the history

  2. Add "validation" loss calculation.

    Neil Shepperd committed May 4, 2019
    Configuration menu
    Copy the full SHA
    c46ed99 View commit details
    Browse the repository at this point in the history

Commits on May 5, 2019

  1. Add toposort to requirements

    @Tenoke
    Tenoke committed May 5, 2019
    Configuration menu
    Copy the full SHA
    941a762 View commit details
    Browse the repository at this point in the history

Commits on May 6, 2019

  1. Merge pull request nshepperd#3 from Tenoke/finetuning 

    Add toposort to requirements
    Neil Shepperd authored May 6, 2019
    Configuration menu
    Copy the full SHA
    13c5412 View commit details
    Browse the repository at this point in the history

Commits on May 14, 2019

  1. Add option to use SGD for optimizer

    Neil Shepperd committed May 14, 2019
    Configuration menu
    Copy the full SHA
    3985cc7 View commit details
    Browse the repository at this point in the history

  2. Record learning rate in tensorboard logs

    Neil Shepperd committed May 14, 2019
    Configuration menu
    Copy the full SHA
    7fc2a44 View commit details
    Browse the repository at this point in the history

  3. Add text in README for --optimizer flag

    Neil Shepperd committed May 14, 2019
    Configuration menu
    Copy the full SHA
    a464925 View commit details
    Browse the repository at this point in the history

  4. Reduce default learning rate of train.py.

    Neil Shepperd committed May 14, 2019
    Configuration menu
    Copy the full SHA
    ae535b6 View commit details
    Browse the repository at this point in the history

  5. Merge remote-tracking branch 'origin/master' into finetuning

    Neil Shepperd committed May 14, 2019
    Configuration menu
    Copy the full SHA
    2d4fd0c View commit details
    Browse the repository at this point in the history

Commits on May 15, 2019

  1. New feature: add noise to network inputs to regularize against overre… 

    …acting to typos.
    Neil Shepperd committed May 15, 2019
    Configuration menu
    Copy the full SHA
    6a77a7b View commit details
    Browse the repository at this point in the history

  2. Add top-p sampling

    Neil Shepperd committed May 15, 2019
    Configuration menu
    Copy the full SHA
    87fe3d7 View commit details
    Browse the repository at this point in the history

  3. Add top_p to interactive_conditional_samples.py and generate_uncondit… 

    …ional_samples.py.
    Neil Shepperd committed May 15, 2019
    Configuration menu
    Copy the full SHA
    e99ee37 View commit details
    Browse the repository at this point in the history

  4. fix typo in top_p

    Neil Shepperd committed May 15, 2019
    Configuration menu
    Copy the full SHA
    2b24145 View commit details
    Browse the repository at this point in the history

  5. Fix top_p sampling for batch_size>1

    Neil Shepperd committed May 15, 2019
    Configuration menu
    Copy the full SHA
    6c1f21d View commit details
    Browse the repository at this point in the history

Commits on Aug 15, 2019

  1. Updated README.md 

    Added the medium blog link "Beginner’s Guide to Retrain GPT-2 (117M) to Generate Custom Text Content"
    @biranchi2018
    biranchi2018 authored Aug 15, 2019
    Configuration menu
    Copy the full SHA
    cca7144 View commit details
    Browse the repository at this point in the history

Commits on Aug 27, 2019

  1. Merge pull request nshepperd#22 from biranchi2018/biranchi2018-patch-1 

    Updated README.md
    N Shepperd authored Aug 27, 2019
    Configuration menu
    Copy the full SHA
    a070f38 View commit details
    Browse the repository at this point in the history

  2. Add note to install cudnn, re nshepperd#8

    Neil Shepperd committed Aug 27, 2019
    Configuration menu
    Copy the full SHA
    50fa3b6 View commit details
    Browse the repository at this point in the history

  3. Add flag to set encoding for text reading and writing, defaulting to … 

    …utf-8.
    Neil Shepperd committed Aug 27, 2019
    Configuration menu
    Copy the full SHA
    b7cda3f View commit details
    Browse the repository at this point in the history

Commits on Sep 13, 2019

  1. fix for bigger models (774M)

    @mgrankin
    mgrankin committed Sep 13, 2019
    Configuration menu
    Copy the full SHA
    2ebc511 View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2019

  1. it's better to remove this lines and use params to train 774M and 345M

    @mgrankin
    mgrankin committed Sep 15, 2019
    Configuration menu
    Copy the full SHA
    ca7602a View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2019

  1. smooth avg_loss on 10k batches

    @mgrankin
    mgrankin committed Sep 20, 2019
    Configuration menu
    Copy the full SHA
    9b2cab5 View commit details
    Browse the repository at this point in the history