Skip to content

Commit ef6f4cd

Browse files
kmeng01kmeng01
committed
init
0 parents  commit ef6f4cd

File tree

138 files changed

+13723
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

138 files changed

+13723
-0
lines changed

‎.gitattributes

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
*.ipynb filter=clean_ipynb

‎.gitignore

+165
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,165 @@
1+
# Pipeline dumps, data directory
2+
results
3+
data
4+
!notebooks/data
5+
*_tmp_*_.json
6+
*_kmeng01g1gn*
7+
8+
# Pre-trained hypernetworks
9+
baselines/*/weights
10+
11+
# Mac specific
12+
.idea
13+
.vscode
14+
.DS_Store
15+
16+
# Latex
17+
*.aux
18+
*.dvi
19+
*.fdb_latexmk
20+
*.fls
21+
*.log
22+
*.pdf
23+
*.synctex.gz
24+
*.out
25+
*.toc
26+
*.nps
27+
28+
# Byte-compiled / optimized / DLL files
29+
__pycache__/
30+
*.py[cod]
31+
*$py.class
32+
*.py.swp
33+
34+
# C extensions
35+
*.so
36+
37+
# Distribution / packaging
38+
.Python
39+
build/
40+
develop-eggs/
41+
dist/
42+
downloads/
43+
eggs/
44+
.eggs/
45+
lib/
46+
lib64/
47+
parts/
48+
sdist/
49+
var/
50+
wheels/
51+
share/python-wheels/
52+
*.egg-info/
53+
.installed.cfg
54+
*.egg
55+
MANIFEST
56+
57+
# PyInstaller
58+
# Usually these files are written by a python script from a template
59+
# before PyInstaller builds the exe, so as to inject date/other infos into it.
60+
*.manifest
61+
*.spec
62+
63+
# Installer logs
64+
pip-log.txt
65+
pip-delete-this-directory.txt
66+
67+
# Unit test / coverage reports
68+
htmlcov/
69+
.tox/
70+
.nox/
71+
.coverage
72+
.coverage.*
73+
.cache
74+
nosetests.xml
75+
coverage.xml
76+
*.cover
77+
*.py,cover
78+
.hypothesis/
79+
.pytest_cache/
80+
cover/
81+
82+
# Translations
83+
*.mo
84+
*.pot
85+
86+
# Django stuff:
87+
*.log
88+
local_settings.py
89+
db.sqlite3
90+
db.sqlite3-journal
91+
92+
# Flask stuff:
93+
instance/
94+
.webassets-cache
95+
96+
# Scrapy stuff:
97+
.scrapy
98+
99+
# Sphinx documentation
100+
docs/_build/
101+
102+
# PyBuilder
103+
.pybuilder/
104+
target/
105+
106+
# Jupyter Notebook
107+
.ipynb_checkpoints
108+
109+
# IPython
110+
profile_default/
111+
ipython_config.py
112+
113+
# pyenv
114+
# For a library or package, you might want to ignore these files since the code is
115+
# intended to run in multiple environments; otherwise, check them in:
116+
# .python-version
117+
118+
# pipenv
119+
# According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.
120+
# However, in case of collaboration, if having platform-specific dependencies or dependencies
121+
# having no cross-platform support, pipenv may install dependencies that don't work, or not
122+
# install all needed dependencies.
123+
#Pipfile.lock
124+
125+
# PEP 582; used by e.g. github.com/David-OConnor/pyflow
126+
__pypackages__/
127+
128+
# Celery stuff
129+
celerybeat-schedule
130+
celerybeat.pid
131+
132+
# SageMath parsed files
133+
*.sage.py
134+
135+
# Environments
136+
.venv
137+
env/
138+
venv/
139+
ENV/
140+
env.bak/
141+
venv.bak/
142+
143+
# Spyder project settings
144+
.spyderproject
145+
.spyproject
146+
147+
# Rope project settings
148+
.ropeproject
149+
150+
# mkdocs documentation
151+
/site
152+
153+
# mypy
154+
.mypy_cache/
155+
.dmypy.json
156+
dmypy.json
157+
158+
# Pyre type checker
159+
.pyre/
160+
161+
# pytype static type analyzer
162+
.pytype/
163+
164+
# Cython debug symbols
165+
cython_debug/

‎CITATION.cff

+18
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
cff-version: 1.2.0
2+
message: "If you use this software, please cite it as below."
3+
preferred-citation:
4+
type: article
5+
authors:
6+
- family-names: "Meng"
7+
given-names: "Kevin"
8+
- family-names: "Sen Sharma"
9+
given-names: "Arnab"
10+
- family-names: "Andonian"
11+
given-names: "Alex"
12+
- family-names: "Belinkov"
13+
given-names: "Yonatan"
14+
- family-names: "Bau"
15+
given-names: "David"
16+
journal: "arXiv preprint arXiv:2210.07229"
17+
title: "Mass-Editing Memory in a Transformer"
18+
year: 2022

‎LICENSE

+21
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
MIT License
2+
3+
Copyright (c) 2022 Kevin Meng
4+
5+
Permission is hereby granted, free of charge, to any person obtaining a copy
6+
of this software and associated documentation files (the "Software"), to deal
7+
in the Software without restriction, including without limitation the rights
8+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9+
copies of the Software, and to permit persons to whom the Software is
10+
furnished to do so, subject to the following conditions:
11+
12+
The above copyright notice and this permission notice shall be included in all
13+
copies or substantial portions of the Software.
14+
15+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21+
SOFTWARE.

‎README.md

+89
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,89 @@
1+
# MEMIT: Mass-Editing Memory in a Transformer
2+
3+
Editing thousands of facts into a transformer memory at once.
4+
5+
<!-- [![Colab MEMIT Demo](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/kmeng01/memit/blob/main/notebooks/memit.ipynb) -->
6+
7+
## Table of Contents
8+
9+
- [Installation](#installation)
10+
- [MEMIT Algorithm Demo](#memit-algorithm-demo)
11+
- [Running the Full Evaluation Suite](#running-the-full-evaluation-suite)
12+
- [Generating Scaling Curves](#generating-scaling-curves)
13+
- [How to Cite](#how-to-cite)
14+
15+
## Installation
16+
17+
We recommend `conda` for managing Python, CUDA, and PyTorch; `pip` is for everything else. To get started, simply install `conda` and run:
18+
```bash
19+
CONDA_HOME=$CONDA_HOME ./scripts/setup_conda.sh
20+
```
21+
22+
`$CONDA_HOME` should be the path to your `conda` installation, e.g., `~/miniconda3`.
23+
24+
## MEMIT Algorithm Demo
25+
26+
[`notebooks/memit.ipynb`](notebooks/memit.ipynb) demonstrates MEMIT. The API is simple; simply specify a *requested rewrite* of the following form:
27+
28+
```python
29+
request = [
30+
{
31+
"prompt": "{} plays the sport of",
32+
"subject": "LeBron James",
33+
"target_new": {
34+
"str": "football"
35+
}
36+
},
37+
{
38+
"prompt": "{} plays the sport of",
39+
"subject": "Michael Jordan",
40+
"target_new": {
41+
"str": "baseball"
42+
}
43+
},
44+
]
45+
```
46+
47+
Other similar example(s) are included in the notebook.
48+
49+
## Running the Full Evaluation Suite
50+
51+
[`experiments/evaluate.py`](experiments/evaluate.py) can be used to evaluate any method in [`baselines/`](baselines/).
52+
53+
For example:
54+
```
55+
python3 -m experiments.evaluate \
56+
--alg_name=MEMIT \
57+
--model_name=EleutherAI/gpt-j-6B \
58+
--hparams_fname=EleutherAI_gpt-j-6B.json \
59+
--num_edits=10000 \
60+
--use_cache
61+
```
62+
Results from each run are stored at `results/<method_name>/run_<run_id>` in a specific format:
63+
```bash
64+
results/
65+
|__ MEMIT/
66+
|__ run_<run_id>/
67+
|__ params.json
68+
|__ case_0.json
69+
|__ case_1.json
70+
|__ ...
71+
|__ case_10000.json
72+
```
73+
74+
To summarize the results, you can use [`experiments/summarize.py`](experiments/summarize.py):
75+
```bash
76+
python3 -m experiments.summarize --dir_name=MEMIT --runs=run_<run1>,run_<run2>
77+
```
78+
79+
Running `python3 -m experiments.evaluate -h` or `python3 -m experiments.summarize -h` provides details about command-line flags.
80+
81+
## Generating Scaling Curves
82+
83+
asdfasdf
84+
85+
## How to Cite
86+
87+
```bibtex
88+
Coming soon!
89+
```

‎baselines/README.md

+6
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,6 @@
1+
We compare ROME against several open sourced state-of-the-art model editors. All are implemented in their respective folders. Implementations other than FT/FT+L are adapted from third parties.
2+
- Fine-Tuning (`ft`): Direct fine-tuning.
3+
- Constrained Fine-Tuning (`ft`): FT with $L_\infty$ norm constraint. Inspired by Zhu et al. [[Paper]](https://arxiv.org/abs/2012.00363)
4+
- Knowledge Neurons (`kn`): Dai et al. [[Code]](https://github.com/EleutherAI/knowledge-neurons) [[Paper]](https://arxiv.org/abs/2104.08696)
5+
- Knowledge Editor (`efk`): De Cao et al. [[Code]](https://github.com/eric-mitchell/mend) [[Paper]](https://arxiv.org/abs/2104.08164)
6+
- Model Editor Networks with Gradient Decomposition (`mend`): Mitchell et al. [[Code]](https://github.com/eric-mitchell/mend) [[Paper]](https://arxiv.org/abs/2110.11309)

‎baselines/ft/__init__.py

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
from .ft_main import FTHyperParams, apply_ft_to_model, execute_ft

‎baselines/ft/ft_hparams.py

+27
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
from dataclasses import dataclass
2+
from typing import List
3+
4+
from util.hparams import HyperParams
5+
6+
7+
@dataclass
8+
class FTHyperParams(HyperParams):
9+
# Method
10+
layers: List[int]
11+
num_steps: int
12+
lr: float
13+
weight_decay: float
14+
kl_factor: float
15+
norm_constraint: float
16+
17+
# Module templates
18+
rewrite_module_tmp: str
19+
layer_module_tmp: str
20+
mlp_module_tmp: str
21+
attn_module_tmp: str
22+
ln_f_module: str
23+
lm_head_module: str
24+
25+
# Defaults
26+
batch_size: int = 64
27+
wd_power_law: tuple = None # Scale weight decay by number of edits

0 commit comments

Comments
 (0)