Unable to train BERT-NER model after adding output path

[zz-em]

Everything looks good with the following except not saving the trained model:

!python -m spacy train ./config/config.cfg --paths.train ./data/train.spacy --paths.dev ./data/train.spacy --gpu-id 0

After adding output path:

!python -m spacy train ./config/config.cfg --output ./output/ --paths.train ./data/train.spacy --paths.dev ./data/train.spacy --gpu-id 0

Error message pops up:

⚠ Aborting and saving the final best model. Encountered exception:
OSError(22, 'Invalid argument')
Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/thinc/model.py", line 550, in to_disk
    file_.write(self.to_bytes())
OSError: [Errno 22] Invalid argument

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/training/loop.py", line 124, in train
    raise e
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/training/loop.py", line 112, in train
    save_checkpoint(is_best_checkpoint)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/training/loop.py", line 68, in save_checkpoint
    before_to_disk(nlp).to_disk(output_path / DIR_MODEL_LAST)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/language.py", line 2036, in to_disk
    util.to_disk(path, serializers, exclude)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/util.py", line 1356, in to_disk
    writer(path / key)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/language.py", line 2034, in <lambda>
    serializers[name] = lambda p, proc=proc: proc.to_disk(p, exclude=["vocab"])  # type: ignore[misc]
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy_transformers/pipeline_component.py", line 373, in to_disk
    util.to_disk(path, serialize, exclude)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/util.py", line 1356, in to_disk
    writer(path / key)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy_transformers/pipeline_component.py", line 372, in <lambda>
    serialize["model"] = lambda p: self.model.to_disk(p)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/thinc/model.py", line 550, in to_disk
    file_.write(self.to_bytes())
OSError: [Errno 22] Invalid argument

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/cli/_util.py", line 74, in setup_cli
    command(prog_name=COMMAND)
  File "/databricks/python/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/databricks/python/lib/python3.9/site-packages/click/core.py", line 1053, in main
    rv = self.invoke(ctx)
  File "/databricks/python/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/databricks/python/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/databricks/python/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/databricks/python/lib/python3.9/site-packages/typer/main.py", line 532, in wrapper
    return callback(**use_params)  # type: ignore
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/cli/train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/cli/train.py", line 75, in train
    train_nlp(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/training/loop.py", line 128, in train
    save_checkpoint(False)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/training/loop.py", line 68, in save_checkpoint
    before_to_disk(nlp).to_disk(output_path / DIR_MODEL_LAST)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/language.py", line 2036, in to_disk
    util.to_disk(path, serializers, exclude)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/util.py", line 1356, in to_disk
    writer(path / key)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/language.py", line 2034, in <lambda>
    serializers[name] = lambda p, proc=proc: proc.to_disk(p, exclude=["vocab"])  # type: ignore[misc]
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy_transformers/pipeline_component.py", line 373, in to_disk
    util.to_disk(path, serialize, exclude)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy/util.py", line 1356, in to_disk
    writer(path / key)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/spacy_transformers/pipeline_component.py", line 372, in <lambda>
    serialize["model"] = lambda p: self.model.to_disk(p)
  File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-26ff7be3-a618-4556-a557-840c6fa45aa5/lib/python3.9/site-packages/thinc/model.py", line 549, in to_disk
    with path.open("wb") as file_:
  File "/usr/lib/python3.9/pathlib.py", line 1242, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/usr/lib/python3.9/pathlib.py", line 1110, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'output/model-last/transformer/model'

Here is the config file:

# This is an auto-generated partial config. To use it with 'spacy train'
# you can run spacy init fill-config to auto-fill all default settings:
# python -m spacy init fill-config ./base_config.cfg ./config.cfg
[paths]
train = null
dev = null
vectors = null
[system]
gpu_allocator = "pytorch"

[nlp]
lang = "en"
pipeline = ["transformer","ner"]
batch_size = 128

[components]

[components.transformer]
factory = "transformer"

[components.transformer.model]
@architectures = "spacy-transformers.TransformerModel.v3"
name = "nlpaueb/bert-base-uncased-contracts"
tokenizer_config = {"use_fast": true}

[components.transformer.model.get_spans]
@span_getters = "spacy-transformers.strided_spans.v1"
window = 128
stride = 96

[components.ner]
factory = "ner"

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = false
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy-transformers.TransformerListener.v1"
grad_factor = 1.0

[components.ner.model.tok2vec.pooling]
@layers = "reduce_mean.v1"

[corpora]

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0

[training]
accumulate_gradient = 3
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"

[training.optimizer]
@optimizers = "Adam.v1"

[training.optimizer.learn_rate]
@schedules = "warmup_linear.v1"
warmup_steps = 250
total_steps = 20000
initial_rate = 5e-5

[training.batcher]
@batchers = "spacy.batch_by_padded.v1"
discard_oversize = true
size = 2000
buffer = 256

[initialize]
vectors = ${paths.vectors}

[danieldk]

This is an error that you would typically get when the user or process does not have permissions to write to the current directory. You could try to debug this by e.g. creating a directory with os.mkdir to see if that works. You could also try use an absolute path as an argument to --output of which you are sure that the process has write access.

[zz-em]

Thank you for the response. I do believe it's the 'write access' issue. However, based on the following experiment, I think SpaCy somehow modified the write permission of the directory.

• With the following config, I can train model and save it without any problem.

✔ Saved pipeline to output directory
output/model-last

# This is an auto-generated partial config. To use it with 'spacy train'
# you can run spacy init fill-config to auto-fill all default settings:
# python -m spacy init fill-config ./base_config.cfg ./config.cfg
[paths]
train = null
dev = null
vectors = null
[system]
gpu_allocator = null

[nlp]
lang = "en"
pipeline = ["tok2vec","ner"]
batch_size = 1000

[components]

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"]
rows = [5000, 1000, 2500, 2500]
include_static_vectors = false

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 96
depth = 4
window_size = 1
maxout_pieces = 3

[components.ner]
factory = "ner"

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}

[corpora]

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"

[training.optimizer]
@optimizers = "Adam.v1"

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001

[initialize]
vectors = ${paths.vectors}

• However, similar error like the 'transformer model' occurred when I set vectors = 'en_core_web_lg". I tried to create a directory with os.mkdir. I confirmed I had 'write permission' after creating the directory. However, after running the training pipeline, the write permission of directory output/model-last/vocab/vectors became False.

# This is an auto-generated partial config. To use it with 'spacy train'
# you can run spacy init fill-config to auto-fill all default settings:
# python -m spacy init fill-config ./base_config.cfg ./config.cfg
[paths]
train = null
dev = null
vectors = "en_core_web_lg"
[system]
gpu_allocator = null

[nlp]
lang = "en"
pipeline = ["tok2vec","ner"]
batch_size = 1000

[components]

[components.tok2vec]
factory = "tok2vec"

[components.tok2vec.model]
@architectures = "spacy.Tok2Vec.v2"

[components.tok2vec.model.embed]
@architectures = "spacy.MultiHashEmbed.v2"
width = ${components.tok2vec.model.encode.width}
attrs = ["NORM", "PREFIX", "SUFFIX", "SHAPE"]
rows = [5000, 1000, 2500, 2500]
include_static_vectors = true

[components.tok2vec.model.encode]
@architectures = "spacy.MaxoutWindowEncoder.v2"
width = 256
depth = 8
window_size = 1
maxout_pieces = 3

[components.ner]
factory = "ner"

[components.ner.model]
@architectures = "spacy.TransitionBasedParser.v2"
state_type = "ner"
extra_state_tokens = false
hidden_width = 64
maxout_pieces = 2
use_upper = true
nO = null

[components.ner.model.tok2vec]
@architectures = "spacy.Tok2VecListener.v1"
width = ${components.tok2vec.model.encode.width}

[corpora]

[corpora.train]
@readers = "spacy.Corpus.v1"
path = ${paths.train}
max_length = 0

[corpora.dev]
@readers = "spacy.Corpus.v1"
path = ${paths.dev}
max_length = 0

[training]
dev_corpus = "corpora.dev"
train_corpus = "corpora.train"

[training.optimizer]
@optimizers = "Adam.v1"

[training.batcher]
@batchers = "spacy.batch_by_words.v1"
discard_oversize = false
tolerance = 0.2

[training.batcher.size]
@schedules = "compounding.v1"
start = 100
stop = 1000
compound = 1.001

[initialize]
vectors = ${paths.vectors}

⚠ Aborting and saving the final best model. Encountered exception:
OSError(22, 'Invalid argument')
Traceback (most recent call last):
  File "spacy/vectors.pyx", line 579, in spacy.vectors.Vectors.to_disk.save_vectors
  File "spacy/vectors.pyx", line 569, in spacy.vectors.Vectors.to_disk.lambda2
  File "<__array_function__ internals>", line 5, in save
  File "/databricks/python/lib/python3.9/site-packages/numpy/lib/npyio.py", line 529, in save
    format.write_array(fid, arr, allow_pickle=allow_pickle,
  File "/databricks/python/lib/python3.9/site-packages/numpy/lib/format.py", line 691, in write_array
    array.tofile(fp)
OSError: 154247100 requested and 52397024 written

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/training/loop.py", line 124, in train
    raise e
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/training/loop.py", line 112, in train
    save_checkpoint(is_best_checkpoint)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/training/loop.py", line 68, in save_checkpoint
    before_to_disk(nlp).to_disk(output_path / DIR_MODEL_LAST)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/language.py", line 2036, in to_disk
    util.to_disk(path, serializers, exclude)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/util.py", line 1356, in to_disk
    writer(path / key)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/language.py", line 2035, in <lambda>
    serializers["vocab"] = lambda p: self.vocab.to_disk(p, exclude=exclude)
  File "spacy/vocab.pyx", line 472, in spacy.vocab.Vocab.to_disk
  File "spacy/vectors.pyx", line 587, in spacy.vectors.Vectors.to_disk
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/util.py", line 1356, in to_disk
    writer(path / key)
  File "spacy/vectors.pyx", line 583, in spacy.vectors.Vectors.to_disk.lambda5
  File "spacy/vectors.pyx", line 578, in spacy.vectors.Vectors.to_disk.save_vectors
OSError: [Errno 22] Invalid argument

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/__main__.py", line 4, in <module>
    setup_cli()
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/cli/_util.py", line 74, in setup_cli
    command(prog_name=COMMAND)
  File "/databricks/python/lib/python3.9/site-packages/click/core.py", line 1128, in __call__
    return self.main(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/typer/core.py", line 778, in main
    return _main(
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/typer/core.py", line 216, in _main
    rv = self.invoke(ctx)
  File "/databricks/python/lib/python3.9/site-packages/click/core.py", line 1659, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/databricks/python/lib/python3.9/site-packages/click/core.py", line 1395, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/databricks/python/lib/python3.9/site-packages/click/core.py", line 754, in invoke
    return __callback(*args, **kwargs)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/typer/main.py", line 683, in wrapper
    return callback(**use_params)  # type: ignore
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/cli/train.py", line 45, in train_cli
    train(config_path, output_path, use_gpu=use_gpu, overrides=overrides)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/cli/train.py", line 75, in train
    train_nlp(nlp, output_path, use_gpu=use_gpu, stdout=sys.stdout, stderr=sys.stderr)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/training/loop.py", line 128, in train
    save_checkpoint(False)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/training/loop.py", line 68, in save_checkpoint
    before_to_disk(nlp).to_disk(output_path / DIR_MODEL_LAST)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/language.py", line 2036, in to_disk
    util.to_disk(path, serializers, exclude)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/util.py", line 1356, in to_disk
    writer(path / key)
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/language.py", line 2035, in <lambda>
    serializers["vocab"] = lambda p: self.vocab.to_disk(p, exclude=exclude)
  File "spacy/vocab.pyx", line 472, in spacy.vocab.Vocab.to_disk
  File "spacy/vectors.pyx", line 587, in spacy.vectors.Vectors.to_disk
  File "/local_disk0/.ephemeral_nfs/cluster_libraries/python/lib/python3.9/site-packages/spacy/util.py", line 1356, in to_disk
    writer(path / key)
  File "spacy/vectors.pyx", line 583, in spacy.vectors.Vectors.to_disk.lambda5
  File "spacy/vectors.pyx", line 578, in spacy.vectors.Vectors.to_disk.save_vectors
  File "/usr/lib/python3.9/pathlib.py", line 1242, in open
    return io.open(self, mode, buffering, encoding, errors, newline,
  File "/usr/lib/python3.9/pathlib.py", line 1110, in _opener
    return self._accessor.open(self, flags, mode)
FileNotFoundError: [Errno 2] No such file or directory: 'output/model-last/vocab/vectors'

[adrianeboyd]

From this error it looks like you may be running out of disk space:

OSError: 154247100 requested and 52397024 written