tensorflow InvalidArgumentError: num_lower must be negative or less or equal to number of rows

[arne-cl]

I discovered via issue NLPbox/stagedp-service#1 that NeuralEDUSeg crashes on some, mostly very short, input sentences:

$ cat /tmp/bad_input.txt                                                                                                                                                                        good food .

$ docker run -v /tmp:/tmp -it neural-edu-seg /tmp/bad_input.txt
Loading NeuralEDUSeg models...
[...]
Finished loading NeuralEDUSeg models.

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1329, in _run_fn
    status, run_metadata)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: num_lower must be negative or less or equal to number of rows (3) got: 5
         [[Node: encoding/bilinear_attention/MatrixBandPart = MatrixBandPart[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](encoding/bilinear_attention/ones_like, encoding/bilinear_attention/MatrixBandPart/num_lower,
encoding/bilinear_attention/MatrixBandPart/num_lower)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "neuralseg/splitter.py", line 334, in <module>
    main()
  File "neuralseg/splitter.py", line 323, in main
    result = segment_text(input_text, rst_data, model, spacy_nlp, output_format=args.output_format, debug=args.debug)
  File "neuralseg/splitter.py", line 279, in segment_text
    edus, spacy_edus, json_edus = segment_batches(model, data_batches, text, debug=debug)
  File "neuralseg/splitter.py", line 231, in segment_batches
    batch_edu_breaks = model.segment(batch)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/elmo_crf_seg.py", line 125, in segment
    feed_dict)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1128, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1344, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1363, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: num_lower must be negative or less or equal to number of rows (3) got: 5
         [[Node: encoding/bilinear_attention/MatrixBandPart = MatrixBandPart[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](encoding/bilinear_attention/ones_like, encoding/bilinear_attention/MatrixBandPart/num_lower,
encoding/bilinear_attention/MatrixBandPart/num_lower)]]

Caused by op 'encoding/bilinear_attention/MatrixBandPart', defined at:
  File "neuralseg/splitter.py", line 334, in <module>
    main()
  File "neuralseg/splitter.py", line 321, in main
    rst_data, model, spacy_nlp = load_models(args)
  File "neuralseg/splitter.py", line 74, in load_models
    model = AttnSegModel(args, word_vocab)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/elmo_crf_seg.py", line 17, in __init__
    super().__init__(args, word_vocab)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/base_seg.py", line 37, in __init__
    self._build_graph()
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/lstm_seg.py", line 20, in _build_graph
    self._encode()
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/atten_seg.py", line 24, in _encode
    self.window_size)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/layers.py", line 51, in self_attention
    window_size, window_size)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2409, in matrix_band_part
    num_upper=num_upper, name=name)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3160, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1625, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): num_lower must be negative or less or equal to number of rows (3) got: 5
         [[Node: encoding/bilinear_attention/MatrixBandPart = MatrixBandPart[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](encoding/bilinear_attention/ones_like, encoding/bilinear_attention/MatrixBandPart/num_lower,
encoding/bilinear_attention/MatrixBandPart/num_lower)]]

Expected behaviour would be sth. like:

$ cat /tmp/good_input.txt 
Judging from previous posts this used to be a good place, but not any longer.

$ docker run -v /tmp:/tmp -it neural-edu-seg /tmp/good_input.txt
Loading NeuralEDUSeg models...
[...]
Finished loading NeuralEDUSeg models.
{"paragraphs": [{"start": 0, "end": 78}], "sentences": [{"start": 0, "end": 77}], "edus": [{"start": 0, "end": 28}, {"start": 28, "end": 77}]}

[arne-cl]

Note: the example given in the description is using my wrapper around NeuralEDUSeg.
This is how it looks like using the original NeuralEDUSeg:

root@f697ebdcda53:/opt/neural-edu-seg/neuralseg# python run.py --segment --input_files /tmp/bad_input.txt --result_dir /tmp/results/
 2021-02-24 22:00:12,457 - SegEDU - INFO - Running with args : Namespace(batch_size=32, dev_files=None, dropout_keep_prob=0.9, ema_decay=0.9999, epochs=50, evaluate=False, gpu=None, hidden_size=200, input_files=['/tmp/bad_input.txt'], lea
rning_rate=0.001, log_path=None, max_grad_norm=5.0, model_dir='/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/data/models', optim='adam', prepare=False, result_dir='/tmp/results/', rst_dir='../data/rst/', seed=123, seg
ment=True, test_files=None, train=False, train_files=None, weight_decay=0.0001, window_size=5, word_embed_path='../data/embeddings/glove.840B.300d.txt', word_embed_size=300, word_vocab_path='../data/vocab/word.vocab')
 2021-02-24 22:00:12,457 - SegEDU - INFO - Loading vocab...
 2021-02-24 22:00:12,522 - SegEDU - INFO - Word vocab size: 17243
 2021-02-24 22:00:12,522 - SegEDU - INFO - Loading the model...
2021-02-24 22:00:12.522775: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:97: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
 2021-02-24 22:00:17,873 - SegEDU - INFO - There are 4043611 parameters in the model
 2021-02-24 22:00:17,873 - SegEDU - INFO - Using Exp Moving Average to train the model with decay 0.9999.
 2021-02-24 22:00:19,063 - SegEDU - INFO - Time to build graph: 6.5401506423950195 s
/usr/local/lib/python3.6/site-packages/sklearn/utils/linear_assignment_.py:22: FutureWarning: The linear_assignment_ module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead.
  FutureWarning)
 2021-02-24 22:00:32,036 - SegEDU - INFO - Model restored from /usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/data/models/best
 2021-02-24 22:00:32,947 - SegEDU - INFO - Segmenting /tmp/bad_input.txt...
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1350, in _do_call
    return fn(*args)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1329, in _run_fn
    status, run_metadata)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/errors_impl.py", line 473, in __exit__
    c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.InvalidArgumentError: num_lower must be negative or less or equal to number of rows (3) got: 5
         [[Node: encoding/bilinear_attention/MatrixBandPart = MatrixBandPart[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](encoding/bilinear_attention/ones_like, encoding/bilinear_attention/MatrixBandPart/num_lower,
encoding/bilinear_attention/MatrixBandPart/num_lower)]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "run.py", line 50, in <module>
    segment(args)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/api.py", line 157, in segment
    batch_pred_segs = model.segment(batch)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/elmo_crf_seg.py", line 125, in segment
    feed_dict)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 895, in run
    run_metadata_ptr)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1128, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1344, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/client/session.py", line 1363, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: num_lower must be negative or less or equal to number of rows (3) got: 5
         [[Node: encoding/bilinear_attention/MatrixBandPart = MatrixBandPart[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](encoding/bilinear_attention/ones_like, encoding/bilinear_attention/MatrixBandPart/num_lower,
encoding/bilinear_attention/MatrixBandPart/num_lower)]]

Caused by op 'encoding/bilinear_attention/MatrixBandPart', defined at:
  File "run.py", line 50, in <module>
    segment(args)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/api.py", line 133, in segment
    model = AttnSegModel(args, word_vocab)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/elmo_crf_seg.py", line 17, in __init__
    super().__init__(args, word_vocab)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/base_seg.py", line 37, in __init__
    self._build_graph()
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/lstm_seg.py", line 20, in _build_graph
    self._encode()
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/atten_seg.py", line 24, in _encode
    self.window_size)
  File "/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/neuralseg/layers.py", line 51, in self_attention
    window_size, window_size)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py", line 2409, in matrix_band_part
    num_upper=num_upper, name=name)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3160, in create_op
    op_def=op_def)
  File "/usr/local/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1625, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InvalidArgumentError (see above for traceback): num_lower must be negative or less or equal to number of rows (3) got: 5
         [[Node: encoding/bilinear_attention/MatrixBandPart = MatrixBandPart[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"](encoding/bilinear_attention/ones_like, encoding/bilinear_attention/MatrixBandPart/num_lower,
encoding/bilinear_attention/MatrixBandPart/num_lower)]]

It should look like this:

root@f697ebdcda53:/opt/neural-edu-seg/neuralseg# python run.py --segment --input_files /tmp/good_input.txt --result_dir /tmp/results/
 2021-02-24 21:58:50,750 - SegEDU - INFO - Running with args : Namespace(batch_size=32, dev_files=None, dropout_keep_prob=0.9, ema_decay=0.9999, epochs=50, evaluate=False, gpu=None, hidden_size=200, input_files=['/tmp/good_input.txt'], learning_rate=0.001, log_path=None, max_grad_norm=5.0, model_dir='/usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/data/models', optim='adam', prepare=False, result_dir='/tmp/results/', rst_dir='../data/rst/', seed=123, segment=True, test_files=None, train=False, train_files=None, weight_decay=0.0001, window_size=5, word_embed_path='../data/embeddings/glove.840B.300d.txt', word_embed_size=300, word_vocab_path='../data/vocab/word.vocab')
 2021-02-24 21:58:50,750 - SegEDU - INFO - Loading vocab...
 2021-02-24 21:58:50,822 - SegEDU - INFO - Word vocab size: 17243
 2021-02-24 21:58:50,823 - SegEDU - INFO - Loading the model...
2021-02-24 21:58:50.823501: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
/usr/local/lib/python3.6/site-packages/tensorflow/python/ops/gradients_impl.py:97: UserWarning: Converting sparse IndexedSlices to a dense Tensor of unknown shape. This may consume a large amount of memory.
  "Converting sparse IndexedSlices to a dense Tensor of unknown shape. "
 2021-02-24 21:58:55,833 - SegEDU - INFO - There are 4043611 parameters in the model
 2021-02-24 21:58:55,834 - SegEDU - INFO - Using Exp Moving Average to train the model with decay 0.9999.
 2021-02-24 21:58:56,994 - SegEDU - INFO - Time to build graph: 6.170761823654175 s
/usr/local/lib/python3.6/site-packages/sklearn/utils/linear_assignment_.py:22: FutureWarning: The linear_assignment_ module is deprecated in 0.21 and will be removed from 0.23. Use scipy.optimize.linear_sum_assignment instead.
  FutureWarning)
 2021-02-24 21:59:10,462 - SegEDU - INFO - Model restored from /usr/local/lib/python3.6/site-packages/neuralseg-0.1.0a0-py3.6.egg/data/models/best
 2021-02-24 21:59:11,021 - SegEDU - INFO - Segmenting /tmp/good_input.txt...
 2021-02-24 21:59:11,929 - SegEDU - INFO - Saving into /tmp/results/good_input.txt
root@f697ebdcda53:/opt/neural-edu-seg/neuralseg# cat /tmp/results/good_input.txt 
Judging from previous posts
this used to be a good place , but not any longer .

[arne-cl]

Potential solution: PKU-TANGENT#1 (issue was closed, seemingly without fixing it).

I reported the issue again upstream: PKU-TANGENT#9.