numpy arrays read from attributes are not writable

[kmuehlbauer]

I'm trying to wrap my head around the following issue. Arrays read from Datasets are created with numpy.memmap with "copyonwrite" flag set. So these numpy arrays can be changed in memory but not in the file. This is not true for arrays read from attributes. There we get back a numpy.ndarray with the writable-flag set to FALSE. An MCVE is attached below.

I was already looking where the flag might be set and why, but didn't come far. Any hints much appreciated.

MCVE:

import numpy as np
import h5py
import pyfive

with h5py.File("test.h5", mode="w") as f:
    f.attrs["att"] = np.arange(10)
    f["arr"] = np.arange(10)

with pyfive.File("test.h5") as f:
    # dataset
    arr = f["arr"][:]
    print(type(arr))
    print(arr.flags)
    print(arr.base)
    arr += 1
    print(arr)
    # attribute
    att = f.attrs["att"]
    print(type(att))
    print(att.flags)
    print(att.base)
    att += 1
    print(att)

<class 'numpy.memmap'>
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : True
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

[0 1 2 3 4 5 6 7 8 9]
[ 1  2  3  4  5  6  7  8  9 10]
<class 'numpy.ndarray'>
  C_CONTIGUOUS : True
  F_CONTIGUOUS : True
  OWNDATA : False
  WRITEABLE : False
  ALIGNED : True
  WRITEBACKIFCOPY : False
  UPDATEIFCOPY : False

[0 1 2 3 4 5 6 7 8 9]

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Input In [23], in <cell line: 9>()
     20 print(att.flags)
     21 print(att.base)
---> 22 att += 1
     23 print(att)

ValueError: output array is read-only

[kmuehlbauer]

The main issue is obviously that np.frombuffer in DataObjects._attr_value() returns an immutable np.array as it is created via bytes:

https://github.com/jjhelmus/pyfive/blob/63482314a2fc6a538053237bc07dd8331030ddaa/pyfive/dataobjects.py#L223

I've tested this adding a .copy() to the np.frombuffer call and the everything works fine.

Wouldn't it make sense to use the same approach here as in get_contiguous_data (via np.memmap)?

https://github.com/jjhelmus/pyfive/blob/63482314a2fc6a538053237bc07dd8331030ddaa/pyfive/dataobjects.py#L445-L459

Not sure how much will be involved to make that change. @bmaranville Do you have any thoughts on that?

[bmaranville]

I think the data access methods for attributes and datasets are almost the same, but we are handling them separately, and with different datatypes enabled for each:

Attribute

• VLEN_SEQUENCE
• VLEN_STRING
• REFERENCE (single)
• simple datatypes {float, int, fixed-length string} (frombuffer)

Dataset

• REFERENCE(array)
• simple datatypes (memmap)

I suspect there would be a big benefit from combining the handling of reading data from attributes and datasets, so that the combined feature set would be better for both.

[bnlawrence]

Just a comment. This interacts with #60, insofar as I have had to introduce a fall back which investigates whether or not the underlying file is a "true posix file", and if it's not, it can't use the memory map feature, which means for contiguous data, you have to read the entire array in a very unlazy way. (I do have code for introducing what I am calling pseudo chunking, but that seems very klunky).