how to achieve limited "look ahead" for calc of offset to endian flag

[schiebel]

Hello,

Apologies if this is the wrong venue for questions, but... I am working on a project to implement a Kaitai Struct reader for a complicated storage format.

I would like to use the endian calc feature, but the endian key occurs after useful metadata (convenient!). This is the start of the metadata spec:

seq:
    - id: magic
      contents: [ 0xbe, 0xbe, 0xbe, 0xbe ]
      doc: |
        casa magic unsigned int 3200171710  190 190 190 190
    - id: size
      type: u4
    - id: type
      type: string
    - id: version
      type: u4
    - id: nrows
      type:
          switch-on: version
          cases:
            1: u4
            2: u4
            3: u8
    - id: little_endian
      size: 4

The 4 little_endian bytes will be either [ 0x0, 0x0, 0x0, 0x0] or [ 0x0, 0x0, 0x0, 0x1]. What is the best way to handle this need to "look ahead" to determine the endianness to be able to get the proper size and version values? The extended metadata ksy (with endianness hard-coded) can be found here.

Many thanks, in advance, for any help.

[Mingun]

Use

...
instances:
  little_endian:
    pos: ...
    size: 4

[schiebel]

Great, thanks @Mingun !

This is very helpful. I am very close to getting this to work. Unfortunately, the offset of the big_endian field (which was incorrectly labeled as "little" above) is determined by the version field. I tried to work around the limited contexts in which switch-on is allowed using work arounds others have shared. However, this fails because even a u1 cannot be parsed without having an endianness defined. So for example, this code:

meta:
    id: metadata
    endian:
        switch-on: _root.big_endian_value
        cases:
          '[0x0, 0x0, 0x0, 0x0]': le
          '[0x0, 0x0, 0x0, 0x1]': be
seq:
    - id: magic
      contents: [ 0xbe, 0xbe, 0xbe, 0xbe ]
      doc: |
        casa magic unsigned int 3200171710  190 190 190 190
    - id: size
      type: u4
    - id: type
      type: string
    - id: version
      type: u4
    - id: nrows
      type:
          switch-on: version
          cases:
            1: u4
            2: u4
            3: u8
    - id: big_endian
      type: u4
instances:
    version_bytes:
        pos: 17
        size: 4
    version_size:
        size: 0
        type:
            switch-on: version_bytes
            cases:
              '[0x0, 0x0, 0x0, 0x1]': u1_wrapper(4)
              '[0x0, 0x0, 0x0, 0x2]': u1_wrapper(4)
              '[0x0, 0x0, 0x0, 0x3]': u1_wrapper(8)
              '[0x0, 0x1, 0x0, 0x0]': u1_wrapper(4)
              '[0x0, 0x2, 0x0, 0x0]': u1_wrapper(4)
              '[0x0, 0x3, 0x0, 0x0]': u1_wrapper(8)
    big_endian_value:
        pos: 21 + version_size.as<u1_wrapper>.value
        size: 4
types:
    u1_wrapper:
        params:
          - id: value
            type: u1

results in the error:

/metadata.rb:2934:in 'Metadata::U1Wrapper#_read': /types/u1_wrapper: unable to decide on endianness for a type (Kaitai::Struct::UndecidedEndiannessError)
	from metadata.rb:3059:in 'Metadata#version_size'
	from metadata.rb:2987:in 'Metadata#big_endian_value'

Attempting to add endian: be within u1_wrapper results in a syntax error.

Is there some other way to work around this problem?

Thanks again for the help....

[Mingun]

It seems that version and big_endian fields are always writted in BE endianness, so you can explicitly use corresponding type. Try to use something like this:

meta:
  id: metadata
  endian:
    switch-on: _root.big_endian
    cases:
      0: le
      1: be
seq:
  - id: magic
    contents: [ 0xbe, 0xbe, 0xbe, 0xbe ]
    doc: |
      casa magic unsigned int 3200171710  190 190 190 190
  - id: size
    type: u4
  - id: type
    type: string
  - type: u4 # version
  - id: nrows
    type:
      switch-on: version
      cases:
        1: u4
        2: u4
        3: u8
  - type: u4 # big_endian
instances:
    version:
      pos: 17
      type: u4be
    big_endian:
      pos: '21 + (version == 3 ? 8 : 4)'
      type: u4be

[schiebel]

Hi @Mingun thanks again for your help. Unfortunately, the version field is not written in big endian format (although this would have been better). The fact that the size of nrows depends on version complicates parsing because big_endian (which is endian neutral because in the little endian case it is 0) lies beyond the size varying nrows and version is dependent on the endianness.

However, your comment gave me the idea that since there are only three versions the version flag could be used to determine the endianness:

meta:
  id: metadata
  endian:
    switch-on: version_bytes
    cases:
      '[0x0, 0x0, 0x0, 0x1]': be
      '[0x0, 0x0, 0x0, 0x2]': be
      '[0x0, 0x0, 0x0, 0x3]': be
      '[0x0, 0x1, 0x0, 0x0]': le
      '[0x0, 0x2, 0x0, 0x0]': le
      '[0x0, 0x3, 0x0, 0x0]': le
seq:
  - id: magic
    contents: [ 0xbe, 0xbe, 0xbe, 0xbe ]
  - id: size
    type: u4
  - id: type
    type: string
  - id: version
    type: u4
instances:
  version_bytes:
    pos: 17
    size: 4

It does seem like having the switch-on available in more contexts would be very helpful. However, this alternative resolves my problem (although please let me know if this solution could be expressed in a better or more compact way).

I appreciate your help very much.

[Mingun]

You also can parse version as u4be, but check it against the numbers written both in BE and LE formats. This should be more performant than comparing byte arrays:

meta:
  id: metadata
  endian:
    switch-on: version_bytes
    cases:
      0x00000001: be
      0x00000002: be
      0x00000003: be
      # You are not wrong here? It seems that correct numbers should be
      # 0x01000000: le
      # 0x02000000: le
      # 0x03000000: le
      0x00010000: le
      0x00020000: le
      0x00030000: le
seq:
  - id: magic
    contents: [ 0xbe, 0xbe, 0xbe, 0xbe ]
  - id: size
    type: u4
  - id: type
    type: string
  - id: version
    type: u4
instances:
  version_bytes:
    pos: 17
    type: u4be

[schiebel]

  # You are not wrong here? It seems that correct numbers should be
  # 0x01000000: le
  # 0x02000000: le
  # 0x03000000: le

Indeed, it's true! Both this (of course) and u4be seem like improvements. Thanks again @Mingun !