Ruby: `b1` arrays do not work

[lafrecciablu]

Background: I am working on Eurocontrol ASTERIX format, which strongly relies on bit fields for knowing if a field is present in the stream or not.
Having decoding problems, I nailed it down to a simple one byte example, which shows the problem.
In short:

• Using a bit array through a repeat expression does not work.
• On the other side, defining each bit one by one works.
  This may be related to Incorrect insertion of align_to_byte() / alignToByte() calls #1070, even though I wouldn't understand why we may have an alignment problem on one byte.

Example:

meta:
  id: fspec_one_byte_repeat_bits
  endian: be

doc: |
  To unit test it, you may use the following procedure:
  - Create one byte binary file having first 4 bits set to 0 and next 4 bits set to 1:
    echo -ne '\x0F' | dd of=/tmp/fspec0F.bin bs=1
  - Optional: display it
    xxd /tmp/fspec0F.bin
  - Test it:
    ksdump /tmp/fspec0F.bin fspec_one_byte_repeat_bits.ksy

seq:
  - id: fspec
    type: fspec_byte

types:
  fspec_byte:
    seq:
      - id: indicators
        type: b1
        repeat: expr
        repeat-expr: 8

instances:
  ind_0:
    doc: returns 0 for 0 value, which is correct value, but wrong type (should be bool false)
    value: fspec.indicators[0]
  ind_0_is_false:
    doc: returns false for 0 value, which is wrong
    value: ind_0 == false
  ind_0_is_true:
    doc: returns false for 0 value, which is correct
    value: ind_0 == true
  ind_0_to_i:
    doc: returns 1 for 0 value, which is wrong
    value: ind_0.to_i
  ind_0_is_zero:
    doc: |
      returns false for 0 value, which is wrong
      note: Even though ind_0 is displayed as 0 (not false), it needs to be casted to integer to be compared to 0,
            otherwise you get
              "can't compare BitsType1(BigBitEndian) and Int1Type(true)"
            This is inconsistent.
    value: ind_0.to_i == 0
  ind_0_is_one:
    doc: returns true for 0 value, which is wrong
    value: ind_0.to_i == 1
  ind_0_ternary:
    doc: returns 1 for 0 value, which is wrong
    value: 'ind_0 ? 1 : 0'

  ind_4:
    value: fspec.indicators[4]
  ind_4_is_true:
    doc: returns false for 1 value, which is wrong
    value: ind_4 == true
  ind_4_is_false:
    doc: returns false for 1 value, which is correct
    value: ind_4 == false
  ind_4_to_i:
    doc: returns 1 for 1 value, which is correct
    value: ind_4.to_i
  ind_4_is_zero:
    doc: returns false for 1 value, which is correct
    value: ind_4.to_i == 0
  ind_4_is_one:
    doc: returns true for 1 value, which is correct
    value: ind_4.to_i == 1
  ind_4_ternary:
    doc: returns 1 for 1 value, which is correct
    value: 'ind_4 ? 1 : 0'

I have documented in the doc fields above what is correct and what is not.
But here is the corresponding ksdump output:

$ echo -ne '\x0F' | dd of=/tmp/fspec0F.bin bs=1
$ ksdump /tmp/fspec0F.bin fspec_one_byte_repeat_bits.ksy
fspec:
  indicators:
  - 0
  - 0
  - 0
  - 0
  - 1
  - 1
  - 1
  - 1
ind_0: 0
ind_0_is_false: false
ind_0_is_one: true
ind_0_is_true: false
ind_0_is_zero: false
ind_0_ternary: 1
ind_0_to_i: 1
ind_4: 1
ind_4_is_false: false
ind_4_is_one: true
ind_4_is_true: false
ind_4_is_zero: false
ind_4_ternary: 1
ind_4_to_i: 1

For the records, the same example using dedicated explicit bit fields works as expected:

meta:
  id: fspec_one_byte_on_separate_bits
  endian: be

doc: |
  To unit test it, you may use the following procedure:
  - Create one byte binary file having first 4 bits set to 0 and next 4 bits set to 1:
    echo -ne '\x0F' | dd of=/tmp/fspec0F.bin bs=1
  - Optional: display it
    xxd /tmp/fspec0F.bin
  - Test it:
    ksdump /tmp/fspec0F.bin fspec_one_byte_on_separate_bits.ksy

seq:
  - id: fspec
    type: fspec_byte

types:
  fspec_byte:
    seq:
      - id: bit_0
        type: b1
      - id: bit_1
        type: b1
      - id: bit_2
        type: b1
      - id: bit_3
        type: b1
      - id: bit_4
        type: b1
      - id: bit_5
        type: b1
      - id: bit_6
        type: b1
      - id: bit_7
        type: b1

instances:
  ind_0:
    doc: returns false for 0 value, which is correct value and type
    value: fspec.bit_0
  ind_0_is_false:
    doc: returns true for 0 value, which is correct
    value: ind_0 == false
  ind_0_is_zero:
    doc: returns true for 0 value, which is correct
    value: ind_0.to_i == 0
  ind_0_ternary:
    doc: returns 0 for 0 value, which is correct
    value: 'ind_0 ? 1 : 0'

  ind_4:
    doc: returns true for 1 value, which is correct value and type
    value: fspec.bit_4
  ind_4_is_true:
    doc: returns true for 1 value, which is correct
    value: ind_4 == true
  ind_4_is_one:
    doc: returns true for 1 value, which is correct
    value: ind_4.to_i == 1
  ind_4_to_i:
    doc: returns 1 for 1 value, which is correct
    value: ind_4.to_i
  ind_4_ternary:
    doc: returns 1 for 1 value, which is correct
    value: 'ind_4 ? 1 : 0'

$ ksdump /tmp/fspec0F.bin fspec_one_byte_on_separate_bits.ksy
fspec:
  bit_0: false
  bit_1: false
  bit_2: false
  bit_3: false
  bit_4: true
  bit_5: true
  bit_6: true
  bit_7: true
ind_0: false
ind_0_is_false: true
ind_0_is_zero: true
ind_0_ternary: 0
ind_4: true
ind_4_is_one: true
ind_4_is_true: true
ind_4_ternary: 1
ind_4_to_i: 1

ksdump --version
ksdump 0.7
kaitai-struct-compiler 0.10
kaitai-struct 0.10 (Kaitai Struct runtime library for Ruby)

Is there any chance to get it fixed in the near future?
Thank you for any feedback on this issue.

[generalmimon]

@lafrecciablu Thanks for reporting this. It seems to be a Ruby-specific issue: when given the fspec_one_byte_repeat_bits .ksy example that you provided in #1229 (comment), both KSC 0.10 and 0.11-SNAPSHOT at 91aeb53e produce the following Ruby code:

  class FspecByte < Kaitai::Struct::Struct
    # ...

    def _read
      @indicators = []
      (8).times { |i|
        @indicators << @_io.read_bits_int_be(1) != 0
      }
      self
    end

The problem is that due to Ruby operator precedence, this is interpreted as (@indicators << @_io.read_bits_int_be(1)) != 0, as can be clearly seen at https://peter-leonov.github.io/ruby-parser.js/:

[
  "send",
  [
    "send",
    ["ivar", "@indicators"],
    "<<",
    ["send", ["ivar", "@_io"], "read_bits_int_be", ["int", 1]]
  ],
  "!=",
  ["int", 0]
]

So clearly the compiler must add parentheses around @_io.read_bits_int_be(1) != 0 to make sure it's interpreted the way we want.

This doesn't happen in the Web IDE, for example, because the generated JavaScript code looks like this:

    FspecByte.prototype._read = function() {
      this.indicators = [];
      for (var i = 0; i < 8; i++) {
        this.indicators.push(this._io.readBitsIntBe(1) != 0);
      }
    }