time-interleaved files with same monoBN causes earlier raw data records to be ignored

[jbrzusto]

reified from MotusDev/Motus-TO-DO#434
Somewhat like #320 and #407.
In this case, there are detections in files from the original boot session 3, but because this is
a beaglebone white SG that was redeployed with a fresh SD card, and which had a bug whereby
boot numbers did not increase, there are several distinct boot sessions 3.
And unfortunately, there are files from later boot session 3 which have earlier pre-GPS timestamps
than some such files from an earlier boot session 3.
These later files are read early and bump the tag finder's clock forward before any of the
post-GPS timestamped files from the truly earlier boot session 3 can be processed. When
the latter are seen, their records are ignored because they contain time reversals.

This whole situation needs a rethink, as further elaborated in the issues linked above.

[jbrzusto]

This problem requires a dive into the deep end of sensorgnome / motus
design and implementation.

Here are notes that sketch out enough (hopefully) background to
guide a solution.

Data Flow

• a sensorgnome (SG) writes pulse detection data to a sequence of files

• an SG begins a new file every hour, or every megabyte of uncompressed
  data, whichever comes first; compressed and uncompressed files are written
  in tandem, with the uncompressed file deleted upon switching to a new file

• filenames include the SG serial number, timestamp, and boot session count
  (the latter is supposed to increase by one each time the SG reboots, but
  this isn't always the case)

• when users download files from an SG, they might get a partial copy of
  the last file (i.e. the file transfer process is not sync'd with file writing)

• generally, batches of files from an SG reach the motus server in inreasing
  temporal order, but not always (sometimes, files are located later, as some
  SGs have more than one onboard storage location, which users are not always
  aware of; or apparently corrupt SD cards are later scanned for data)

• pulses from data files must be run against a full database of active
  tags and their pulse patterns in order to assemble them into tag
  detections; a pulse is deemed to belong to at most one tag

• the tag database exists only on the motus server

• the interpretation of an individual pulse depends on context:

  • what pulses are nearby in time
  • what tags are known to be active at the time

• the tag finder (find_tags_motus) uses a "greedy" approach to
  extract tag detections from pulse data in a single pass. ("greedy" means
  that the first confirmed tag detection sequence that is compatible
  with a pulse gets to claim it).

• it's not feasible to re-run the tag finder on the entire pulse dataset for an
  SG every time we receive new data from it; this is especially true for networked
  receivers, from which we sync data hourly: the cumulative time spent processing data
  from each receiver would grow quadratically over time if we reprocessed from the
  beginning with each new batch of files.

• instead, we split the sequence of files from an SG into time periods, and when new
  data arrive from an SG, we only re-run those time periods for which there are new files.

• the time periods we chose are "boot sessions" (i.e. the maximal
  period of time during which a receiver ran without a reboot).

Here are the different ways the tag finder can be called to process some files:

1. old files: all files from a boot session are re-run in temporal sequence.

2. new files in a new boot session: when new files arrive, they are grouped by boot
   session, and files in each are processed in a single run of the tag finder (i.e. one run
   per boot session)

3. new files in an existing boot session: as an optimization, the tag finder always saves
   its internal state at the end of a run, so that new files for an existing boot session can
   be processed incrementally. This is how we avoid quadratic growth in processing time.

So a single run of the tag finder handles files from a single boot session (and not necessarily
all of those files). This single run produces output called a batch, which consists of
individual tag detections (hits) grouped into runs (which are on the same antenna).

The problem: boot sessions aren't monotonic

The decision to use boot sessions to organized data was made when almost all SG data were
coming from beaglebone-black (BBBK) sensorgnomes, which have internal flash memory where we can
store the boot count. This works, but:

• beaglebone-white (BBW) sensorgnomes (the original model, of which there are still maybe a dozen
  gathering data) and raspberry-pi sensorgnomes (most new SGs in the past couple of years) do
  not have this internal persistent storage, and as users run through different SD cards
  in the same unit, boot counts get reset or mixed up between receivers

• there was a bug in incrementing the boot count (I know; pathetic; how do you fail to
  implement ++x?) in at least one version of SG software, even on BBBK SGs.

• some users appear to have customized their SG's software in ways that mess with the boot count

So overall, the fact that N > M does not necessarily mean that a file (labelled as being) from
boot session 'N' was really written later than a file from boot session 'M'

The consequences of non-monotonic boot sessions

• the first few files recorded by an SG after it boots often have incorrect timestamps: the
  SG boots thinking it is the year 2000, but real SG timestamps only begin in 2010 or later.
  Eventually, the GPS sets the system clock, and a correct timestamp is written, so the tagfinder
  uses this to back-correct those pre-2010 timestamps.

• so if the system boots at different times but with the same boot number, there will be multiple
  files labelled with pre-GPS timestamps and the same boot numbers. One of these files eventually
  has a valid timestamp, and the tag finder will use that to back-correct the preceding timestamps.

The Catch

• the tag finder isn't very smart about dealing with non-monotonic
  timestamps in pulse data. If it sees consecutive records where the
  clock appears to jump backward more than a few seconds (to allow for
  USB timing lag when reading from multiple radios on a single hub), it
  ignores the later records (with earlier timestamps). So when running
  files in the same nominal boot session which were written at different
  real times, a later post-GPS timestamp can cause huge amounts of data
  to be skipped in subsequent processing.

Possible ways forward

• calculate monotonic boot numbers for each receiver; there is some code in the motusServer
  R package that does this, but hasn't been integrated into normal file processing

• re-organize file processing around some other marker. e.g. every two-week period

  • this would be a good optimization for the frequently-required re-runs of data; when
    new or changed tag registrations need to be taken into account, we would only go
    back to those two-week periods affected by the change, and re-run them. (each period
    would save state, so we'd be doing a resume).

These aren't necessarily mutually exclusive.

[leberrigan]

Thanks for laying this out clearly. Do you have any further thoughts on moving forward? Should I assign this issue to somebody?

[jbrzusto]

Sorry, way behind on stuff. If someone else wants to take it on, great. It is a substantial chunk of
work, so best to coordinate efforts on it to avoid duplication.

[joeybernard]

I should be diving into this soon. Just dealing with a few other items first.