Issues with scan trigger timing

[davidt0x]

Issues with scan image triggering starting too early or not starting at all. Sometimes it triggers on the second playlist item but not the first.

[davidt0x]

This issue sounds to me like race condition having to do with different startup times for different threads. FlyVR spawns a lot of threads a processes and some of them seem to go about processing things before others are totally initialized. This can lead to tricky bugs when they depend on each other's state. I will need to investigate the scan image triggering code and see if we can create a reproducible example.

[nzjrs]

This issue sounds to me like race condition having to do with different startup times for different threads. FlyVR spawns a lot of threads a processes and some of them seem to go about processing things before others are totally initialized. This can lead to tricky bugs when they depend on each other's state. I will need to investigate the scan image triggering code and see if we can create a reproducible example.

All backend processes should take the opportunity to pre-roll or prepare any buffers before waiting for the start signal over IPC if their start needs to be coordinated.

• start signal:

  fly-vr/flyvr/main.py

  Line 156 in 5dac72a

  flyvr_shared_state.signal_start()

• example setup:

  fly-vr/flyvr/hwio/phidget.py

  Line 32 in 5dac72a

  if tp_enable:

• example of backend which does wait for go signal:

  fly-vr/flyvr/audio/sound_server.py

  Line 306 in 5dac72a

  if not self.flyvr_shared_state.wait_for_start():

• scan trigger doesn't nee to implicitly wait for backends to be ready (although it could spin like the audio backend) because it does things in response to other backends playing things, so the logic is those other backends will be waiting for the start signal, and only after then will they play items which will then be picked up by the loop https://github.com/murthylab/fly-vr/blob/master/flyvr/hwio/phidget.py#L153

it would be harmless to make the daqio backend also spin until start signal but there shouldn't be any other backends signaling they are playing items until they have got the start signal.

(main process waits for all other backends here

fly-vr/flyvr/main.py

Line 146 in 5dac72a

if flyvr_shared_state.wait_for_backends(*backend_wait, timeout=60):

)

[nzjrs]

(I never saw any races at the start, place to look would be to check that main is waiting for all backends and that they only play the first item after they have signaled their readyness and then waited).

aka main does

start audio, start video, etc
wait for ready from above
signal start for all

and backends should do

init
signal ready
wait for start
loop (playlist items)

[mwangq]

I think this is related to issue #18 but it's more specifically about timing so I'll post it here. now that I have self.stop() working (thanks again, @nzjrs!) I would like to, for example, present stim A for 5 s, stim B for 5 s, A for 5 s, B for 5 s, and then stop the experiment.

the experiment.py's process_state has a stim block and a stop block. what I notice is that I'm inside the stim block, and the timer is already running, before the flyvr shared state says "signaling start". the config file delay is 5, and the first stim plays for a split second before moving on.

flyvr.main                              : INFO     delaying startup 5s
       inside the stim block: dt is 5.007123231887817, numStimNow is 0, aStim was sin200hz
flyvr.main                              : INFO     sending start signal
flyvr.common.SharedState(in='main')     : INFO     signaling start

if I remove the delay in the config file, then the stimulus plays more normally.

@nzjrs maybe I'm misinterpreting the delay field in the config file? what is that supposed to be used for? for now, setting it to 0 mostly solves this problem. I thought that the delay field would delay the experiment start time (i.e. stimulus presentation/stimulus-triggered scan image/various input into the daq). setting the delay to 10, means that the first 10 sec of stim are not played out thru the speakers....

so, this could actually be related to why scan triggering (issue described above) seems to trigger on a second playlist item rather than the first (if the first item's "go" signal occurred during the "delay")...?

[nzjrs]

the experiment.py's process_state has a stim block and a stop block. what I notice is that I'm inside the stim block, and the timer is already running, before the flyvr shared state says "signaling start". the config file delay is 5, and the first stim plays for a split second before moving on.

flyvr.main                              : INFO     delaying startup 5s
       inside the stim block: dt is 5.007123231887817, numStimNow is 0, aStim was sin200hz
flyvr.main                              : INFO     sending start signal
flyvr.common.SharedState(in='main')     : INFO     signaling start

if I remove the delay in the config file, then the stimulus plays more normally.

I'm not sure why I added delay - I think it was an old feature from FlyVR1 that I just kept. In simiar assays some long delay at the start to allow (for example) people to leave the room etc is a common feature.

As you note, it delays the start signal of the experment main process but does not stop the loop in the experiment being run. You should check Experiment.is_started() e.g. self.is_started() if you want to wait for the delay

https://github.com/murthylab/fly-vr/blob/master/flyvr/control/experiment.py#L147

there is an example of this here

https://github.com/murthylab/fly-vr/blob/master/experiments/cl_random_audio_daq.py#L48

you should feel free to make your Experiment() class as complicated as possible and do any relative-time accounting in process_state() as you see fit.

@nzjrs maybe I'm misinterpreting the delay field in the config file? what is that supposed to be used for? for now, setting it to 0 mostly solves this problem. I thought that the delay field would delay the experiment start time (i.e. stimulus presentation/stimulus-triggered scan image/various input into the daq). setting the delay to 10, means that the first 10 sec of stim are not played out thru the speakers....

so, this could actually be related to why scan triggering (issue described above) seems to trigger on a second playlist item rather than the first (if the first item's "go" signal occurred during the "delay")...?

could be. if you send a command to play something while the rest of the backends are not yet ready (or waiting in the delay phase for the start signal from the main flyvr.exe process) then the first stimulus would be 'missing'.