Skip to content

Conversation

h-mayorquin
Copy link
Contributor

@h-mayorquin h-mayorquin commented Oct 3, 2025

This will fix #1778 and fix #1678.

NeuralynxRawIO on main is incorrectly grouping channels with fundamentally different acquisition configurations into the same stream. Eye-tracking channels and electrophysiology channels recorded at the same sampling rate would be merged together, despite having vastly different voltage ranges (100mV vs 1mV) and filter settings. This incorrect grouping has led to user errors and confusion. Additionally, stream names like "Stream (rate,#packet,t0): (32000.0, 300, 6382425039112)" are difficult to parse, type and use programmatically.

I've refactored stream identification to be based on acquisition hardware and signal processing parameters: sampling rate, input voltage range, gain, input inversion, and DSP filter configuration (low-cut/high-cut enabled status, frequencies, filter types, taps, delays). Examples:

  • Stream (rate,#packet,t0): (32000.0, 300, 6382425039112)stream0_32000Hz_100mVRange_DSPFilter0
  • Stream (rate,#packet,t0): (32000.0, 100, 6382425039112)stream1_32000Hz_1mVRange_DSPFilter1
  • Stream (rate,#packet,t0): (2000.0, 450, 6382425039112)stream2_2000Hz_100mVRange_DSPFilter2

These parameters define how signals are acquired and processed, sampling rate determines temporal resolution, voltage range and gain control amplitude scaling, and DSP filters alter frequency content. Channels with different values for any of these parameters represent fundamentally different signal types and cannot be meaningfully grouped together. Temporal properties (t_start, n_packets) have been removed from stream identification since they describe when/how much data was recorded, not the acquisition configuration itself. I display input voltage range in the stream name rather than gain because the range directly indicates the expected signal amplitude (e.g., 100mV for eye-tracking vs 1mV for ephys), which is more intuitive for users than a gain value that requires additional context to interpret. I am open to other suggestions in this front.

Since DSP filter configurations can be complex (combining multiple parameters like DSPLowCutFilterEnabled, DspLowCutFrequency, DSPHighCutFilterEnabled, DspHighCutFrequency, DspFilterType, DspDelayCompensation, etc.), I assign each unique filter configuration an enumerated ID and display it as DSPFilter{id} in the stream name. I choose to enumerate the filters because while they fully specify the acquisition stream and should be included in stream identification, displaying all filter parameters would make stream names too verbose (see example below).The detailed filter parameters are stored internally in _dsp_filter_configurations (kept private for now as an experimental feature, open to discussion if I want to keep it like that, expose via public API or through the annotations mechanism). Example structure:

_dsp_filter_configurations = {
    0: {
        'DSPLowCutFilterEnabled': False,
        'DspLowCutFrequency': 0,
        'DSPLowCutNumTaps': 0,
        'DSPLowCutFilterType': 'None',
        'DSPHighCutFilterEnabled': True,
        'DspHighCutFrequency': 8000,
        'DSPHighCutNumTaps': 256,
        'DSPHighCutFilterType': 'FIR',
        'DspDelayCompensation': 'Enabled',
        'DspFilterDelay_µs': 1968
    },
    1: {
        'DSPLowCutFilterEnabled': True,
        'DspLowCutFrequency': 0.1,
        'DSPLowCutNumTaps': 256,
        'DSPLowCutFilterType': 'FIR',
        'DSPHighCutFilterEnabled': True,
        'DspHighCutFrequency': 8000,
        'DSPHighCutNumTaps': 256,
        'DSPHighCutFilterType': 'FIR',
        'DspDelayCompensation': 'Enabled',
        'DspFilterDelay_µs': 1968
    }
}

We also added type normalization in NlxHeader to ensure header values are proper Python types (booleans, integers, floats) rather than strings.

See the following table with our test suite examples to see how the stream naming will be affected:

Dataset Old Stream Name New Stream Name Description
two_streams_different_header_encoding Stream (rate,#packet,t0): (32000.0, N, T) stream0_32000Hz_100mVRange_DSPFilter0 2 ch: CSC145, CSC146 (eye-tracking)
two_streams_different_header_encoding Stream (rate,#packet,t0): (32000.0, N, T) stream1_32000Hz_1mVRange_DSPFilter1 1 ch: CSC76 (ephys)
Cheetah_v5.5.1 Stream (rate,#packet,t0): (32000.0, N, T) stream0_32000Hz_1mVRange_DSPFilter0 2 ch: Tet3a, Tet3b
Cheetah_v5.6.3 Stream (rate,#packet,t0): (2000.0, N, T) stream0_2000Hz_1mVRange_DSPFilter0 2 ch: CSC1, CSC2
Cheetah_v6.4.1dev Stream (rate,#packet,t0): (32000.0, N, T) stream0_32000Hz_2mVRange_DSPFilter0 1 ch: CSC1
Cheetah_v6.4.1dev Stream (rate,#packet,t0): (2666.0, N, T) stream1_2666Hz_5mVRange_DSPFilter1 1 ch: LFP4
Cheetah_v6.4.1dev Stream (rate,#packet,t0): (2000.0, N, T) stream2_2000Hz_100mVRange_DSPFilter2 1 ch: WE1
Cheetah_v4.0.2 Stream (rate,#packet,t0): (27789.0, N, T) stream0_27789Hz_0mVRange_DSPFilter0 1 ch: unknown (no filtering)
Cheetah_v5.4.0 Stream (rate,#packet,t0): (1017.0, N, T) stream0_1017Hz_0mVRange_DSPFilter0 1 ch: CSC5
Cheetah_v5.7.4 Stream (rate,#packet,t0): (32000.0, N, T) stream0_32000Hz_1mVRange_DSPFilter0 5 ch: CSC1, CSC2, CSC3, CSC4, CSC5
Cheetah_v6.3.2 Stream (rate,#packet,t0): (32000.0, N, T) stream0_32000Hz_10mVRange_DSPFilter0 1 ch: CSC1
BML Stream (rate,#packet,t0): (24000.0, N, T) stream0_24000Hz_0mVRange_DSPFilter0 1 ch: unknown
Cheetah_v1.1.0 Stream (rate,#packet,t0): (32768.0, N, T) stream0_32768Hz_1mVRange_DSPFilter0 1 ch: RMH3

@h-mayorquin h-mayorquin self-assigned this Oct 3, 2025
h-mayorquin and others added 2 commits October 3, 2025 15:46
- DSPFilter{id} more accurately represents DSP filter configurations
- Clearer naming: explicitly mentions Digital Signal Processing
- Works correctly even when filtering is disabled (DSPFilter0 = configuration 0)
- Updated tests to reflect new naming convention

The stream name format is now: stream{id}_{Hz}Hz_{mV}mVRange_DSPFilter{id}

Examples:
- stream0_32000Hz_100mVRange_DSPFilter0 (eye-tracking, low-cut disabled)
- stream1_32000Hz_1mVRange_DSPFilter1 (ephys, low-cut enabled)

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@h-mayorquin h-mayorquin marked this pull request as ready for review October 3, 2025 22:41
@h-mayorquin h-mayorquin changed the title Separate segments on neuralynx and fix streams Separate segments on neuralynx and improve streams Oct 3, 2025
@h-mayorquin h-mayorquin changed the title Separate segments on neuralynx and improve streams Separate signals on neuralynx and improve streams names Oct 3, 2025
@zm711
Copy link
Contributor

zm711 commented Oct 4, 2025

Not sure if I will have full time to review before the version deadline. Is this urgent to get into the next version?

@h-mayorquin
Copy link
Contributor Author

It would be great to have this but I understand you are busy, man. No problem, thanks for reviewing it.

@samuelgarcia
Copy link
Contributor

Thanks for this. The reading is very clear.
I hope that no analysis pipeline use to be based on teh old stream_id schema.
I guess no.

@h-mayorquin
Copy link
Contributor Author

Thanks for the review, @samuelgarcia.

@h-mayorquin h-mayorquin merged commit 58dac7d into NeuralEnsemble:master Oct 7, 2025
3 checks passed
@h-mayorquin h-mayorquin deleted the fix_neuralynx_streams branch October 7, 2025 15:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] NeuralynxRawIO doesn't differentiate signals per stream The stream names of neuralynx are hard to use and affected by numpy > 2.1 changes

3 participants