Skip to content

Commit c3b118c

Browse files
jonnewcjsha
jonnew
and
cjsha
authored
Add python data loading script for neuropixels 1.0e headstage workflow (#135)
* Add spike interface python data loading script for np1, np2 headstage workflows - save files with suffic in workflows - also refactor some np hardware UIDs --------- Co-authored-by: cjsha <[email protected]>
1 parent feeb22b commit c3b118c

File tree

15 files changed

+495
-42
lines changed

15 files changed

+495
-42
lines changed

‎articles/hardware/np1e/load-data.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
---
2+
uid: np1e_load-data
3+
title: Load NeuropixelsV1e Headstage Data
4+
---
5+
6+
The following python script can be used to load and plot the data produced by the NeuropixelsV1e Headstage [example workflow](xref:np1e).
7+
8+
[!code-python[](../../../workflows/hardware/np1e/load-np1e.py)]
9+
10+
> [!NOTE]
11+
> To plot probeinterface data, [save the probe configuration file](xref:np1e_gui#save-probeinterface-file) into the same directory of your data.
12+
13+
> [!NOTE]
14+
> This script will attempt to load entire files into arrays. For long recordings, data will need to
15+
> be split into more manageable chunks by:
16+
> - Modifying this script to partially load files
17+
> - Modifying the workflow to cyclically create new files after a certain duration

‎articles/hardware/np1e/npv1e.md renamed to ‎articles/hardware/np1e/np1.md

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,13 @@
11
---
2-
uid: np1e_npv1e
2+
uid: np1e_np1
33
title: NeuropixelsV1e
44
hardware: NeuropixelsV1e Headstage
55
---
66

7-
The following excerpt from the NeuropixelsV1e Headstage [example workflow](xref:np1e_npv1e-headstage) demonstrates NeuropixelsV1e functionality by streaming and saving probe data.
7+
The following excerpt from the NeuropixelsV1e Headstage [example workflow](xref:np1e) demonstrates NeuropixelsV1e functionality by streaming and saving probe data.
88

99
::: workflow
10-
![/workflows/hardware/breakout/np1.bonsai workflow](../../../workflows/hardware/np1e/np1.bonsai)
10+
![/workflows/hardware/np1e/np1.bonsai workflow](../../../workflows/hardware/np1e/np1.bonsai)
1111
:::
1212

1313
The <xref:OpenEphys.Onix1.NeuropixelsV1eData> operator generates a sequence of <xref:OpenEphys.Onix1.NeuropixelsV1DataFrame>s using the following settings:

‎articles/hardware/np1e/overview.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,11 @@
11
---
2-
uid: np1e_npv1e-headstage
2+
uid: np1e
33
title: NeuropixelsV1e Headstage
44
---
55

66
These are the devices available on the NeuropixelsV1e Headstage:
77

8-
- One [NeuropixelsV1e](xref:np1e_npv1e):
8+
- One [NeuropixelsV1e](xref:np1e_np1):
99
- A single 1cm long shank probe with a 70 x 24 µm shank cross-section.
1010
- 960-electrode low-impedance TiN electrodes total.
1111
- 384 parallel, dual-band (AP, LFP), low-noise recording channels.

‎articles/hardware/np2e/load-data.md

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
---
2+
uid: np2e_load-data
3+
title: Load NeuropixelsV2e Headstage Data
4+
---
5+
6+
The following python script can be used to load and plot the data produced by the NeuropixelsV1e Headstage [example workflow](xref:np2e).
7+
8+
[!code-python[](../../../workflows/hardware/np2e/load-np2e.py)]
9+
10+
> [!NOTE]
11+
> To plot probeinterface data, [save the probe configuration file](xref:np2e_gui#save-probeinterface-file) into the same directory of your data.
12+
13+
> [!NOTE]
14+
> This script will attempt to load entire files into arrays. For long recordings, data will need to
15+
> be split into more manageable chunks by:
16+
> - Modifying this script to partially load files
17+
> - Modifying the workflow to cyclically create new files after a certain duration

‎articles/hardware/np2e/npv2e.md renamed to ‎articles/hardware/np2e/np2.md

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
---
2-
uid: np2e_npv2e
2+
uid: np2e_np2
33
title: NeuropixelsV2e
44
hardware: NeuropixelsV2e Headstage
55
---
@@ -10,7 +10,7 @@ hardware: NeuropixelsV2e Headstage
1010
The following excerpt from the NeuropixelsV2e Headstage [example workflow](xref:np2e) demonstrates Neuropixels 2.0 probe functionality by streaming data and saves Neuropixels 2.0 probe data. The second chain is disabled by default, assuming that only one probe is connected to the headstage. If two probes are connected, the second `NeuropixelsV2eData` chain can be enabled to stream data from both probes simultaneously. To enable, select all nodes in the disabled chain and press <kbd>Ctrl</kbd>+<kbd>Shift</kbd>+<kbd>D</kbd>, or click `Enable` right-clicking the selected nodes.
1111

1212
::: workflow
13-
![/workflows/hardware/breakout/np2.bonsai workflow](../../../workflows/hardware/np2e/np2.bonsai)
13+
![/workflows/hardware/np2e/np2.bonsai workflow](../../../workflows/hardware/np2e/np2.bonsai)
1414
:::
1515

1616
The <xref:OpenEphys.Onix1.NeuropixelsV2eData> operator generates a sequence of <xref:OpenEphys.Onix1.NeuropixelsV2eDataFrame>s. In the NeuropixelsV2e Headstage example workflow,

‎articles/hardware/np2e/overview.md

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -5,7 +5,7 @@ title: NeuropixelsV2e Headstage
55

66
These are the devices available on the NeuropixelsV2e Headstage:
77

8-
- Supports up to two [IMEC Neuropixels 2.0 probes](xref:np2e_npv2e):
8+
- Supports up to two [IMEC Neuropixels 2.0 probes](xref:np2e_np2):
99
- Either 1x or 4x silicon shanks with a 70 x 24 µm cross-section.
1010
- 1280 electrodes low-impedance TiN electrodes per shank (5120 total for quad-shank probes).
1111
- 384 parallel, full-band (AP, LFP), low-noise recording channels.

‎articles/toc.yml

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -42,18 +42,19 @@
4242
- name: Port Status
4343
href: hardware/np1e/port-status.md
4444
- name: Neuropixels 1.0 Data
45-
href: hardware/np1e/npv1e.md
45+
href: hardware/np1e/np1.md
4646
- name: Orientation Data & Commutation
4747
href: hardware/np1e/bno055.md
4848
- href: hardware/np1e/gui.md
49+
- href: hardware/np1e/load-data.md
4950
- href: hardware/np2e/overview.md
5051
items:
5152
- name: Configuration
5253
href: hardware/np2e/configuration.md
5354
- name: Port Status
5455
href: hardware/np2e/port-status.md
5556
- name: Neuropixels 2.0 Data
56-
href: hardware/np2e/npv2e.md
57+
href: hardware/np2e/np2.md
5758
- name: Orientation Data & Commutation
5859
href: hardware/np2e/bno055.md
5960
- href: hardware/np2e/gui.md

‎workflows/hardware/np1e/bno055.bonsai

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -9,15 +9,16 @@
99
<Expression xsi:type="Combinator">
1010
<Combinator xsi:type="onix1:PolledBno055Data">
1111
<onix1:DeviceName>NeuropixelsV1eHeadstage/PolledBno055</onix1:DeviceName>
12+
<onix1:PolledRegisters>All</onix1:PolledRegisters>
1213
</Combinator>
1314
</Expression>
1415
<Expression xsi:type="io:CsvWriter">
1516
<io:FileName>bno055_.csv</io:FileName>
1617
<io:Append>false</io:Append>
1718
<io:Overwrite>false</io:Overwrite>
18-
<io:Suffix>Timestamp</io:Suffix>
19+
<io:Suffix>FileCount</io:Suffix>
1920
<io:IncludeHeader>false</io:IncludeHeader>
20-
<io:Selector>it</io:Selector>
21+
<io:Selector>Clock,EulerAngle,Quaternion,Acceleration,Gravity,Temperature</io:Selector>
2122
</Expression>
2223
<Expression xsi:type="MemberSelector">
2324
<Selector>Quaternion</Selector>

‎workflows/hardware/np1e/configuration.bonsai

Lines changed: 76 additions & 6 deletions
Large diffs are not rendered by default.

‎workflows/hardware/np1e/load-np1e.py

Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
import os
2+
import numpy as np
3+
import matplotlib.pyplot as plt
4+
import matplotlib.patches as mpatches
5+
import spikeinterface.extractors as se
6+
import spikeinterface.widgets as sw
7+
import probeinterface
8+
import probeinterface.plotting
9+
10+
ap_gain = 500 # Change to the ap band gain used
11+
lfp_gain = 500 # Change to the lfp band gain used
12+
suffix = 0 # Change to match file names' suffix
13+
num_channels = 384 # Decrease channels to expedite plotting and inspect fewer traces
14+
# Change this to the directory of your data. In this example, data's in the same directory as this data loading Python script
15+
data_directory = os.path.dirname(os.path.realpath(__file__))
16+
mode = 'auto' # This uses colormap plot above 50 channels and line plot below 50 channels. Refer to the spikeinterface docs for more options
17+
# Change this to the name of your probeinterface file (should be in the same directory as the rest of your data for this example script to work as-is)
18+
probeinterface_filename = 'np1-config.json'
19+
20+
21+
plt.close('all')
22+
23+
#%% Metadata
24+
dt = {'names': ('time', 'acq_clk_hz', 'block_read_sz', 'block_write_sz'),
25+
'formats': ('datetime64[us]', 'u4', 'u4', 'u4')}
26+
meta = np.genfromtxt(os.path.join(data_directory, f'start-time_{suffix}.csv'), delimiter=',', dtype=dt, skip_header=1)
27+
print(f"Recording was started at {meta['time']} GMT")
28+
29+
#%% Neuropixels 1.0 probeinterface
30+
fig, ax = plt.subplots()
31+
np1_config = probeinterface.io.read_probeinterface(os.path.join(data_directory, 'np1-config.json'))
32+
contacts_colors = ['cyan' if device_channel_index > -1 else 'red' for device_channel_index in np1_config.probes[0].device_channel_indices]
33+
probeinterface.plotting.plot_probegroup(np1_config, ax=ax, contacts_colors=contacts_colors, contacts_kargs={'alpha' : 1, 'zorder' : 10}, show_channel_on_click=True)
34+
fig.set_size_inches(2, 9)
35+
enabled = mpatches.Patch(color='cyan', label='Enabled')
36+
disabled = mpatches.Patch(color='red', label='Disabled')
37+
fig.legend(handles=[enabled, disabled], loc='outside upper center')
38+
plt.tight_layout()
39+
40+
41+
#%% Neuropixels 1.0 Data
42+
bit_depth = 10
43+
fig, ax = plt.subplots(1,2)
44+
fig.suptitle('Neuropixels 1.0 Data')
45+
fig.set_size_inches(9, 9)
46+
plt.subplots_adjust(wspace=0.3)
47+
48+
ap_scalar = 1.2e6 / (2 ** bit_depth) / ap_gain
49+
ap_offset = (2 ** (bit_depth - 1)) * ap_scalar
50+
ap_recording = se.read_binary(os.path.join(data_directory, f"np1-spike_{suffix}.raw"),
51+
3e5,
52+
np.uint16,
53+
num_channels,
54+
gain_to_uV=ap_scalar,
55+
offset_to_uV=-ap_offset)
56+
ap_traces_plot = sw.plot_traces(ap_recording,
57+
backend='matplotlib',
58+
return_scaled=True,
59+
mode=mode,
60+
clim=(-ap_offset,ap_offset),
61+
ax=ax[0])
62+
ax[0].set_xlabel("time (sec)")
63+
ax[0].set_ylabel("AP (µV)")
64+
65+
lfp_scalar = 1.2e6 / (2 ** bit_depth) / lfp_gain
66+
lfp_offset = (2 ** (bit_depth - 1)) * lfp_scalar
67+
lfp_recording = se.read_binary(os.path.join(data_directory, f"np1-lfp_{suffix}.raw"),
68+
3e5/12,
69+
np.uint16,
70+
num_channels,
71+
gain_to_uV=lfp_scalar,
72+
offset_to_uV=-lfp_offset)
73+
lfp_traces_plot = sw.plot_traces(lfp_recording,
74+
backend='matplotlib',
75+
return_scaled=True,
76+
mode=mode,
77+
clim=(-lfp_offset,lfp_offset),
78+
ax=ax[1])
79+
ax[1].set_xlabel("time (sec)")
80+
ax[1].set_ylabel("LFP (µV)")
81+
82+
#%% Bno055
83+
dt = {'names': ('euler', 'quat', 'is_quat_id', 'accel', 'grav', 'temp', 'calibration', 'clock'),
84+
'formats': ('(1,3)f8', '(1,4)f8', '?', '(1,3)f8', '(1,3)f8', 'f8', '?', 'u8')}
85+
bno055 = np.genfromtxt(os.path.join(data_directory, f'bno055_{suffix}.csv'), delimiter=',', dtype=dt)
86+
87+
bno055_time = bno055['clock'] / meta['acq_clk_hz']
88+
89+
plt.figure()
90+
plt.subplot(231)
91+
plt.plot(bno055_time, bno055['euler'].squeeze())
92+
plt.xlabel("time (sec)")
93+
plt.ylabel("angle (deg.)")
94+
plt.ylim(-185, 365)
95+
plt.legend(['yaw', 'pitch', 'roll'])
96+
plt.title('Euler')
97+
98+
plt.subplot(232)
99+
plt.plot(bno055_time, bno055['quat'].squeeze())
100+
plt.xlabel("time (sec)")
101+
plt.ylim(-1.1, 1.1)
102+
plt.legend(['X', 'Y', 'Z', 'W'])
103+
plt.title('Quaternion')
104+
105+
plt.subplot(233)
106+
plt.plot(bno055_time, bno055['accel'].squeeze())
107+
plt.xlabel("time (sec)")
108+
plt.ylabel("accel. (m/s^2)")
109+
plt.legend(['X', 'Y', 'Z'])
110+
plt.title('Lin. Accel.')
111+
112+
plt.subplot(234)
113+
plt.plot(bno055_time, bno055['grav'].squeeze())
114+
plt.xlabel("time (sec)")
115+
plt.ylabel("accel. (m/s^2)")
116+
plt.legend(['X', 'Y', 'Z'])
117+
plt.title('Gravity Vec.')
118+
119+
plt.subplot(235)
120+
plt.plot(bno055_time, bno055['temp'].squeeze())
121+
plt.xlabel("time (sec)")
122+
plt.ylabel("temp. (°C)")
123+
plt.title('Headstage Temp.')
124+
125+
plt.tight_layout()
126+
127+
plt.show()

‎workflows/hardware/np1e/memory-monitor.bonsai

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
<?xml version="1.0" encoding="utf-8"?>
2+
<WorkflowBuilder Version="2.8.5"
3+
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
4+
xmlns:onix1="clr-namespace:OpenEphys.Onix1;assembly=OpenEphys.Onix1"
5+
xmlns:io="clr-namespace:Bonsai.IO;assembly=Bonsai.System"
6+
xmlns="https://bonsai-rx.org/2018/workflow">
7+
<Workflow>
8+
<Nodes>
9+
<Expression xsi:type="Combinator">
10+
<Combinator xsi:type="onix1:MemoryMonitorData">
11+
<onix1:DeviceName>BreakoutBoard/MemoryMonitor</onix1:DeviceName>
12+
</Combinator>
13+
</Expression>
14+
<Expression xsi:type="io:CsvWriter">
15+
<io:FileName>memory-use_.csv</io:FileName>
16+
<io:Append>false</io:Append>
17+
<io:Overwrite>false</io:Overwrite>
18+
<io:Suffix>FileCount</io:Suffix>
19+
<io:IncludeHeader>false</io:IncludeHeader>
20+
<io:Selector>Clock,BytesUsed,PercentUsed</io:Selector>
21+
</Expression>
22+
<Expression xsi:type="MemberSelector">
23+
<Selector>PercentUsed</Selector>
24+
</Expression>
25+
</Nodes>
26+
<Edges>
27+
<Edge From="0" To="1" Label="Source1" />
28+
<Edge From="1" To="2" Label="Source1" />
29+
</Edges>
30+
</Workflow>
31+
</WorkflowBuilder>

‎workflows/hardware/np1e/np1.bonsai

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -18,7 +18,7 @@
1818
<Expression xsi:type="Combinator">
1919
<Combinator xsi:type="dsp:MatrixWriter">
2020
<dsp:Path>np1-clock_.raw</dsp:Path>
21-
<dsp:Suffix>Timestamp</dsp:Suffix>
21+
<dsp:Suffix>FileCount</dsp:Suffix>
2222
<dsp:Overwrite>false</dsp:Overwrite>
2323
<dsp:Layout>ColumnMajor</dsp:Layout>
2424
</Combinator>
@@ -29,7 +29,7 @@
2929
<Expression xsi:type="Combinator">
3030
<Combinator xsi:type="dsp:MatrixWriter">
3131
<dsp:Path>np1-spike_.raw</dsp:Path>
32-
<dsp:Suffix>Timestamp</dsp:Suffix>
32+
<dsp:Suffix>FileCount</dsp:Suffix>
3333
<dsp:Overwrite>false</dsp:Overwrite>
3434
<dsp:Layout>ColumnMajor</dsp:Layout>
3535
</Combinator>
@@ -40,7 +40,7 @@
4040
<Expression xsi:type="Combinator">
4141
<Combinator xsi:type="dsp:MatrixWriter">
4242
<dsp:Path>np1-lfp_.raw</dsp:Path>
43-
<dsp:Suffix>Timestamp</dsp:Suffix>
43+
<dsp:Suffix>FileCount</dsp:Suffix>
4444
<dsp:Overwrite>false</dsp:Overwrite>
4545
<dsp:Layout>ColumnMajor</dsp:Layout>
4646
</Combinator>

0 commit comments

Comments
 (0)