Merge branch 'main' of https://github.com/mouseland/suite2p

Author: carsen-stringer

README.md

@@ -1,11 +1,11 @@
 # suite2p <img src="suite2p/logo/logo_unshaded.png" width="250" title="sweet two pea" alt="sweet two pea" align="right" vspace = "50">
 
-[![Documentation Status](https://readthedocs.org/projects/cellpose/badge/?version=latest)](https://cellpose.readthedocs.io/en/latest/?badge=latest)
-![tests](https://github.com/mouseland/cellpose/actions/workflows/test_and_deploy.yml/badge.svg)
+[![Documentation Status](https://readthedocs.org/projects/suite2p/badge/?version=latest)](https://suite2p.readthedocs.io/en/latest/?badge=latest)
+![tests](https://github.com/mouseland/suite2p/actions/workflows/test_and_deploy.yml/badge.svg)
 [![codecov](https://codecov.io/gh/MouseLand/suite2p/branch/main/graph/badge.svg?token=OJEC3mty85)](https://codecov.io/gh/MouseLand/suite2p)
 [![PyPI version](https://badge.fury.io/py/suite2p.svg)](https://badge.fury.io/py/suite2p)
-[![Downloads](https://pepy.tech/badge/suite2p)](https://pepy.tech/project/suite2p)
-[![Downloads](https://pepy.tech/badge/suite2p/month)](https://pepy.tech/project/suite2p)
+[![Downloads](https://static.pepy.tech/badge/suite2p)](https://pepy.tech/project/suite2p)
+[![Downloads](https://static.pepy.tech/badge/suite2p/month)](https://pepy.tech/project/suite2p)
 [![Python version](https://img.shields.io/pypi/pyversions/suite2p)](https://pypistats.org/packages/suite2p)
 [![Licence: GPL v3](https://img.shields.io/github/license/MouseLand/suite2p)](https://github.com/MouseLand/suite2p/blob/main/LICENSE)
 [![Contributors](https://img.shields.io/github/contributors-anon/MouseLand/suite2p)](https://github.com/MouseLand/suite2p/graphs/contributors)
@@ -27,16 +27,19 @@ suite2p includes the following modules:
 
 This code was written by Carsen Stringer and Marius Pachitariu.
 For support, please open an [issue](https://github.com/MouseLand/suite2p/issues).
+
 The reference paper is [here](https://www.biorxiv.org/content/early/2017/07/20/061507). The deconvolution algorithm is based on [this paper](https://journals.plos.org/ploscompbiol/article?id=10.1371/journal.pcbi.1005423), with settings based on [this paper](http://www.jneurosci.org/content/early/2018/08/06/JNEUROSCI.3339-17.2018).
 
-You can now run suite2p in google colab, no need to locally install (although we recommend doing so eventually): [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/MouseLand/suite2p/blob/main/jupyter/run_suite2p_colab_2021.ipynb). Note you do not have access to the GUI via google colab, but you can download the processed files and view them locally in the GUI.
+You can now run suite2p in google colab, no need to locally install (although we recommend doing so eventually): [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/MouseLand/suite2p/blob/main/jupyter/run_suite2p_colab_2023.ipynb). Note you do not have access to the GUI via google colab, but you can download the processed files and view them locally in the GUI.
 
 See this **twitter [thread](https://twitter.com/marius10p/status/1032804776633880583)** for GUI demonstrations.
 
 The matlab version is available [here](https://github.com/cortex-lab/Suite2P). Note that the algorithm is older and will not work as well on non-circular ROIs.
 
 Lectures on how suite2p works are available [here](https://youtu.be/HpL5XNtC5wU?list=PLutb8FMs2QdNqL4h4NrNhSHgLGk4sXarb).
 
+**Note on pull requests**: we accept very few pull requests due to the maintenance efforts required to support new code, and we do not accept pull requests from automated code checkers. If you wrote code that interfaces/changes suite2p behavior, a common approach would be to keep that in a fork and pull periodically from the main branch to make sure you have the latest updates.
+
 ### CITATION
 
 If you use this package in your research, please cite the [paper](https://www.biorxiv.org/content/early/2017/07/20/061507):
@@ -89,7 +92,7 @@ If you want to download and edit the code, and use that version,
 ### Installation for developers
 
 1. Clone the repository and `cd suite2p` in an anaconda prompt / command prompt with `conda` for **python 3** in the path
-2. Run `conda env create --name suite2p`
+2. Run `conda create --name suite2p python=3.9`
 3. To activate this new environment, run `conda activate suite2p` (you will have to activate every time you want to run suite2p)
 4. Install the local version of suite2p into this environment in develop mode with the command `pip install -e .[all]`
 5. Run tests: `python setup.py test` or `pytest -vs`, this will automatically download the test data into your `suite2p` folder. The test data is split into two parts: test inputs and expected test outputs which will be downloaded in `data/test_inputs` and `data/test_outputs` respectively. The .zip files for these two parts can be downloaded from these links: [test_inputs](https://www.suite2p.org/static/test_data/test_inputs.zip) and [test_outputs](https://www.suite2p.org/static/test_data/test_outputs.zip).

docs/gui.rst

@@ -65,9 +65,9 @@ Correlations with 1D var
 You can load an external stimulus or behavioral trace (1D) using "File -
 Load behavior or stim trace (1D only)". The GUI expects a \*.npy file
 that is the same length as the data in time (F.shape[1] from "F.npy").
-You can then look at the correlation of each cell with this trace. And
-it will be plotted along with the cell traces if you select multiple
-cells or in the "Visualize" menu.
+The length should match the number of frames in "F.npy". You can then look at the correlation of each 
+cell with this trace. And it will be plotted along with the cell traces 
+if you select multiple cells or in the "Visualize" menu.
 
 .. _rastermap--custom:
 
@@ -198,6 +198,18 @@ and it will ask you to specify a file location for the new classifier.
 Then you can load the classifier that you built into the GUI, or you can
 save it as your default classifier.
 
+Applying a custom classifier
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Go to the "Classifier" menu and click "Load / from file". A window will 
+pop up and allow you to select a classfier from a file that you have 
+already built. Upon loading, the GUI will recolor ROIs according to their
+iscell probability according to the new classifier, but they will retain 
+their previous category and the ``iscell.npy`` file will not be updated. 
+If you want to apply this new classifier to the ROIs category and update 
+the ``iscell.npy`` file, then click the classifier probability box, enter
+your threshold, and press enter. 
+
 Visualizing activity
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
@@ -316,4 +328,4 @@ non-rigid registered. The metrics suggest that non-rigid registration
 should also be performed on this recording.
 
 .. image:: _static/reg_metrics.png
-   :width: 600
+   :width: 600

docs/inputs.rst

@@ -93,11 +93,22 @@ imageJ and suite2p can recognize (see matlab tiff writing
 Bruker
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+**Single Page Tifs**:
 Using Bruker Prairie View system, .RAW files are batch converted to single .ome.tifs.
 Now, you can load the resulting multiple tif files (i.e. one per frame per channel) to suite2p to be converted to binary.
 This looks for files containing 'Ch1', and will assume all additional files are 'Ch2'.
 Select "input_format" as "bruker" in the drop down menu in the GUI or set ``ops['input_format'] = "bruker"``.
 
+**Multi Page Tifs**:
+To speed up the processing of input from bruker scopes, we recommend you save your .RAW files as multipage tifs.  This can be done using the Bruker Prairie View system. 
+
+In the PrairieView software, set your preferences to convert your raw files to multipage TIFFs. 
+
+* Preferences > Save Multipage TIFFs
+* Preferences > Automatically Convert Raw Files > After Acquisition
+
+This will cause the GUI to be unresponsive for some time after each acquisition. This should work for both single-channel and 2-channel recordings.
+
 Mesoscope tiffs
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
@@ -118,10 +129,10 @@ you're using this and having trouble because it's not straightforward.
 Thorlabs raw files
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Christoph Schmidt-Hieber (@neurodroid) has written `haussmeister`_ which
-can load and convert ThorLabs \*.raw files to suite2p binary files!
-suite2p will automatically use this if you have pip installed it
-(``pip install haussmeister``).
+Suite2p has been upgraded with internal support for Thorlabs raw files (Yael Prilutski).
+Specify "raw" for "input_format".
+Designed to work with one or several planes and/or channels.
+
 
 .. _hdf5-files-and-sbx:
 

setup.py

@@ -25,7 +25,9 @@
     "paramiko",
     "nd2",
     "sbxreader",
-    "h5py"
+    "h5py",
+    "opencv-python-headless",
+    "xmltodict"
 ]
 
 nwb_deps = [
@@ -62,7 +64,7 @@
 except:
     pass
 
-all_deps = gui_deps + nwb_deps + test_deps + io_deps
+all_deps = gui_deps + nwb_deps + test_deps + io_deps 
 
 try:
     import torch
@@ -102,6 +104,7 @@
       ],
       "gui": gui_deps,
       "nwb": nwb_deps,
+      "dcimg": dcimg_deps,
       "io": io_deps,
       "tests": test_deps,
       "all": all_deps,

suite2p/detection/anatomical.py

@@ -105,7 +105,7 @@ def roi_detect(mproj, diameter=None, cellprob_threshold=0.0, flow_threshold=1.5,
         model = CellposeModel(model_type=pretrained_model)
     else:
         model = CellposeModel(pretrained_model=pretrained_model)
-    masks = model.eval(mproj, net_avg=True, channels=[0, 0], diameter=diameter,
+    masks = model.eval(mproj, channels=[0, 0], diameter=diameter,
                        cellprob_threshold=cellprob_threshold,
                        flow_threshold=flow_threshold)[0]
     shape = masks.shape

suite2p/detection/detect.py

@@ -125,6 +125,8 @@ def detection_wrapper(f_reg, mov=None, yrange=None, xrange=None, ops=default_ops
     n_frames, Ly, Lx = f_reg.shape
     yrange = ops.get("yrange", [0, Ly]) if yrange is None else yrange
     xrange = ops.get("xrange", [0, Lx]) if xrange is None else xrange
+    ops["yrange"] = yrange
+    ops["xrange"] = xrange
 
     if mov is None:
         bin_size = int(
@@ -138,6 +140,10 @@ def detection_wrapper(f_reg, mov=None, yrange=None, xrange=None, ops=default_ops
         elif mov.shape[2] != xrange[-1] - xrange[0]:
             raise ValueError("mov.shape[2] is not same size as xrange")
 
+    if "meanImg" not in ops:
+        ops["meanImg"] = mov.mean(axis=0)
+        ops["max_proj"] = mov.max(axis=0)
+
     if ops.get("inverted_activity", False):
         mov -= mov.min()
         mov *= -1

suite2p/gui/gui2p.py

@@ -208,7 +208,7 @@ def make_buttons(self):
         return b0
 
     def roi_text(self, state):
-        if state == QtCore.Qt.Checked:
+        if QtCore.Qt.CheckState(state) == QtCore.Qt.Checked:
             for n in range(len(self.roi_text_labels)):
                 if self.iscell[n] == 1:
                     self.p1.addItem(self.roi_text_labels[n])
@@ -232,7 +232,7 @@ def roi_text(self, state):
 
     def zoom_cell(self, state):
         if self.loaded:
-            if state == QtCore.Qt.Checked:
+            if QtCore.Qt.CheckState(state) == QtCore.Qt.Checked:
                 self.zoomtocell = True
             else:
                 self.zoomtocell = False
@@ -554,7 +554,7 @@ def number_chosen(self):
             self.show()
 
     def ROIs_on(self, state):
-        if state == QtCore.Qt.Checked:
+        if QtCore.Qt.CheckState(state) == QtCore.Qt.Checked:
             self.ops_plot["ROIs_on"] = True
             self.p1.addItem(self.color1)
             self.p2.addItem(self.color2)

suite2p/gui/io.py

@@ -4,6 +4,7 @@
 import os, time
 import numpy as np
 import scipy.io
+from scipy.ndimage import gaussian_filter1d
 from qtpy import QtGui
 from qtpy.QtWidgets import QFileDialog, QMessageBox
 

suite2p/gui/merge.py

@@ -316,7 +316,7 @@ def set_merge_list(self, parent, goodind):
         self.n = 0
         if len(self.merge_list) > 0:
             self.suggestMerge.setEnabled(True)
-            self.unmerged = np.ones(len(self.merge_list), np.bool)
+            self.unmerged = np.ones(len(self.merge_list), bool)
             self.suggest_merge(parent)
 
     def suggest_merge(self, parent):

suite2p/gui/reggui.py

@@ -13,6 +13,7 @@
 from scipy.ndimage import gaussian_filter1d
 from natsort import natsorted
 from tifffile import imread
+import json
 
 from . import masks, views, graphics, traces, classgui, utils
 from .. import registration
@@ -354,8 +355,7 @@ def plot_trace(self):
         self.p2.setLimits(xMin=0, xMax=self.nframes)
 
     def open(self):
-        filename = QFileDialog.getOpenFileName(self, "Open single-plane ops.npy file",
-                                               filter="ops*.npy")
+        filename = QFileDialog.getOpenFileName(self, "Open single-plane ops.npy file or single-plane ops.json file")
         # load ops in same folder
         if filename:
             print(filename[0])
@@ -430,7 +430,19 @@ def openCombined(self, save_folder):
 
     def openFile(self, filename, fromgui):
         try:
-            ops = np.load(filename, allow_pickle=True).item()
+            ext = os.path.splitext(filename)[1]
+            if ext == ".npy":
+                ops = np.load(filename, allow_pickle=True).item()
+                dirname = os.path.dirname(filename)
+            elif ext == ".json":
+                with open(filename, "r") as f:
+                    ops = json.load(f)
+                ops["Ly"] = ops["Lys"] if isinstance(ops["Lys"], int) else ops["Lys"][0]
+                ops["Lx"] = ops["Lxs"] if isinstance(ops["Lxs"], int) else ops["Lxs"][0]
+                dirname = os.path.join(os.path.dirname(filename), "suite2p/plane0/")
+                ops["reg_file"] = os.path.join(dirname, "data.bin")
+                nbytesread = np.int64(2 * ops["Ly"] * ops["Lx"])
+                ops["nframes"] = os.path.getsize(ops["reg_file"]) // nbytesread
             self.LY = ops["Ly"]
             self.LX = ops["Lx"]
             self.Ly = [ops["Ly"]]
@@ -442,7 +454,7 @@ def openFile(self, filename, fromgui):
                 self.reg_loc = [ops["reg_file"]]
             else:
                 self.reg_loc = [
-                    os.path.abspath(os.path.join(os.path.dirname(filename), "data.bin"))
+                    os.path.abspath(os.path.join(dirname, "data.bin"))
                 ]
             self.reg_file = [open(self.reg_loc[-1], "rb")]
             self.wraw = False

suite2p/gui/rungui.py

@@ -276,7 +276,7 @@ def create_buttons(self):
         self.inputformat = QComboBox()
         [
             self.inputformat.addItem(f)
-            for f in ["tif", "bruker", "sbx", "h5", "mesoscan", "haus", "nd2"]
+            for f in ["tif", "binary", "bruker", "sbx", "h5", "movie", "nd2", "mesoscan", "raw", "dcimg"]
         ]
         self.inputformat.currentTextChanged.connect(self.parse_inputformat)
         self.layout.addWidget(self.inputformat, 2, 0, 1, 1)
@@ -438,7 +438,7 @@ def run_S2P(self):
         # pre-download model
         pretrained_model_string = self.ops.get("pretrained_model", "cyto")
         pretrained_model_string = pretrained_model_string if pretrained_model_string is not None else "cyto"
-        pretrained_model_path = model_path(pretrained_model_string, 0, True)
+        pretrained_model_path = model_path(pretrained_model_string, 0)
         self.finish = True
         self.error = False
         ops_file = os.path.join(self.ops_path, "ops.npy")

suite2p/io/__init__.py

@@ -2,10 +2,13 @@
 Copyright © 2023 Howard Hughes Medical Institute, Authored by Carsen Stringer and Marius Pachitariu.
 """
 from .h5 import h5py_to_binary
+from .raw import raw_to_binary
 from .nwb import save_nwb, read_nwb, nwb_to_binary
 from .save import combined, compute_dydx, save_mat
 from .sbx import sbx_to_binary
+from .movie import movie_to_binary
 from .tiff import mesoscan_to_binary, ome_to_binary, tiff_to_binary, generate_tiff_filename, save_tiff
 from .nd2 import nd2_to_binary
+from .dcam import dcimg_to_binary
 from .binary import BinaryFile, BinaryFileCombined
 from .server import send_jobs