dev/v0.4.3 merge to master (#231)

Sylvain Chevallier · web-flow · commit a7f66ea57fd6 · 2021-09-03T16:10:28.000+02:00
* Avoid leaking information between sessions (#222) * clone pipelines in CV * PR should be made against develop branch * add WithinSubject update * add test for devel branch * Add coverage (#224) * add coverage * correct test * update coverage * investigate coverage issue * change coverage dir * change coverage file * correct error in set_download_dir (#225) * correct error in set_download_dir * add whats new * update readme with BCI definition (#228) * Ensure consistent channel order in paradigm (#229) * correct channel order, add ch_names to FakeDataset * correct ref to PR in whatsnew * prepare update to 0.4.3 (#230) * update download badge * add whats new in the doc menu * correct docstring * update to moabb 0.4.3 * correct doc generation (#232) * correct doc generation * build doc on push
diff --git a/.coveragerc b/.coveragerc
@@ -0,0 +1,14 @@
+[run]
+branch = True
+source = moabb
+include = */moabb/*
+omit =
+    */docs/*
+    */pipelines/*
+
+[report]
+exclude_lines =
+    pragma: no cover
+    if __name__ == .__main__.:
+    @abstractmethod
+    @abstractclassmethod
diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml
@@ -2,7 +2,7 @@ name: Docs
 
 on:
   push:
-    branches: [master]
+    branches: [master, develop]
   pull_request:
     branches: [master]
 
diff --git a/.github/workflows/test-devel.yml b/.github/workflows/test-devel.yml
@@ -0,0 +1,72 @@
+name: Test-devel
+
+on:
+  push:
+    branches: [develop]
+  pull_request:
+    branches: [develop]
+
+jobs:
+  test:
+    name: dev ${{ matrix.os }}, py-${{ matrix.python-version }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: true
+      matrix:
+        os: [ubuntu-18.04, windows-latest, macOS-latest]
+        python-version: ["3.6"]
+    defaults:
+      run:
+        shell: bash
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Setup Python
+        uses: actions/setup-python@v2
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - uses: pre-commit/action@v2.0.0
+
+      - name: Install Poetry
+        uses: snok/install-poetry@v1.1.6
+        with:
+          virtualenvs-create: true
+          virtualenvs-in-project: true
+
+      - name: Load cached venv
+        if: runner.os != 'Windows'
+        id: cached-poetry-dependencies
+        uses: actions/cache@v2
+        with:
+          path: .venv
+          key:
+            testvenv-${{ matrix.os }}-py${{matrix.python-version}}-${{
+            hashFiles('**/poetry.lock') }}
+
+      - name: Install dependencies
+        if: |
+          (runner.os != 'Windows') &&
+          (steps.cached-poetry-dependencies.outputs.cache-hit != 'true')
+        run: poetry install --no-interaction --no-root --no-dev
+
+      - name: Install library
+        run: poetry install --no-interaction --no-dev
+
+      - name: Run tests
+        run: |
+          source $VENV
+          poetry run coverage run -m unittest moabb.tests
+
+      - name: Run pipelines
+        run: |
+          source $VENV
+          poetry run python -m moabb.run --pipelines=./moabb/tests/test_pipelines/ --verbose
+
+      - name: Upload Coverage to Codecov
+        uses: codecov/codecov-action@v2
+        if: success()
+        with:
+          verbose: true
+          directory: /home/runner/work/moabb/moabb
+          files: ./.coverage
diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md
@@ -40,15 +40,16 @@ pull request to the master branch referencing the specific issue you addressed.
 1. Look for open issues or open one
 1. Discuss the problem and or propose a solution
 1. Fork it! (and clone fork locally)
-1. Branch from `master`: `git checkout master`
+1. Branch from `develop`: `git checkout --track develop`
 1. [Setup development environment](#setup-development-environment)
 1. Create your feature branch: `git checkout -b my-new-feature`
 1. Make changes
 1. Commit your changes: `git commit -m 'Add some feature'`
 1. Don't forget to fix issues from `pre-commit` pipeline (either add changes made by hooks
    or fix them manually in case of `flake8`)
 1. Push to the branch: `git push origin my-new-feature`
-1. Submit a pull request. Make sure it is based on the `master` branch when submitting! :D
+1. Submit a pull request. Make sure it is based on the `develop` branch when submitting!
+   :D
 1. Don't forget to update the
    [what's new](http://moabb.neurotechx.com/docs/whats_new.html) and
    [documentation](http://moabb.neurotechx.com/docs/index.html) pages if needed
diff --git a/README.md b/README.md
@@ -4,7 +4,7 @@
   <img alt="banner" src="/images/M.png/">
 </p>
 <p align=center>
-  Build a comprehensive benchmark of popular BCI algorithms applied on an extensive list of freely available EEG datasets.
+  Build a comprehensive benchmark of popular Brain-Computer Interface ([BCI](https://en.wikipedia.org/wiki/Brain%E2%80%93computer_interface)) algorithms applied on an extensive list of freely available EEG datasets.
 </p>
 
 ## Disclaimer
@@ -15,7 +15,7 @@ community.**
 [![Build Status](https://github.com/NeuroTechX/moabb/workflows/Test/badge.svg)](https://github.com/NeuroTechX/moabb/actions?query=branch%3Amaster)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![PyPI](https://img.shields.io/pypi/v/moabb?color=blue&style=plastic)](https://img.shields.io/pypi/v/moabb)
-[![PyPI - Downloads](https://img.shields.io/pypi/dm/moabb?color=brightgreen&style=plastic)](https://img.shields.io/pypi/dm/moabb)
+[![Downloads](https://pepy.tech/badge/moabb)](https://pepy.tech/project/moabb)
 
 ## Welcome!
 
@@ -41,6 +41,12 @@ one of the sections below, or just scroll down to find out more.
 
 ### The problem
 
+[Brain-Computer Interfaces](https://en.wikipedia.org/wiki/Brain%E2%80%93computer_interface)
+allow to interact with a computer using brain signals. In this project, we focus mostly on
+electroencephalographic signals
+([EEG](https://en.wikipedia.org/wiki/Electroencephalography)), that is a very active
+research domain, with worldwide scientific contributions. Still:
+
 - Reproducible Research in BCI has a long way to go.
 - While many BCI datasets are made freely available, researchers do not publish code, and
   reproducing results required to benchmark new algorithms turns out to be trickier than
diff --git a/codecov.yml b/codecov.yml
@@ -0,0 +1,7 @@
+comment: false
+github_checks:
+  annotations: false
+
+codecov:
+  notify:
+    require_ci_to_pass: no
diff --git a/docs/source/README.md b/docs/source/README.md
@@ -5,7 +5,7 @@
 ![banner](images/M.png)
 
 <p align=center>
-  Build a comprehensive benchmark of popular BCI algorithms applied on an extensive list of freely available EEG datasets.
+  Build a comprehensive benchmark of popular Brain-Computer Interface ([BCI](https://en.wikipedia.org/wiki/Brain%E2%80%93computer_interface)) algorithms applied on an extensive list of freely available EEG datasets.
 </p>
 
 ## Disclaimer
@@ -16,7 +16,7 @@ community.**
 [![Build Status](https://github.com/NeuroTechX/moabb/workflows/Test/badge.svg)](https://github.com/NeuroTechX/moabb/actions?query=branch%3Amaster)
 [![Code style: black](https://img.shields.io/badge/code%20style-black-000000.svg)](https://github.com/psf/black)
 [![PyPI](https://img.shields.io/pypi/v/moabb?color=blue&style=plastic)](https://img.shields.io/pypi/v/moabb)
-[![PyPI - Downloads](https://img.shields.io/pypi/dm/moabb?color=brightgreen&style=plastic)](https://img.shields.io/pypi/dm/moabb)
+[![Downloads](https://pepy.tech/badge/moabb)](https://pepy.tech/project/moabb)
 
 ## Welcome!
 
@@ -40,6 +40,12 @@ one of the sections below, or just scroll down to find out more.
 
 ### The problem
 
+[Brain-Computer Interfaces](https://en.wikipedia.org/wiki/Brain%E2%80%93computer_interface)
+allow to interact with a computer using brain signals. In this project, we focus mostly on
+electroencephalographic signals
+([EEG](https://en.wikipedia.org/wiki/Electroencephalography)), that is a very active
+research domain, with worldwide scientific contributions. Still:
+
 - Reproducible Research in BCI has a long way to go.
 - While many BCI datasets are made freely available, researchers do not publish code, and
   reproducing results required to benchmark new algorithms turns out to be trickier than
diff --git a/docs/source/conf.py b/docs/source/conf.py
@@ -233,6 +233,7 @@ def linkcode_resolve(domain, info):  # noqa: C901
     # Note the "1" or "True" value above as the third argument to indicate
     # an arbitrary url.
     "navbar_links": [
+        ("What's new", "whats_new"),
         ("API", "api"),
         ("Gallery", "auto_examples/index"),
         ("Tutorials", "auto_tutorials/index"),
diff --git a/docs/source/whats_new.rst b/docs/source/whats_new.rst
@@ -11,22 +11,47 @@ What's new
    - "API changes" for backward-incompatible changes
 .. _current:
 
+
 Develop branch
 ---------------
 
 Enhancements
 ~~~~~~~~~~~~
 
+- None
+
+Bugs
+~~~~
+
+- None
+
+API changes
+~~~~~~~~~~~
+
+- None
+
+
+Version - 0.4.3 (Stable - PyPi)
+---------------
+
+Enhancements
+~~~~~~~~~~~~
+
 - Rewrite Lee2019 to add P300 and SSVEP datasets (:gh:`217` by `Pierre Guetchel`_)
 
-  Bugs
+Bugs
 ~~~~
 
+- Avoid information leakage for MNE Epochs pipelines in evaluation (:gh:`222` by `Sylvain Chevallier`_)
+- Correct error in set_download_dir (:gh:`225` by `Sylvain Chevallier`_)
+- Ensure that channel order is consistent across dataset when channel argument is specified in paradigm (:gh:`229` by `Sylvain Chevallier`_)
 
 API changes
 ~~~~~~~~~~~
 
-Version - 0.4.2 (Stable - PyPi)
+- ch_names argument added to init of moabb.datasets.fake.FakeDataset (:gh:`229` by `Sylvain Chevallier`_)
+
+Version - 0.4.2
 ---------------
 
 Enhancements
diff --git a/moabb/__init__.py b/moabb/__init__.py
@@ -1,4 +1,4 @@
 # flake8: noqa
-__version__ = "0.4.2"
+__version__ = "0.4.3"
 
 from moabb.utils import set_log_level
diff --git a/moabb/datasets/base.py b/moabb/datasets/base.py
@@ -44,8 +44,7 @@ class BaseDataset(metaclass=abc.ABCMeta):
         Imagery interval as defined in the dataset description
 
     paradigm: ['p300','imagery', 'ssvep']
-        Defines what sort of dataset this is (currently only imagery is
-        implemented)
+        Defines what sort of dataset this is
 
     doi: DOI for dataset, optional (for now)
     """
diff --git a/moabb/datasets/fake.py b/moabb/datasets/fake.py
@@ -9,8 +9,24 @@
 class FakeDataset(BaseDataset):
     """Fake Dataset for test purpose.
 
-    The dataset has 2 sessions, 10 subjects, and 3 classes.
+    By default, the dataset has 2 sessions, 10 subjects, and 3 classes.
 
+    Parameters
+    ----------
+    event_list: list or tuple of str
+        List of event to generate, default: ("fake_c1", "fake_c2", "fake_c3")
+    n_sessions: int, default 2
+        Number of session to generate
+    n_runs: int, default 2
+        Number of runs to generate
+    n_subjects: int, default 10
+        Number of subject to generate
+    paradigm: ['p300','imagery', 'ssvep']
+        Defines what sort of dataset this is
+    channels: list or tuple of str
+        List of channels to generate, default ("C3", "Cz", "C4")
+
+        .. versionadded:: 0.4.3
     """
 
     def __init__(
@@ -20,9 +36,11 @@ def __init__(
         n_runs=2,
         n_subjects=10,
         paradigm="imagery",
+        channels=("C3", "Cz", "C4"),
     ):
         self.n_runs = n_runs
         event_id = {ev: ii + 1 for ii, ev in enumerate(event_list)}
+        self.channels = channels
         super().__init__(
             list(range(1, n_subjects + 1)),
             n_sessions,
@@ -42,21 +60,18 @@ def _get_single_subject_data(self, subject):
         return data
 
     def _generate_raw(self):
-
-        ch_names = ["C3", "Cz", "C4"]
-
         montage = make_standard_montage("standard_1005")
         sfreq = 128
         duration = len(self.event_id) * 60
-        eeg_data = 2e-5 * np.random.randn(duration * sfreq, len(ch_names))
+        eeg_data = 2e-5 * np.random.randn(duration * sfreq, len(self.channels))
         y = np.zeros((duration * sfreq))
         for ii, ev in enumerate(self.event_id):
             start_idx = (1 + 5 * ii) * 128
             jump = 5 * len(self.event_id) * 128
             y[start_idx::jump] = self.event_id[ev]
 
-        ch_types = ["eeg"] * len(ch_names) + ["stim"]
-        ch_names = ch_names + ["stim"]
+        ch_types = ["eeg"] * len(self.channels) + ["stim"]
+        ch_names = list(self.channels) + ["stim"]
 
         eeg_data = np.c_[eeg_data, y]
 
diff --git a/moabb/evaluations/evaluations.py b/moabb/evaluations/evaluations.py
@@ -121,8 +121,9 @@ def _evaluate(self, dataset, pipelines):
                         X_ = X[ix]
                         y_ = y[ix] if self.mne_labels else y_cv
                         for train, test in cv.split(X_, y_):
-                            clf.fit(X_[train], y_[train])
-                            acc.append(scorer(clf, X_[test], y_[test]))
+                            cvclf = clone(clf)
+                            cvclf.fit(X_[train], y_[train])
+                            acc.append(scorer(cvclf, X_[test], y_[test]))
                         acc = np.array(acc)
                     else:
                         acc = cross_val_score(
@@ -322,8 +323,9 @@ def evaluate(self, dataset, pipelines):
                 for train, test in cv.split(X, y, groups):
                     t_start = time()
                     if isinstance(X, BaseEpochs):
-                        clf.fit(X[train], y[train])
-                        score = scorer(clf, X[test], y[test])
+                        cvclf = clone(clf)
+                        cvclf.fit(X[train], y[train])
+                        score = scorer(cvclf, X[test], y[test])
                     else:
                         score = _fit_and_score(
                             clone(clf),
diff --git a/moabb/paradigms/base.py b/moabb/paradigms/base.py
@@ -118,7 +118,9 @@ def process_raw(self, raw, dataset, return_epochs=False):  # noqa: C901
         if self.channels is None:
             picks = mne.pick_types(raw.info, eeg=True, stim=False)
         else:
-            picks = mne.pick_types(raw.info, stim=False, include=self.channels)
+            picks = mne.pick_channels(
+                raw.info["ch_names"], include=self.channels, ordered=True
+            )
 
         # pick events, based on event_id
         try:
diff --git a/moabb/paradigms/p300.py b/moabb/paradigms/p300.py
@@ -106,10 +106,15 @@ def process_raw(self, raw, dataset, return_epochs=False):  # noqa: C901
         else:
             events, _ = mne.events_from_annotations(raw, verbose=False)
 
-        channels = () if self.channels is None else self.channels
-
         # picks channels
+        channels = () if self.channels is None else self.channels
         picks = mne.pick_types(raw.info, eeg=True, stim=False, include=channels)
+        if self.channels is None:
+            picks = mne.pick_types(raw.info, eeg=True, stim=False)
+        else:
+            picks = mne.pick_channels(
+                raw.info["ch_names"], include=channels, ordered=True
+            )
 
         # get event id
         event_id = self.used_events(dataset)
diff --git a/moabb/tests/paradigms.py b/moabb/tests/paradigms.py
diff --git a/moabb/utils.py b/moabb/utils.py
diff --git a/poetry.lock b/poetry.lock
diff --git a/pyproject.toml b/pyproject.toml
