From 567e73a731fcfe243a26b3cfae73e4ab3fd58c6e Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Tue, 17 Dec 2024 08:49:07 -0600 Subject: [PATCH 01/10] Add `--allow-any-path` option to the docs on uploading data --- docs/upload.md | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/docs/upload.md b/docs/upload.md index 4b545f5..61316cc 100644 --- a/docs/upload.md +++ b/docs/upload.md @@ -178,6 +178,13 @@ cd /path/to/my/staging/area/101010 dandi upload -i linc ``` +If you would like to upload non-Zarr files, please use the following command: + +``` +cd /path/to/my/staging/area/101010 +DANDI_DEVEL=1 dandi upload -i linc --allow-any-path +``` + Check the output in your terminal for validation errors. If there were no errors, your data files should now appear on lincbrain.org. ## Delete data files or directories From a4473befa3a75aafaf19d1c929a5081406528c1d Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Tue, 24 Dec 2024 10:10:16 -0600 Subject: [PATCH 02/10] Remove developer options --- docs/upload.md | 7 ------- 1 file changed, 7 deletions(-) diff --git a/docs/upload.md b/docs/upload.md index 61316cc..4b545f5 100644 --- a/docs/upload.md +++ b/docs/upload.md @@ -178,13 +178,6 @@ cd /path/to/my/staging/area/101010 dandi upload -i linc ``` -If you would like to upload non-Zarr files, please use the following command: - -``` -cd /path/to/my/staging/area/101010 -DANDI_DEVEL=1 dandi upload -i linc --allow-any-path -``` - Check the output in your terminal for validation errors. If there were no errors, your data files should now appear on lincbrain.org. ## Delete data files or directories From fd031747b9c62824a05325a8c4747a1cbdaa8804 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Tue, 24 Dec 2024 10:14:30 -0600 Subject: [PATCH 03/10] Add notes about local files --- docs/upload.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/docs/upload.md b/docs/upload.md index 4b545f5..9165cd8 100644 --- a/docs/upload.md +++ b/docs/upload.md @@ -171,6 +171,9 @@ sub-Ken1 sample-slice0010 tissue ## Upload your data + +If you are uploading to a new or existing dataset on lincbrain.org, please ensure that you have the `dandiset.yaml` and `dataset_description.json` files in your local directory (`/path/to/my/staging/area/101010`). + Upload the data from your local machine to lincbrain.org: ``` From cda1ef8587ba745a373d3f9cd6a790268adef159 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Tue, 24 Dec 2024 10:29:26 -0600 Subject: [PATCH 04/10] Add instructions for downloading a dataset without imaging files --- docs/upload.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/upload.md b/docs/upload.md index 9165cd8..1ce02e6 100644 --- a/docs/upload.md +++ b/docs/upload.md @@ -25,6 +25,8 @@ Log into lincbrain.org and click on the button with your initials at the top rig `export DANDI_API_KEY=` ## Download your new (empty) dataset locally + +### For a new dataset You can find the command that you need to run to download a specific dataset by navigating to the dataset landing page on lincbrain.org, clicking on the `DOWNLOAD` drop-down menu that you'll see at the top right corner of that page, and copying the `dandi download ...` command that you see when you click on that menu. On your local machine, create a directory that you will use as a staging area for uploading data. Then cd into this directory, and run the download command that you copied above. For example: @@ -35,6 +37,14 @@ dandi download https://lincbrain.org/dandiset/101010/draft The above example will create a directory called `/path/to/my/staging/area/101010` with a file called `dandiset.yaml` in it. Any data files that you want to upload to your new lincbrain.org dataset have to first be saved here, and organized according to the [Brain Imaging Data Structure (BIDS)](https://bids-specification.readthedocs.io/). +### For an existing dataset +For an existing dataset you will probably not want to download the entire dataset from lincbrain.org which can be many terabyes in size. You can download an empty version of the dataset using the command below. You will need to replace `101010` with your dataset id. + +``` +cd /path/to/my/staging/area +dandi download --preserve-tree dandi://linc/101010@draft/dataset_description.json +``` + ## Organize your data An example of how to organize a dataset that includes dMRI and histology data from two brains is shown below: From 10c845985bf1fde94dbb548bcc7eccdc69970fe5 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Mon, 30 Dec 2024 10:30:17 -0600 Subject: [PATCH 05/10] Update text --- docs/upload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/upload.md b/docs/upload.md index 1ce02e6..ffbc154 100644 --- a/docs/upload.md +++ b/docs/upload.md @@ -29,7 +29,7 @@ Log into lincbrain.org and click on the button with your initials at the top rig ### For a new dataset You can find the command that you need to run to download a specific dataset by navigating to the dataset landing page on lincbrain.org, clicking on the `DOWNLOAD` drop-down menu that you'll see at the top right corner of that page, and copying the `dandi download ...` command that you see when you click on that menu. -On your local machine, create a directory that you will use as a staging area for uploading data. Then cd into this directory, and run the download command that you copied above. For example: +On your local machine, create a directory that you will use as a staging area for uploading data. Then change into this directory, and run the download command that you copied above. For example: ``` cd /path/to/my/staging/area dandi download https://lincbrain.org/dandiset/101010/draft From e93c3d5ed4d3a0824084e75f0b5a17aea70cbb96 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Mon, 30 Dec 2024 10:32:18 -0600 Subject: [PATCH 06/10] Update existing dataset section --- docs/upload.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/docs/upload.md b/docs/upload.md index ffbc154..f02bec5 100644 --- a/docs/upload.md +++ b/docs/upload.md @@ -38,13 +38,17 @@ dandi download https://lincbrain.org/dandiset/101010/draft The above example will create a directory called `/path/to/my/staging/area/101010` with a file called `dandiset.yaml` in it. Any data files that you want to upload to your new lincbrain.org dataset have to first be saved here, and organized according to the [Brain Imaging Data Structure (BIDS)](https://bids-specification.readthedocs.io/). ### For an existing dataset -For an existing dataset you will probably not want to download the entire dataset from lincbrain.org which can be many terabyes in size. You can download an empty version of the dataset using the command below. You will need to replace `101010` with your dataset id. +For an existing dataset you will probably not want to download the entire dataset from lincbrain.org which can be many terabyes in size. Using the command below, you can download the dataset with just the `dandiset.yaml` and `dataset_description.json` files. You will need to replace `101010` with your `dandiset-id`. ``` cd /path/to/my/staging/area dandi download --preserve-tree dandi://linc/101010@draft/dataset_description.json ``` +The above example will create a directory called `/path/to/my/staging/area/101010`. Any data files that you want to upload to your new lincbrain.org dataset have to first be saved here, and organized according to the [Brain Imaging Data Structure (BIDS)](https://bids-specification.readthedocs.io/). + +For more information, on the `--preserve-tree` option please see the [DANDI Handbook](https://www.dandiarchive.org/handbook/12_download/#download-the-dandisetyaml-file-and-a-specific-file-within-the-directory-tree-of-the-dandiset). + ## Organize your data An example of how to organize a dataset that includes dMRI and histology data from two brains is shown below: From b982ebaaf385fa1ffccc5a530bf2178b6f3d9400 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Mon, 30 Dec 2024 10:37:30 -0600 Subject: [PATCH 07/10] Remove spacing --- docs/upload.md | 3 --- 1 file changed, 3 deletions(-) diff --git a/docs/upload.md b/docs/upload.md index f02bec5..5b86234 100644 --- a/docs/upload.md +++ b/docs/upload.md @@ -182,10 +182,7 @@ sub-Ken1 sample-slice0002 tissue sub-Ken1 sample-slice0009 tissue sub-Ken1 sample-slice0010 tissue ``` - - ## Upload your data - If you are uploading to a new or existing dataset on lincbrain.org, please ensure that you have the `dandiset.yaml` and `dataset_description.json` files in your local directory (`/path/to/my/staging/area/101010`). Upload the data from your local machine to lincbrain.org: From 09887bc77bdab40ffad3d8a6b6643b56f96c29e0 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Mon, 30 Dec 2024 10:41:49 -0600 Subject: [PATCH 08/10] Revert spacing removal --- docs/upload.md | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/upload.md b/docs/upload.md index 5b86234..9135844 100644 --- a/docs/upload.md +++ b/docs/upload.md @@ -182,6 +182,7 @@ sub-Ken1 sample-slice0002 tissue sub-Ken1 sample-slice0009 tissue sub-Ken1 sample-slice0010 tissue ``` + ## Upload your data If you are uploading to a new or existing dataset on lincbrain.org, please ensure that you have the `dandiset.yaml` and `dataset_description.json` files in your local directory (`/path/to/my/staging/area/101010`). From 52483c5abf884d51ac78052d511d35523436d468 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Mon, 30 Dec 2024 10:42:36 -0600 Subject: [PATCH 09/10] Update text --- docs/upload.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/upload.md b/docs/upload.md index 9135844..9d37bc8 100644 --- a/docs/upload.md +++ b/docs/upload.md @@ -184,7 +184,7 @@ sub-Ken1 sample-slice0010 tissue ``` ## Upload your data -If you are uploading to a new or existing dataset on lincbrain.org, please ensure that you have the `dandiset.yaml` and `dataset_description.json` files in your local directory (`/path/to/my/staging/area/101010`). +If you are uploading to a new or existing dataset on lincbrain.org, please ensure that you have the `dandiset.yaml` and `dataset_description.json` files in your local directory (i.e. the `/path/to/my/staging/area/101010` in the example above). Upload the data from your local machine to lincbrain.org: From e678e686773a27c1a53f88eceab744c765749ea9 Mon Sep 17 00:00:00 2001 From: Kabilar Gunalan Date: Mon, 30 Dec 2024 15:30:58 -0600 Subject: [PATCH 10/10] Minor text fixes --- docs/upload.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/upload.md b/docs/upload.md index 1b6ae8f..d52c58c 100644 --- a/docs/upload.md +++ b/docs/upload.md @@ -48,9 +48,9 @@ cd /path/to/my/staging/area dandi download --preserve-tree dandi://linc/101010@draft/dataset_description.json ``` -The above example will create a directory called `/path/to/my/staging/area/101010`. Any data files that you want to upload to your new lincbrain.org dataset have to first be saved here, and organized according to the [Brain Imaging Data Structure (BIDS)](https://bids-specification.readthedocs.io/). +The above example will create a directory called `/path/to/my/staging/area/101010`. Any data files that you want to upload to the lincbrain.org dataset have to first be saved here, and organized according to the [Brain Imaging Data Structure (BIDS)](https://bids-specification.readthedocs.io/). -For more information, on the `--preserve-tree` option please see the [DANDI Handbook](https://www.dandiarchive.org/handbook/12_download/#download-the-dandisetyaml-file-and-a-specific-file-within-the-directory-tree-of-the-dandiset). +For more information on the `--preserve-tree` option please see the [DANDI Handbook](https://www.dandiarchive.org/handbook/12_download/#download-the-dandisetyaml-file-and-a-specific-file-within-the-directory-tree-of-the-dandiset). ## Organize your data An example from a (fictional) dataset that includes dMRI and histology data from two brains is shown below. This can be used as a guide for organizing your own dataset. A few things to watch out for: @@ -230,7 +230,7 @@ index name ``` ## Upload your data -If you are uploading to a new or existing dataset on lincbrain.org, please ensure that you have the `dandiset.yaml` and `dataset_description.json` files in your local directory (i.e. the `/path/to/my/staging/area/101010` in the example above). +If you are uploading to a new or existing dataset on lincbrain.org, please ensure that you have the `dandiset.yaml` and `dataset_description.json` files in your local directory (i.e. the `/path/to/my/staging/area/101010` directory in the example above). Upload the data from your local machine to lincbrain.org: