Allow renaming datasets & dataset with duplicate names #8075

MichaelBuessemeyer · 2024-09-12T16:02:08Z

Further Notes:

Quite some of the line changes are the result of moving the ObjectId class to the utils package so that all wk backend servers have access to this class.

URL of deployed dev instance (used for testing):

https://allowdatasetrenaming.webknossos.xyz

Steps to test:

Give two datasets the same name and check whether annotations and so on works
Test whether the task system still works with duplicate dataset names
check dataset upload
- dataset upload
- add remote
- compose
...

TODOs:

I would also suggest to

Execute the screenshot tests once for this branch. Take care to change the branch name in the CI config (or local command), so that the tests are actually ran on this branch.

Looks much better now :) Did you re-test that exceptions are caught as expected?
undo application.conf & snapshot changes

Issues:

(Please delete unneeded items, merge only when none are left open)

Updated changelog
Updated migration guide if applicable
Updated documentation if applicable
Adapted wk-libs python client if relevant API parts change
Removed dev-only changes like prints and application.conf edits
Considered common edge cases
Needs datastore update after deployment

…ir dataset addressing scheme

… the check for already existing name)

…gaId - Includes moving ObjectId to uitls package

…store & add legacy routes - Undo renaming DataSourceId to LegacyDataSourceId

…ow-dataset-renaming

…ataset object

frontend/javascripts/admin/task/task_create_form_view.tsx

…ow-dataset-renaming

MichaelBuessemeyer · 2024-11-12T15:06:00Z

app/controllers/AnnotationIOController.scala

    } else {
-      Future.successful(JsonBadRequest(Messages("nml.file.noFile")))
+      Fox.paramFailure("NML upload failed", Empty, Empty, None)


@fm3 this does not work as expected. The paramFailure is not caught in EtendedController.scala. Do you maybe know more about this? / How to make this work?

…ow-dataset-renaming

daniel-wer

First part of my frontend code review. Good stuff! I'll continue tomorrow :)

daniel-wer · 2024-11-12T18:53:59Z

conf/webknossos.versioned.routes

@@ -3,7 +3,9 @@
 # example: assume, the features route has changed, introducing v2. The older v1 needs to be provided in the legacyApiController
 # Note: keep this in sync with the reported version numbers in the utils.ApiVersioning trait

-# version log:
+# version log:updateDatasetV8
+ # changed in v9: Datasets are now identified by their id, not their name. The routes now need to pass a datasets id instead of a name and organization id tuple.


Suggested change

# changed in v9: Datasets are now identified by their id, not their name. The routes now need to pass a datasets id instead of a name and organization id tuple.

# changed in v9: Datasets are now identified by their id, not their name. The routes now need to pass a dataset id instead of a name and organization id tuple.

daniel-wer · 2024-11-12T18:54:22Z

conf/webknossos.versioned.routes

-# version log:
+# version log:updateDatasetV8
+ # changed in v9: Datasets are now identified by their id, not their name. The routes now need to pass a datasets id instead of a name and organization id tuple.
+   # Requests to the TracingStore and DatasStore need to address a dataset based on it directoryName and organization id.


Suggested change

# Requests to the TracingStore and DatasStore need to address a dataset based on it directoryName and organization id.

# Requests to the TracingStore and DatasStore need to address a dataset based on its directoryName and organization id.

daniel-wer · 2024-11-12T19:00:43Z

frontend/javascripts/admin/admin_rest_api.ts

+  // Formatting the dataSourceId to the old format so that the backend can parse it.
+  // And removing the datasetId as the datastore cannot use it.


Why can't this be adapted to the new dataset id in the backend?

Because @fm3 and I agreed upon that the tracing & data store now identify a dataset by its directoryName and organization id. This is especially needed for the datastore, as the datastore needs the orgaId and directory name of a dataset to locate the folder whether the dataset is saved on disk.

Otherwise, each request to the datastore needs a (cached) request to the core backend to resolve the datasetId to its orgaId & directoryName tuple.

And orgaId needs to be named team and the directoryName name due to legacy reasons.

daniel-wer · 2024-11-12T19:04:27Z

frontend/javascripts/admin/api/jobs.ts

@@ -73,35 +76,32 @@ export async function cancelJob(jobId: string): Promise<APIJob> {
 }

 export async function startConvertToWkwJob(
-  datasetName: string,
-  organizationId: string,
+  datasetId: APIDataset["id"],


This is not unified in this file, and in all other files I've seen, the type was simply denoted as string. I would prefer that as it's easier to see the type when looking at the code. If you agree, please search and replace.

Suggested change

datasetId: APIDataset["id"],

datasetId: string,

sure 👍, thought this would make changing the type easier but honestly this seems very unlikely and a simple string is definitely easier to read :)

daniel-wer · 2024-11-12T19:08:55Z

frontend/javascripts/admin/api/mesh.ts

@@ -31,7 +31,7 @@ type SegmentInfo = {

 export function getMeshfileChunksForSegment(
  dataStoreUrl: string,
-  datasetId: APIDatasetId,
+  datasetId: APIDataSourceId,


Should be renamed to dataSourceId, also for the getMeshfileChunkData method.

Nice, good catch. Thanks 🙏

frontend/javascripts/admin/task/task_create_form_view.tsx

frontend/javascripts/dashboard/advanced_dataset/dataset_action_view.tsx

daniel-wer · 2024-11-12T20:05:56Z

frontend/javascripts/dashboard/advanced_dataset/dataset_table.tsx

+  static getRowKey(dataset: APIDatasetCompact) {
+    return dataset.id;
+  }
  getRowKey() {
-    return this.data.name;
+    return DatasetRenderer.getRowKey(this.data);


Why is this static getRowKey function needed? Same for the other one later in the file.

Becasue I use them in ll. 582ff.

if (selectedDatasets.length > 0) { selectedRowKeys = selectedDatasets.map(DatasetRenderer.getRowKey); } else if (context.selectedFolder && "name" in context.selectedFolder) { selectedRowKeys = [FolderRenderer.getRowKey(context.selectedFolder as FolderItemWithName)]; }

I need them to be static, because In these lines I do not have access to a "row object" but I still want to access the row key here. => I could just map to the respective "key" property, but this would me the logic of what the key is to be splitted up / stored at two locations. Thus, when changing this later, there is only a single place where the "key-generation" needs to be adjusted -> the code is DRYer this way :)

daniel-wer · 2024-11-12T20:07:25Z

frontend/javascripts/dashboard/advanced_dataset/dataset_table.tsx

+          <Link
+            to={`/datasets/${this.data.owningOrganization}/${this.data.name}/view`}
+            title="View Dataset"
+            className="incognito-link dataset-table-name"
+          >
+            Test disambiguate
+          </Link>


Can be removed?

Yes I will definitely do that before merging. But I would currently keep this to make sure, that the disambiguation for old links still works.

daniel-wer

Second half of my PR review :) I haven't done any testing yet. Would you say now is a good time to do so?

I would also suggest to

Execute the screenshot tests once for this branch. Take care to change the branch name in the CI config (or local command), so that the tests are actually ran on this branch.

daniel-wer · 2024-11-13T09:24:27Z

frontend/javascripts/dashboard/dataset/dataset_settings_view.tsx

@@ -367,14 +366,14 @@ class DatasetSettingsView extends React.PureComponent<PropsWithFormAndRouter, St
    const dataSource = JSON.parse(formValues.dataSourceJson);

    if (dataset != null && this.didDatasourceChange(dataSource)) {
-      await updateDatasetDatasource(this.props.datasetId.name, dataset.dataStore.url, dataSource);
+      await updateDatasetDatasource(dataset.directoryName, dataset.dataStore.url, dataSource);


I think it's a useful warning and I would change it from info to warning :)

daniel-wer · 2024-11-13T14:35:04Z

frontend/javascripts/dashboard/dataset/dataset_settings_view.tsx

+    const maybeDataSourceId = {
+      owningOrganization: dataset?.owningOrganization || "",
+      directoryName: dataset?.directoryName || "",
+    };


Why does the dataset sometimes not exist? Is that after an upload or is there another reason?

Because the DatasetSettingsView fetches the dataset given by its props.datsaetId upon mounting. Therefore, there are some rendering cycles in which the dataset does not exist in the state.

daniel-wer · 2024-11-13T14:36:39Z

frontend/javascripts/dashboard/dataset/dataset_settings_view.tsx

@@ -576,7 +590,7 @@ class DatasetSettingsView extends React.PureComponent<PropsWithFormAndRouter, St
        children: (
          <Hideable hidden={this.state.activeTabKey !== "defaultConfig"}>
            <DatasetSettingsViewConfigTab
-              datasetId={this.props.datasetId}
+              dataSourceId={maybeDataSourceId}


Does this component work with owningOrganization and directoryName sometimes being empty strings? On a quick glance it looks like it doesn't as it calls getMappingsForDatasetLayer(dataStoreURL, dataSourceId, layerName) for example.

Hmm this never came up as an error 🤔

The magic seems to happen in line 636
<Spin size="large" spinning={this.state.isLoading}>
which wraps the whole tabs component. Therefore, this should be safe.

I changed the code to only render the DatasetSettingsViewConfigTab component in case the maybeDataSourceId has valid values. Else nothing is rendered and a comment explains that this case shouldn't occur.

frontend/javascripts/libs/utils.ts

daniel-wer · 2024-11-13T18:25:17Z

frontend/javascripts/router.tsx

              />
+              <Route path="/datasets/:datasetNameAndId/view" render={this.tracingViewMode} />
+              {/*maybe this also needs a legacy route?*/}


Yes I would think so, at least there are such links in the wild, for example in the connectome viewer repository and I've also sent such links via email to clients.

daniel-wer · 2024-11-13T18:29:36Z

frontend/javascripts/test/fixtures/dataset_server_object.ts


-export default {
+const apiDataset: APIDataset = {


daniel-wer · 2024-11-13T18:32:44Z

frontend/javascripts/test/model/model.spec.ts

@@ -1,18 +1,22 @@
-// @ts-nocheck


Awesome, thank you!

daniel-wer · 2024-11-13T18:38:27Z

frontend/javascripts/test/puppeteer/dataset_rendering.screenshot.ts

@@ -106,19 +106,38 @@ const datasetConfigOverrides: Record<string, PartialDatasetConfiguration> = {
  },
 };

+const datasetNameToId: Record<string, string> = {};
+datasetNames.map(async (datasetName) => {


I think this test does not do what its description and assertion state. Instead it fetches the ids for all datasets. Doing that as part of a test seems hacky. Can't you use test.before for that?

sure, good find :)

daniel-wer · 2024-11-13T18:44:26Z

...avascripts/test/snapshots/public-test/test-bundle/test/backend-snapshot-tests/misc.e2e.js.md

+      jobsEnabled: true,
      openIdConnectEnabled: false,
      optInTabs: [],
      publicDemoDatasetUrl: 'https://webknossos.org/datasets/scalable_minds/l4dense_motta_et_al_demo',
      recommendWkorgInstance: true,
      segmentAnythingEnabled: false,
      taskReopenAllowedInSeconds: 30,
-      voxelyticsEnabled: false,
+      voxelyticsEnabled: true,


Needs to be reverted before merging, together with the snap file.

daniel-wer · 2024-11-13T18:46:48Z

frontend/javascripts/types/api_flow_types.ts

  | "created"
  | "isEditable"
  | "lastUsedByUser"
  | "tags"
  | "isUnreported"
 >;
 export type APIDatasetCompact = APIDatasetCompactWithoutStatusAndLayerNames & {
-  id?: string;
+  id: string; // Open question: Why was this optional?, The backend code clearly always returns an id ... :thinking:


Could be legacy or an oversight, but I don't know. I think it's fine to change if typescript doesn't complain.

Yeah, I did a double check in the backend and it seems to always send the datasetId 🤷

…ization id

- the backend always sends an id for compacted datasets

…ow-dataset-renaming

MichaelBuessemeyer

Thanks for the feedback @daniel-wer. I applied / commented each of your comments. Please have another look. Testing should hopefully work :)

MichaelBuessemeyer · 2024-11-13T14:40:40Z

frontend/javascripts/admin/admin_rest_api.ts

+  // Formatting the dataSourceId to the old format so that the backend can parse it.
+  // And removing the datasetId as the datastore cannot use it.


And orgaId needs to be named team and the directoryName name due to legacy reasons.

MichaelBuessemeyer · 2024-11-13T14:44:16Z

frontend/javascripts/admin/api/jobs.ts

@@ -73,35 +76,32 @@ export async function cancelJob(jobId: string): Promise<APIJob> {
 }

 export async function startConvertToWkwJob(
-  datasetName: string,
-  organizationId: string,
+  datasetId: APIDataset["id"],


sure 👍, thought this would make changing the type easier but honestly this seems very unlikely and a simple string is definitely easier to read :)

MichaelBuessemeyer · 2024-11-13T14:46:34Z

frontend/javascripts/admin/api/mesh.ts

@@ -31,7 +31,7 @@ type SegmentInfo = {

 export function getMeshfileChunksForSegment(
  dataStoreUrl: string,
-  datasetId: APIDatasetId,
+  datasetId: APIDataSourceId,


Nice, good catch. Thanks 🙏

MichaelBuessemeyer · 2024-11-13T14:49:24Z

frontend/javascripts/admin/api/tasks.ts

Yeah, I just gave it a quick double check: No changes at all :)

MichaelBuessemeyer · 2024-11-13T14:51:17Z

frontend/javascripts/admin/dataset/composition_wizard/02_upload_files.tsx

@@ -166,10 +166,12 @@ async function parseNmlFiles(fileList: FileList): Promise<Partial<WizardContext>
    throw new SoftError("NML files should not be empty.");
  }

-  const { trees: trees1, datasetName: datasetName1 } = await parseNml(nmlString1);
-  const { trees: trees2, datasetName: datasetName2 } = await parseNml(nmlString2);
+  // TODO: Now the datasetName stored in the nml is interpreted as the path of the dataset. -> call to legacy route is necessary.


Yes, good find, this is an open todo: @fm3 / @daniel-wer Could we have please have a small discussion on this? 🙏

MichaelBuessemeyer · 2024-11-13T15:50:35Z

frontend/javascripts/dashboard/advanced_dataset/dataset_table.tsx

+          <Link
+            to={`/datasets/${this.data.owningOrganization}/${this.data.name}/view`}
+            title="View Dataset"
+            className="incognito-link dataset-table-name"
+          >
+            Test disambiguate
+          </Link>


Yes I will definitely do that before merging. But I would currently keep this to make sure, that the disambiguation for old links still works.

MichaelBuessemeyer · 2024-11-14T08:07:20Z

frontend/javascripts/dashboard/dataset/dataset_settings_view.tsx

+    const maybeDataSourceId = {
+      owningOrganization: dataset?.owningOrganization || "",
+      directoryName: dataset?.directoryName || "",
+    };


Because the DatasetSettingsView fetches the dataset given by its props.datsaetId upon mounting. Therefore, there are some rendering cycles in which the dataset does not exist in the state.

MichaelBuessemeyer · 2024-11-14T08:21:16Z

frontend/javascripts/dashboard/dataset/dataset_settings_view.tsx

@@ -576,7 +590,7 @@ class DatasetSettingsView extends React.PureComponent<PropsWithFormAndRouter, St
        children: (
          <Hideable hidden={this.state.activeTabKey !== "defaultConfig"}>
            <DatasetSettingsViewConfigTab
-              datasetId={this.props.datasetId}
+              dataSourceId={maybeDataSourceId}


Hmm this never came up as an error 🤔

The magic seems to happen in line 636
<Spin size="large" spinning={this.state.isLoading}>
which wraps the whole tabs component. Therefore, this should be safe.

I changed the code to only render the DatasetSettingsViewConfigTab component in case the maybeDataSourceId has valid values. Else nothing is rendered and a comment explains that this case shouldn't occur.

MichaelBuessemeyer · 2024-11-14T09:19:54Z

frontend/javascripts/test/puppeteer/dataset_rendering.screenshot.ts

@@ -106,19 +106,38 @@ const datasetConfigOverrides: Record<string, PartialDatasetConfiguration> = {
  },
 };

+const datasetNameToId: Record<string, string> = {};
+datasetNames.map(async (datasetName) => {


sure, good find :)

MichaelBuessemeyer · 2024-11-14T09:24:05Z

frontend/javascripts/types/api_flow_types.ts

  | "created"
  | "isEditable"
  | "lastUsedByUser"
  | "tags"
  | "isUnreported"
 >;
 export type APIDatasetCompact = APIDatasetCompactWithoutStatusAndLayerNames & {
-  id?: string;
+  id: string; // Open question: Why was this optional?, The backend code clearly always returns an id ... :thinking:


Yeah, I did a double check in the backend and it seems to always send the datasetId 🤷

MichaelBuessemeyer

Thanks for your review @daniel-wer 🎉

I think I applied your suggestions, commented on comments and so on :).
The dev instance https://allowdatasetrenaming.webknossos.xyz/ should also be ready for testing 🎉

fm3

Getting close! I added a few more comments/replied to threads.

I think the two open backend topics are the dataset upload protocol and the ParamFailure thing for NML upload. I hope I’ll have time for both on monday.

fm3 · 2024-11-14T14:46:13Z

app/models/task/TaskCreationParameters.scala

@@ -21,6 +23,39 @@ object TaskParameters {
  implicit val taskParametersFormat: Format[TaskParameters] = Json.format[TaskParameters]
 }

+case class TaskParametersWithDatasetId(taskTypeId: String,


Suggested change

case class TaskParametersWithDatasetId(taskTypeId: String,

case class TaskParameters(taskTypeId: String,

I’d move the old TaskParameters to LegacyApiController.scala and name it LegacyTaskParameters. This way, the current code doesn’t have to know/suggest about the backwards compatibility at all.

...datastore/app/com/scalableminds/webknossos/datastore/services/uploading/ComposeService.scala

fm3 · 2024-11-14T15:04:51Z

app/models/annotation/nml/NmlParser.scala

+            basePath: Option[String] = None)(implicit m: MessagesProvider,
+                                             ec: ExecutionContext,
+                                             ctx: DBAccessContext): Fox[NmlParseSuccessWithoutFile] = {
+    val foxInABox = try {


Looks much better now :) Did you re-test that exceptions are caught as expected?

fm3 · 2024-11-14T15:09:26Z

app/models/dataset/DatasetService.scala

    foundDatasetOpt match {
      case Some(foundDataset) if foundDataset._dataStore == dataStore.name =>
        updateKnownDataSource(foundDataset, dataSource, dataStore).toFox.map(Some(_))
      case Some(foundDataset) => // This only returns None for Datasets that are present on a normal Datastore but also got reported from a scratch Datastore
        updateDataSourceDifferentDataStore(foundDataset, dataSource, dataStore)
      case _ =>
-        insertNewDataset(dataSource, dataStore).toFox.map(Some(_))
+        insertNewDataset(dataSource, dataSource.id.directoryName, dataStore).toFox
+          .map(Some(_)) // TODO: Discuss how to better handle this case


Hmm, I see. Yes, it’s kind of unexpected. Unfortunately I don’t really have a good idea here. Since this was not introduced in this PR, I’d leave it as is for the time being.

daniel-wer

I was only able to do a quick test, but will continue testing on Monday :) I tested with the currently deployed dev instance, not sure whether that used the most recent code.

When clicking on the dataset name link at the top of the dataset settings page (e.g. https://allowdatasetrenaming.webknossos.xyz/datasets/l4_sample-6735ed6c010000d2004849e6/edit), the link https://allowdatasetrenaming.webknossos.xyz/datasets/6735ed6c010000d2004849e6 which makes webknossos crash

daniel-wer · 2024-11-15T09:33:33Z

frontend/javascripts/router.tsx

+                  const { datasetId, datasetName } = getDatasetIdOrNameFromReadableURLPart(
+                    match.params.datasetNameAndId,
+                  );
+                  if (datasetName) {


I don't understand why this is needed (same for line 218). Are there links like /datasets/:datasetName/edit? I thought there are only links like /datasets/:organizationId/:datasetName/edit or /datasets/:datasetName-:datasetId/edit. Could you please clarify?

Kinda yes: There are links like /datasets/:datasetName/view out in the wild. Very old links. We even have a special backend route to get the organization id for a dataset base only on its name to resolve such old links. See: getOrganizationForDataset.

The problem I am trying to solve here is that a routes ``/datasets/:datasetName/viewand `/datasets/:datasetId/view` have the same "matching string" and thus need to be decided by programmatically. I modified `getDatasetIdOrNameFromReadableURLPart` to make some guesswork on whether the passed argument for the match (`match.params.datasetNameAndId`) contains an actual datasetId or just a name. In case it is a name I fall back to treating the legacy link and trying to resolve it to the newest schema.

As this is necessary for view links. there might also be the same problem for .../edit routes. I think it is very unlikely that there are some edit links with there very old URL schema out there. But in case they are out there, this code here takes care of it. I'd say this is not needed but the new return type of getDatasetIdOrNameFromReadableURLPart forces the user to consider legacy links. And I think this is a good thing so it won't be forgotten in the future. Therefore, I'd also keep the handling of legacy /edit links, bc it feels more case complete to me. But feel free to argue.

Besides that, handling the very old /view links is a must :)

frontend/javascripts/router.tsx

frontend/javascripts/test/backend-snapshot-tests/datasets.e2e.ts

daniel-wer · 2024-11-15T09:41:22Z

frontend/javascripts/test/puppeteer/dataset_rendering.screenshot.ts

-          condition,
-          `Dataset with name: "${datasetName}" does not look the same, see ${datasetName}.diff.png for the difference and ${datasetName}.new.png for the new screenshot.`,
-        );
+        t.true(condition, `Could not retrieve datasetId for dataset "${datasetName}".`);


I'm not sure what t.true does if it's called from outside a test 🤔 Should this be an assertion instead or does it work as expected?

- make new fields to reserve upload optional (for backward compatibility) - fix find data request from core backend

MichaelBuessemeyer · 2024-11-15T13:51:26Z

app/models/dataset/WKRemoteDataStoreClient.scala

-    rpc(s"${dataStore.url}/data/datasets/${dataset._id}/layers/$dataLayerName/findData")
+    rpc(s"${dataStore.url}/data/datasets/${dataset._organization}/${dataset.directoryName}/layers/$dataLayerName/findData")


@frcroth thanks for adding the dataset upload test. This already saved me twice in this pr 🙈

=> one place is here, where a request to the dataset was accidentally based on the dataset id and not on orga and the new directoryName

MichaelBuessemeyer · 2024-11-15T13:52:42Z

...-datastore/app/com/scalableminds/webknossos/datastore/services/uploading/UploadService.scala

 case class ReserveUploadInformation(
    uploadId: String, // upload id that was also used in chunk upload (this time without file paths)
    name: String, // dataset name
-    directoryName: String, // dataset directory name
-    newDatasetId: String,
+    directoryName: Option[String], // dataset directory name
+    newDatasetId: Option[String],
    organization: String,
    totalFileCount: Long,
    filePaths: Option[List[String]],


And the other occurence is here, where I accidentally made the new upload info a requirement which would have crashed uploads tried with wklibs

WIP: Adjust schema to allow duplicate dataset names & implement new u…

5ffa251

…ir dataset addressing scheme

MichaelBuessemeyer added backend frontend new feature labels Sep 12, 2024

MichaelBuessemeyer self-assigned this Sep 12, 2024

MichaelBuessemeyer and others added 21 commits September 16, 2024 17:13

reimplement proper dataset name checking route (still keep leave away…

d34bdeb

… the check for already existing name)

WIP: implement wk core backend routes to only use datasetId and no or…

49432a9

…gaId - Includes moving ObjectId to uitls package

WIP: finish using dataset id in wk core backend and dataspath in data…

5d3f8bd

…store & add legacy routes - Undo renaming DataSourceId to LegacyDataSourceId

WIP: Fix backend compilation

0c111cb

Fix backend compilation

d8ad983

WIP: Adapt frontend to new api

10724e7

WIP: adapt frontend to new routes

1a8349a

WIP: Adjust frontend to newest api

6b54a7b

first kinda working version

394c023

Try update schema and evolution

62bbd75

fix evolution & add first version of reversion (needs to be tested)

f74a7f1

fix frontend tests

3a203a8

format backend

a65c7fb

fix dataSets.csv

2234b5b

fix e2e tests

3570545

Merge branch 'master' of github.com:scalableminds/webknossos into all…

cbf4470

…ow-dataset-renaming

format backend

50e918a

fix frontend

90990b5

remove occurences of displayName access / variables in context of a d…

1b0aec3

…ataset object

fixed verion routes

7a79ea4

fix reserveUploadRoute

5443912

fm3 mentioned this pull request Oct 7, 2024

Rename organization_name to organization_id in worker args #8038

Open

Michael Büßemeyer added 3 commits October 8, 2024 11:55

rename orga_name in jobs to orga_id

8224f6a

format code

35d62a9

fix finishUploadRoute

1b6ff5c

Michael Büßemeyer added 2 commits November 11, 2024 11:03

Finish refactoring nml backend parsing

fb88b21

ifx nml typing

78e39a6

MichaelBuessemeyer commented Nov 11, 2024

View reviewed changes

frontend/javascripts/admin/task/task_create_form_view.tsx Show resolved Hide resolved

Merge branch 'master' of github.com:scalableminds/webknossos into all…

17f26b5

…ow-dataset-renaming

MichaelBuessemeyer commented Nov 12, 2024

View reviewed changes

Michael Büßemeyer added 2 commits November 12, 2024 16:08

fix nml upload

a2e13af

Merge branch 'master' of github.com:scalableminds/webknossos into all…

0368b45

…ow-dataset-renaming

MichaelBuessemeyer requested a review from fm3 November 12, 2024 15:29

daniel-wer reviewed Nov 12, 2024

View reviewed changes

apply frontend pr review feedback

d90ab6c

daniel-wer reviewed Nov 13, 2024

View reviewed changes

Michael Büßemeyer added 7 commits November 14, 2024 09:46

apply pr frontend feedback

3c95536

add new e2e test to check dataset disambiguation

519f7b9

re-add backwards compatibility for legacy dataset links without organ…

50331f7

…ization id

change screenshot test dataset id retrieval to be a test.before

9fe0f5b

remove outdated comment

4c8319d

- the backend always sends an id for compacted datasets

Merge branch 'master' of github.com:scalableminds/webknossos into all…

3d6a5e1

…ow-dataset-renaming

temp disable upload test

ae41a2f

MichaelBuessemeyer commented Nov 14, 2024

View reviewed changes

Michael Büßemeyer added 2 commits November 14, 2024 11:28

fix linting

fd04148

fix datasetId expected length

aa8a897

MichaelBuessemeyer commented Nov 14, 2024

View reviewed changes

fm3 reviewed Nov 14, 2024

View reviewed changes

replace failure fox by returnError fox, fix json error msg

fc546d2

daniel-wer reviewed Nov 15, 2024

View reviewed changes

Michael Büßemeyer added 3 commits November 15, 2024 14:34

fix upload test

818878b

- make new fields to reserve upload optional (for backward compatibility) - fix find data request from core backend

remove debug logging from dataset upload test

e3f80fa

remove debug logs

1d5ad73

MichaelBuessemeyer commented Nov 15, 2024

View reviewed changes

Merge branch 'master' into allow-dataset-renaming

b7bf68e

	# changed in v9: Datasets are now identified by their id, not their name. The routes now need to pass a datasets id instead of a name and organization id tuple.
	# changed in v9: Datasets are now identified by their id, not their name. The routes now need to pass a dataset id instead of a name and organization id tuple.

	# Requests to the TracingStore and DatasStore need to address a dataset based on it directoryName and organization id.
	# Requests to the TracingStore and DatasStore need to address a dataset based on its directoryName and organization id.

		// Formatting the dataSourceId to the old format so that the backend can parse it.
		// And removing the datasetId as the datastore cannot use it.

	case class TaskParametersWithDatasetId(taskTypeId: String,
	case class TaskParameters(taskTypeId: String,

		rpc(s"${dataStore.url}/data/datasets/${dataset._id}/layers/$dataLayerName/findData")
		rpc(s"${dataStore.url}/data/datasets/${dataset._organization}/${dataset.directoryName}/layers/$dataLayerName/findData")

Allow renaming datasets & dataset with duplicate names #8075

Are you sure you want to change the base?

Allow renaming datasets & dataset with duplicate names #8075

Conversation

MichaelBuessemeyer commented Sep 12, 2024 • edited Loading

Further Notes:

URL of deployed dev instance (used for testing):

Steps to test:

TODOs:

Issues:

Choose a reason for hiding this comment

daniel-wer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daniel-wer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelBuessemeyer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelBuessemeyer left a comment

Choose a reason for hiding this comment

fm3 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

daniel-wer left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MichaelBuessemeyer commented Sep 12, 2024 •

edited

Loading