FEATURE: Simplify marketplace structure

[Sebobo]

Merges source and dist node properties into version nodes and removes them.
This reduces nodes in CR by >60%

To be merged after #608

[Sebobo]

My Schema linter doesn't like the new migrations, have to adjust this

[bwaidelich]

This also overlaps #608 which makes it quite hard to review it. I'll wait until the other one is merged. Alternatively we could stack the depending PRs

[Sebobo]

Sorry, retargeted

[bwaidelich]

Thanks!
I did a first skim over the changes and before getting deeper into it I wonder: What was the issue with the lastSync property?
Or, related to that: Why do we need to introduce a cache and can't compare $packageNode->timings->lastModified?

[Sebobo]

Thanks! I did a first skim over the changes and before getting deeper into it I wonder: What was the issue with the lastSync property? Or, related to that: Why do we need to introduce a cache and can't compare $packageNode->timings->lastModified?

The last sync is an unnecessary write operation to the CR for each package. But we need a custom sync date to update stats (favs, downloads) of packages that haven't been updated by the maintainer in a while. And to skip package if we batch sync.

[bwaidelich]

I understand. But with $packageNode->timings->lastModified we could determine how long a node wasn't updated. Wouldn't that be enough?

I think I got it now: We update the last sync date every time, even if no changes happened..

But I still don't fully understand why we need this check if we make sure that the cronjob is only executed once a day?

[Sebobo]

I understand. But with $packageNode->timings->lastModified we could determine how long a node wasn't updated. Wouldn't that be enough?

I think I got it now: We update the last sync date every time, even if no changes happened..

But I still don't fully understand why we need this check if we make sure that the cronjob is only executed once a day?

The cron job was running hourly until the Neos 9 upgrade and we can let it run hourly or every few hours again after we have all the optimisations done, as the full sync will only takes 10-15 minutes.

[bwaidelich]

The cron job was running hourly until the Neos 9 upgrade and we can let it run hourly or every few hours again after we have all the optimisations done

Makes sense, but again: Can't we just skip nodes that were updated within the last days without introducing a separate storage layer by just looking at the last modification timestamp?

[Sebobo]

We want to quickly sync new releases so they show up in the feeds etc. but don't need to update favs and downloads all the time. So IMO this is still the way to go. The alternative is to find out what would change, but that again would take more time to compute.

[bwaidelich]

OK, I guess I don't fully understand the implications.. But I don't want to block this one and I assume that it improves things!
So let's get this merged and we can always fine tune later if it needs adjustments