Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create files in structure to serialize to disk #24

Open
wants to merge 6 commits into
base: master
Choose a base branch
from

Conversation

maeg02
Copy link

@maeg02 maeg02 commented Mar 8, 2020

DRAFT DRAFT DRAFT

With that said, I have created something that you can review (dismiss), and hopefully give me some pointers if this is the correct setup.

ShuffleRunner would create one of these ShufflerDataSplitter. (Or inject somehow)

This would take the data already exported, as ShuffleBlocks blocks, then recreate them as individual blocks and send them to the already existing shuffle.Serialize.

Could the be something to continue on?

@rappen
Copy link
Member

rappen commented Mar 10, 2020

Looks like it only splits, so it would be a one-way split for version control after export, but no way of combining them back for import?
I think if this gets implemented, there really should be a way to import from that format too, to complete the story.

@rappen
Copy link
Member

rappen commented Mar 20, 2020

Ping @maeg02 - any comment?

@rappen
Copy link
Member

rappen commented Mar 29, 2020

Fun Fact:
Early versions of the Shuffle exported all blocks as individual files, which were finally zipped together with the definition file to a single distributable file.
When importing you selected the zip file, which was unpacked to a temp folder and then imported.

I'm not sure it would be so great to go back to that pattern. If you really want individual files, you can create one definition for each block. On the other hand that would break the functionality build into the Relation element, and id mapping during import does not work across multiple files.

@maeg02
Copy link
Author

maeg02 commented Apr 19, 2020

Still missing merge function...
Would you like to recommend a place to put the split option?

@maeg02
Copy link
Author

maeg02 commented Apr 25, 2020

Now the first version of Merge is in there as well, could this work?

Pointers to consider:
1)
Passing around a Dictionary<string, XmlDocument>, is this a good choice, the first string is really two parts "<block name / Directory name>/<file name, record id>"
Should this string be its own class? If so where to put it, etc
Should this Dictionary be encapsulated to have methods on it like ToString or something?

I have removed Tuple in favour of "( )" return values, these can have names and that is better than Item1, Item2, Item3 etc.
Keep or remove?

What do you think?

@maeg02
Copy link
Author

maeg02 commented Jun 2, 2020

Poking this thread, to see if we can get it moving again

@rappen
Copy link
Member

rappen commented Jun 4, 2020

I'd love to have a look at this, but been busy for quite a while 😕

@maeg02
Copy link
Author

maeg02 commented Jul 2, 2020

Last try, any feedback?
Regards Markus

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants