Iteratively adding and removing files before committing

[quinnwai]

Description

based on slack thread

When doing a g3t add -> g3t meta init on an existing project, the user expected to have 1 document reference, but found 2. Ideally, a user should be able know what files exist in the MANIFEST and remove any particular file from the file manifest (MANIFEST) and update the metadata (META).

Without considering our current g3t first, one optimal path for adding files could look like

1. Try to add files (add command)
2. See what files are available (ls command)
3. Remove any files that are unneeded (rm command)
4. Create metadata for files (metadata command)

Current Issues related to This

Below are a list of bugs associated with this use case. These are listed not just so they can be patched, but also so they can be further analyzed / root caused according to a user's perspective

• g3t rm only works after you do a g3t commit on the file. A user should not need to do a commit before doing their round of g3t adds and g3t meta inits to figure out what files to commit.
• g3t rm does not stage the removed file, so doing things like g3t ls to check your files are right before committing will error: "[Errno 2] No such file or directory: 'MANIFEST/bonk.txt.dvc'". A git add MANIFEST/ resolves this problem.
  • General point: when do we automatically commit file changes for people (eg currently g3t add) vs when do we make them commit it themselves (eg currently g3t rm)?
• after removing a file that has existing metadata, g3t meta init results in exceptions=[ParseResult(resource=Bundle(resource_ty... which corresponds to creating the Bundle.ndjson object if deletions / edits of existing records occur. What is the status of the prototyped "FHIR-server" that is supposed to handle Bundle submission, as we haven't done much / at all testing of this?
• g3t ls is long form by default, it should just be a list of files, see The file ls command format is weird #3
• g3t ls only shows the committed files in the repo. Is this expected behavior? If so, more documentation is needed to explain this.
• Documentation does not exist / is limited for g3t rm, g3t ls, and how it fits into an upload use case.