Add ability to utilize checkpoints from local run

[ilongin]

User should be able to utilize new checkpoints feature with basic datachain query command (or similar).

If we go with datachain query here is the implementation plan:
Currently this command creates Job instance which has name as the name of the script (without path).
We should change this behavior into:

1. Use full path of the script as Job name
2. Try to find Job with existing name and without child (this means script is re-run) and if found add it as parent to newly created Job
3. Add new flag called --reset (maybe there is a better name) that will not use existing checkpoints, but will run the script as it is ran for the first time

Example usage

datachain query run.py   # some error happens
datachain query run.py   # continues where it stopped on the last run using the checkpoints from the last run if they exist
datachain query run.py --reset  # runs the script from the scratch, not utilizing created checkpoints

[shcheklein]

People don't use query AFAIU, it is not even released at the moment.

Can we somehow do this from a Python itself?

Also we have datachain job run

[ilongin]

datachain job run seems to be only for running jobs in Studio for now .. it has things like team_id which are Studio specific.

Regarding running from Python itself the issue is to create job if doesn't exist at beginning and set Job as complete / failed on the end of the script. There are some options to solve this (e.g atexit hooks), I will try it out.

btw @dmpetrov mentioned somewhere that he wants datachain run my_script.py (equivalent to datachain query my_script) so that's another reason I wanted to bring back that command into the life.

[dmpetrov]

We do need two separate actions: run and run-from-scratch.

Python script run should be the run (with checkpoints). We need to introduce run-from-scratch.

Thus can be done as a command: dc run and dc run --reset. Or just a python script with env variable like: DC_NO_CHECKPOINT=True python myscrip.py

I'd introduce both. Python and dc run seems cleaner. Env variable a bit hacky but does not require dc command.

[shcheklein]

Let's please avoid first introducing CLI back. Let's focus first on Python alone. Tbh, I don't like the whole datachain run for many many reasons:

• it is not public yet and will need work to make it public and it takes already long time
• it is ugly and bad for local experience - you really just expect Python to work
• it overlaps and will confuse people with datachain job run (which one to use when? why?)

etc , etc

[dmpetrov]

do you mean python myscript.py and DC_NO_CHECKPOINT=True python myscript.py for the two actions?

[ilongin]

Ok, I will first focus on Python alone and then we can think about CLI command as second step if needed

[shcheklein]

env var - seems a bit ugly, can it be a session setting of some sort? or some other way people do this in Python in other tools?

[dmpetrov]

Yes, I don't like env (it always workaround). But we do need a way to push this without modifying script. So, it might be ok for now.

DC params might be a better solution but params still require 'dc run'