File-based version #360
Replies: 2 comments 7 replies
-
|
Hi @gpetretto, I tried to set up the montydb version. I run into the following error: I am very sure I installed the gh/pipelines branch and not the one for sqlite. I also updated to the latest montydb. The project check also runs without any errors. However, when I try to start the runner I get the error that the database is locked. When I try to submit a new job, I get the error above. Do you see a reason why this could happen? Thanks in advance. |
Beta Was this translation helpful? Give feedback.
-
|
I was able to setup the sqlite version with a local worker and it seems to start simple additions. I had to install sqlite itself, of course, and also "sqlalchemy" I ran into to errors: I cannot run "jf -p my_project project check --errors". This fails with an error. And, the DOWNLOADED job cannot be added to the MontyStoreDB. It looks like the same error as above. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
As already discussed in #338, I have been trying to create a version that would allow users to try running workflows without having to install a database. Of course the idea would be that such a version should be used only to run a few workflows or to test the software without the burden of the complicated setup. It is unlikely that such a version would allow to run in high throughput or produce large output databases, so should be discouraged for production.
At this point I have prepared two versions, one where it is possible to disable the usage of the MongoDB pipelines. This allows to use MontyDB as a backend for the queue database, the other entirely replaces the orginal JobController with one based on SQL and thus can be based with SQLite.
While I believe that the a version based on SQLite would have the potential to be more robust and support larger DB sizes, this test version was mostly implemented through vibe coding and given that the task was quite involved I believe that the code needs extensive review. This will likely be a major task, so I suppose that the current version with some minor improvements would be the best I could afford to provide.
On the other hand the version without pipelines gives up on the atomicity of the operations in some cases, on top of the fact that also the implementation in MontyDB will not guarantee it. This version may be more prone to database corruption if multiple actions are perfomed simultaneously on the DB, but the implementation was way easier and could probably be merged after a standard short review. So I will start from the latter.
No pipelines version
To use this version install the code from the
gp/pipelinesbranch: https://github.com/Matgenix/jobflow-remote/tree/gp/pipelines and upgrade MontyDB to the latest version (2.5.5) where I have implemented thefind_one_and_updatemethod.For the configuration of the stores you can set them up like this:
The rest of the configuration is the standard one.
Note that here jobflow-remote still forces to define a
datastore for atomate2, so maybe switching to a warning as discussed in #331 would be better.My idea would be that this could be set as the default store if
queueandjobstoreare missing from the configuration and use the standard project folder to save the files. In this was a new user could test this with a minimal configuration.SQL JobController
To use this version install the code from the
gp/sqlbranch: https://github.com/Matgenix/jobflow-remote/tree/gp/sqland for the configuration use:
optionally a
filepathinqueue.storecan be set to decide the path of the sqlite db file. By default it is stored in the project folder.Let me know if you have the chance to test it.
Pinging @JaGeo @Andrew-S-Rosen and @utf as they may be interested.
Beta Was this translation helpful? Give feedback.
All reactions