[core] Delay chunk evaluation

[Alxiice]

Description

This PR goal is to change chunk system to be able to work with graphs where nodes can have no specific size set.

Changes

Compute system

• New levels have been added on the computation :
  • extreme (that targets higher specs that what intensive does)
  • script which is a mode for single-process simple process. It targets specific machines on the farm that have the ability to run multiple jobs in parallel so for simple jobs it should allow faster processing
• Delayed chunks creation
  • node
    • A _chunksCreated param has been added to check if the chunks have been correctly initialized
    • Now by default we use resetChunks when loading meshroom, that will create a list with a single chunk, and we can create chunks at any moment with _createChunks.
    • A new NodeStatusData has been added that tracks a nodeStatus file which is similar to the chunk statuses files but is specific to the node. This node status is used to get the cached range parametrization of the node so that we can retrieve it when possible with _createChunksFromCache.
  • taskManager
    • The TaskThread and TaskManager have been modified to launch the chunk creation when the node compute starts

Submitters

• Submitters have been moved to another package (TBD where...)
• The BaseSubmitter API have been updated to allow more flexibility. A new BaseSubmittedJob exists and is used to track jobs that have been created. The goal is to use this as an interface to call actions on it (stop/pause/restart...)
• A new bin/meshroom_createChunks script have been created, and handles the chunk creation and additional chunk tasks spooling
  • Calls the chunk creation
  • Checks if we can spool additioanl tasks
  • If so, execute queue tasks that will compute the chunks
  • If not, execute the chunks serially on the current process

Examples

[codecov]

Codecov Report

❌ Patch coverage is 42.00000% with 203 lines in your changes missing coverage. Please review.
✅ Project coverage is 79.02%. Comparing base (7cac2bf) to head (9c2007d).

Files with missing lines	Patch %	Lines
meshroom/core/node.py	41.04%	135 Missing ⚠️
meshroom/core/submitter.py	40.35%	68 Missing ⚠️

❌ Your patch check has failed because the patch coverage (42.00%) is below the target coverage (70.00%). You can increase the patch coverage or adjust the target coverage.

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #2918      +/-   ##
===========================================
- Coverage    80.61%   79.02%   -1.59%     
===========================================
  Files           55       55              
  Lines         7692     8015     +323     
===========================================
+ Hits          6201     6334     +133     
- Misses        1491     1681     +190     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.