Independently Import Packages

[curtisma]

Problem Statement

Currently, all the packages share a common IMPORT table so all packages must use the same version of a library. This makes it harder to make updates to each package since it can be hard to see what depends on it.

Proposed Solution

Each package should be able to define the version of the libraries it needs. These will then be saved to a IMPORT table contained in the package. When a package is loaded it will select its IMPORT table rather than a global import table. The package import table will need to be loaded first, with each package either coming from the global "PACKAGES" table or it should be loaded into the PACKAGES table if that version is not already available.

Alternatives Considered

Proposed Action Plan

• 1. we'll do this
• 2. We'll do that
• 3. Then we'll add tests for this and that.

[tbrandtner]

Hi Curtis, now it gets really interesting!
I have been thinking about a good solution for this feature - let me share some of my thoughts:

Solution 1: Package defines version of libraries (which may be shipped as part of other packages as well)
This is somewhat yet another level of dependencies. This could be stored in a "manifest file" which could be a toml file again. Or maybe we could even store that inside the already existing pyproject.toml in another section like "virtue.skill.dependencies" or so.
Biggest challenge I see: How do you convince the surrounding Python vent that you want to install different versions of Python packages (with the Skill packages/libs inside)? To my understanding this is not supported - at least not natively with pip or conda.
One solution could be to work with several venvs in parallel, which brings me to another idea:

Solution 2: Parallel venv (which define the package/library version inside)
This means: All packages inside one venv share the same version of a Skill library - but Virtue can handle several venv in parallel - and all those venv are also independent from each other in the Skill "library space".
You could use the normal dependency definitions within pyproject.toml. pip/conda will resolve them for each venv individually - and it will also tell you if it cannot be resolved. This is one disadvantage: You are not completely free anymore within one venv. But you can reuse all features of pip/conda.

I personally tend more towards solution 2. Technically you could solve it by having an internal table with 2 keys: One key for the library name (like today) - and another one for the venv ID. The venv ID could be stored in a Skill variable while you are loading the source code. And inside the Skill source code you use a macro like "getCurrentVirtueVenvID" which evaluates directly in the current venv ID defined above - already during load time. That way the venv ID is stored already as fixed string in Skill code after load - and you can use this venv ID as a key to the internal table.
I can also imagine already a way to do it similarly if we want to "compile" a complete venv into a context file.

OK - let's continue the discussion and try to find an optimum solution for this feature which should make it into the Virtue environment!

[curtisma]

Hey Thomas!
Yeah, definitely.

I was thinking along the lines of Solution 1 actually. The main reason was that I wanted a solution that would work both as part of a conda/venv environment and can be built into a stand-alone skill-only package. The python package would be used within a development environment and for advanced users that setup their own environment. The stand-alone skill-only package would be used in production. Many production environments will only support SKILL and running conda / venv in production doesn't work well.

I think I need to make a module that does the build part of it. It should also generate a context file to make it quick to load.