Fixes to Modules

[ctkjose]

This is just a draft of a PR for fixing some standing issues with module resolution in V. Just trying to get some opinions before I make the PR.

Module handling and Source Code Organization

There are some grey areas and inconsistencies when it comes to the behavior of modules/packages and v.mod files. We still have many notes in the source code regarding this issue.

The vlib is an excellent source of examples of the intention and uses cases for modules in the language.

Lets start with nothing in vlib uses v.mod files. Thats ok because formally v.mod files are used for packages in vpm.

What is a module

From the documentation a module is just a folder with *.v files. A module name as used with the module keyword is the name of the immediate folder, they must use snake_case and no qualified names like "a.b.c" are allowed.

What is a package

For a package we state that it can contain one or more V modules and that a package should have a v.mod file at its top folder describing the contents of the package. In practicality a package is just a module with a v.mod file managed by VPM.

Search Locations

Per documentation the option -path is the official means to set the pref.lookup_path[] array. The full description of this option is:

Specify the order of path V looks up in while searching for imported dependencies, separated by pipes (|). In addition to absolute paths, you can also use these special strings too:

• @vmodules replaced with the location of the global ~/.vmodules/ folder (modules installed with v install are there). You can change its location by setting the environment variable VMODULES.
• @vlib replaced with the location of the v's vlib folder. Using these, you can arrange for very flexible search orders for you project, for example: -path "/v/my_project_private_modules|@vlib|@vmodules"

By default, -path is just "@vlib|@vmodules" .

pref.exclude[] is not used anyware?

In code @vlib is pref.vlib and @vmodules is os.vmodules_paths() which in turn uses getenv('VMODULES') or <HOMDIR>/.vmodules. As expected multiple paths are allowed in $VMODULES using : on unix or ; on windows.

The field pref.path has the path as entered by the user in run or crun. It could be '-' or empty, else was already checked with os.exists.

The Builder.compiled_dir is the projects root path being compiled initialized from pref.path. In the parser everything is atomic, we only know the file's path and have a access to the pref instance.

For sake of simplifying this document we will use <rel_to_file> and <rel_to_dir> to refer to the respective paths of a file being parsed.

The Builder uses module_search_paths set by Builder.set_module_lookup_paths(). Here it starts with Builder.compiled_dir, then if found we add $compiled_dir/src/modules and $compiled_dir/modules. Finally it adds pref.lookup_path[].

In Builder.parse_imports() we use Builder.find_module_path() to resolve a Qualified Module Name (QMN) to a path. This one is more involved. Here the search order is as follows:

• Search the module folder of <rel_to_file>. It uses vmod to find a module folder for <rel_to_file>, if found it will search that folder.
• Adds Builder.module_search_paths to the search paths.
• Adds os.getwd() also.
• If <rel_to_file> contains a modules segment, it will search that modules folder.
• The QMN is converted to a path replacing dots with path delimiter.
• It walks these search paths and concatenates the QMN path, if the folder exists it returns the path.

The parser also has two function to compute a Qualified Module Name (QMN), qualify_import() and qualify_module() that need to resolves names to paths. We will discuss those separate given the atomic nature/scope of the parser. But the gist is that qualify_import() searches pref.lookup_path, then it also forces os.vmodules_paths(), lastly it will use <rel_to_dir>. In qualify_module() we start checking relative to os.getwd(), else it uses <rel_to_dir> since we assume all files in a module are in a single folder.

We have ./project/src, ./project/src/modules and ./project/modules that are treated as special paths. Many VPM packages use this structure. These are presumed to be inspired by Go. Prior to Go 1.11 Go relied $GOPATH env variable to define a workspace that had the GOPATH/src and GOPATH/pkg folders.

To this point we can state that in V:

• Users can modify search order.
• Users can override default locations, thus shadowing @vlib and global modules.
• Users can add n number of paths.

Qualified Module Name

The Qualified Module Name (QMN) is a more specific form (unambiguous) of qualifiers for a module name.

For a module in VLIB, the QMN is always is made up of all folder's name prior to vlib/. For files in .vmodules/ is similar but we stop if a v.mod file is found. In VPM we also stop at '.git', '.hg', '.svn' and '.v.mod.stop'.

A QMN is computed by the functions qualify_import() and qualify_module(). These functions are discussed in more detail later. These functions are used during the atomic parse process; they have no context, state, or reference to the Builder, particularly the project root (Builder.compiled_dir). In lieu of Builder.compiled_dir, we use os.getwd() in qualify_module() and assume that the working directory is the project root.

Module Import

The QMN assigned to an import is computed at parser.v:import_stmt() using qualify_import(). qualify_import() in turn uses mod_path_to_full_name(). In qualify_import we search pref.lookup_path, then it also forces os.vmodules_paths(), lastly it will use <rel_to_dir>.

The mod_path_to_full_name() has a few quirks that we will look at later. Here we build a QMN by walking the folder matched in qualify_import() (or qualify_module()) backwards and stoping when the base name of one of the paths in pref.lookup_path is found or when a v.mod file is found.

In code we ALWAYS have to use the module alias. The module alias is the last segment of a QMN or the name set with as.

In parser.v:name_expr() imported symbols get the QMN as their scope.

Module Statement

Here the rules are simple. A module name is the folder name using snake_case, and no qualified names.

At parser.module_decl() we use qualify_module() to establish the module's QMN. Here we have the advantage that we made the rule that module a must be inside a folder with the same name.

giving the implications of the isolation of the parser these functions m

Issues

Invalid QMNs on import statements

In qualify_import(), the QMN is resolved relative to the file performing the import. This means that a QMN for a module outside vlib/ and .vmodules/ may differ depending on where it is imported from. As a result, the same module can have different QMNs in different locations. When the QMN determined during import does not match the QMN resolved by qualify_module(), it leads to unknown symbol errors.

Invalid QMNs on module declarations

In qualify_module() we use the os.getwd() which may be different than Builder.compiled_dir. This means that the QMN for a module outside vlib/ and .vmodules/ will be different QMN than the import.

Ambiguities when resolving a modules path

When reading and parsing the imported files in Builder.parse_imports(), the files read used for the AST and builder's symbol table may NOT be the same as the one resolved by the parser in qualify_import() and qualify_module().

Using os.getwd() may lead to the wrong file being read at `Builder.parse_imports().

Modules in <compiled_dir>/modules, <compiled_dir>/scr/modules are never resolved in qualify_import(). Imported symbols are scoped to the QMN the user wrote. It is up to Builder.parse_imports() to resolve the path. In qualify_modules() they are resolved simply because it found the base name "modules". If there is more than one modules segment in the path it would stop at the last one which also matches the behavior of Builder.find_module_path().

In qualify_modules() it uses the base names (like "vlib", ".vmodules" and "modules") of key search paths instead of absolute paths this may leads to discrepancies in the QMN generated. For example the look_path '/my_modules/math' will add math as a stop folder when we need to resolve prj/modules/ai/math/vector.

Modules in vlib/ and .vmodules/ are treated consistently in all the scenarios tested.

VPM logic will only be used at builder.parse_imports() which will use .git among other as a stop folder.

We generally interpret the special folder modules/ as <compiled_dir>/modules but in reality is always <rel_to_dir>/modules at the parser and then builder.parse_imports() will instead prioritize relative to <compiled_dir>.

Right now mod_path_to_full_name() ignores v.mod files and continues searching using the last one. The reason reads currently CI clones some modules into the v repo to test, the condition after 'v.mod' in ls can be removed once a proper solution is added. This is one of the reasons nested folder break in project space.

Broken Nested modules on project side

Nested project modules loaded and parsed by Builder.parse_imports() nevertheless the discrepancy in QMN doesn't allow for symbols to be resolved. Playing with the placement of the v.mod file on can get all the QMS to match and the compilation to succeed.

This is really frustrating because you at-least expect for patterns that work on vlib/ to work in the project space.

Discussion of proposed solution

The solution follows the current status-quo of not requiring formal structures and code organization while supporting organic and intuitive code organization.

The key source of use cases for modules is our own vlib, it provides actual patterns that simple and work. Sadly thats not the case in the project-space.

We limit ourself to providing a predictable and consistent way to resolve a module location and its QMN initially in the builder and parser.

Revisiting the definition of modules in V

Conceptually a module is a logical aggregate of source code with a shared module name (namespace).

A module is just a folder with *.v files. All files within that folder must declare the same module name using the module mod_name statement.

Modules can be organized intuitively using folders like /google/api,/google/services/directory, /google/services/gmail. This is a very natural and organic approach to managing code and is used extensibly in @vlib.

The rules to organize modules in folder are as follows:

• The source code files *.v of the module are organized together in one single folder for the module.
• All the *.v files in the module's folder must have the same module name. The name of a module is short and concise. It must follow the snake_case naming convention.
• The folder name must be exactly the same as the module name. There are no restrictions for files names inside the folder.
• Any folder with *.v files is consider a module. We can have folders strictly for the purpose of organizing the modules themselves without the need to have code in them. For examples './google/services' are plain folders without code and inside we could have './google/services/directory' and './google/services/gmail' as modules with code.
• Adding a v.mod file to a module, turns the module into a VPM package. The v.mod file is used to provide information to VPM.
• Using an empty v.mod creates a dummy package that can not be used with VPM, it is just a regular module..
• The presence of a v.mod will be used to anchor the qualified module name (QMN) (defines the module boundary).
• If module is a VPM package, then the root folder (ie: google/) is the package folder which is also the repo and module name. VPM will continue to treat package as a single unit, regardless of its folder hierarchy and sub-modules.

Resolving import names:

The module source name is the name specified with the import keyword.

The alias (mod.shortname) is the las segment of the import name or as specified with the as keyword. The alias is the name used in code.

The module source name is converted to a relative path ( . replaced with os.path_separator).

A root folder is a location where we spect to find modules. These are:

• Paths in pref.lookup_path which may include @vlib, @vmodules but not necessary. The bulk of the imports will be from here. Should we force vlib/?
• Project folder (folder of what we are compiling, like in v run ~/myproject then ~/myproject is the project folder.
• A modules/ folder in the project.
• In the folder of the file doing the import.
• In a modules/ folder of the file doing the import.

In addition to v.mod, VPM adds '.git', '.hg', '.svn' as indicators of a module's boundary. These were not added. We feel that using v.mod is a simple solution when need to anchor to another path that is not one of the root folders. Also we should discuss the uses cases for .v.mod.stop if we want to add it.

The resolution starts with paths in pref.lookup_path which may include @vlib, @vmodules but not necessary. The bulk of the imports will be from here. Should we force vlib/?

Modules can be siblings in the parent folder of the file being parsed. This is here for vlib/v/type_resolver where we find import ast instead of import v.ast. Should we allow this?.

Lastly we search relative to the project folder (if pref.path is a valid folder, else we use os.getwd() in lieu of the project folder).

The QMN (qualified module name) is always anchored to a root folder. The qualified module name will not extend beyond the anchor boundary. Like existing functions we walk the folder backwards until we reach a stop location.

When the resolution occurs in the context of a module name statement (module_decl()) we take some shortcuts since we know that module a can only occur inside folder a, we are already inside the module's folder.

Code Changes

resolve_module function

The resolve_module() function in vlib/v/util/module.v attempts to provide a simple and reliable source of information for modules.

It addressed various concerns and comments left in the code:

• Many places implementing the module location / v.mod rules
• qualify_import() fails to resolve almost everything.
• Use absolute paths for stop locations.
• Standing Todo: TODO: try to merge this & util.module functions to create a reliable multi use function. see comments in util/module.v.
• Expands the functionality of ModFileCacher but using a much simpler code (IMHO). In particular we no longer need to worry about search paths.

The function resolve_module() returns three strings, qualified_mod_name, mod_directory, mod_vmod_file. It has the same parameters as qualify_import()/qualify_module() but added a third boolean to indicate if we are resolving in the context of a module statement.

The function implements a simple cache using the resolved path.

Use -v to output basic info about the resolution. Compile with -d debug_mod_resolve to see more info.

Other changes

The parser.import_stmt() and parser.module_decl() were modified to use resolve_module(), replacing call to qualify_import()/qualify_module(). Modified to reduce code and num of ast.Import created.

The function builder.parse_imports() was modified to use resolve_module() instead of builder.find_module_path().

the flag -d trace_util_qualify is no longer used.

To discuss

Notes

ToDo

Support pref.exclude[] (expand_exclude_paths()).

Discuss supporting Go's internal concept. Discussion

Rescue and investigate daa5be4. What was the issue? seems related to using os.path_separator in Windows.

$if windows {
	m_skip_files << 'vlib/v/checker/tests/modules/deprecated_module'
}

[JalonSolov]

"Lets start with nothing in vlib uses v.mod files. Thats ok because formally v.mod files are used for packages in vpm."

This is incorrect. If you look, there is a v.mod file at the root of the V repo, which is used as the anchor point for finding all the modules under vlib. All of those modules are plain modules, the same as module you might find on vpm - the only difference is that they are all included as part of V instead of separate modules that need to be downloaded.

This is all by design, to keep V modular, and to make the modules under vlib act like any other modules.

[ctkjose]

Thanks. I think that particular v.mod is there just to stop the search in case it leaks out or it is an artifact. So far in my interpretation of the code, modules in vlib/ or in any of the standard search paths are resolved differently compared to modules in the project folder. When it walks back it will stop at vlib.

Assuming I didn't miss it, in our documentation we only officially mention v.mod for packages, modules are just a folder with v files. That matches Go current state, just with the inverse terms.

Im documenting modules here and while learning and testing (bringing all the bad habits ) I found that where things don't work as expected is for different hierarchies of modules local to the project. So far in my testing only prj/modules/mod1, prj/scr/modules/mod1andprj/mod1` work reliably, any creativity organizing things in folders requires you to either put a v.mod in the root of your prj (this works just because doing so just happens to make both functions generate the same qualifier).

I feel that just saying put it there to act as an anchor is a half-ass answer, specially when we have to answer why it works there and not here, why I cant put the anchor here. We need a formal answer as to what is the v.mod for and how it is used.

I think that following the mantra of simplicity and pragmatism you should be able to organize your code as it comes natural to you and V should be able to find it. I love the whole put files in a folder and you have a module. We are almost there! so my changes make modules behave the same way anywhere, without having to worry about a v.mod.

[ctkjose]

Revised code is working, all test passing. Im adding new tests.

ctk@EXWMBPROM301 nv % ./w vlib/v/compiler_errors_test.v
OK [1285/1395] 79.965 ms vlib/v/checker/tests/unused_import_err.vv
SKIP [1286/1395] 0.000 ms vlib/v/checker/tests/unused_param.vv
OK [1395/1395] 892.720 ms vlib/v/checker/tests/run/using_comptime_d_value.vv

Summary for all tests: 1394 passed, 1 skipped, 1395 total. Elapsed time: 25046 ms.
OK [8/8] 336.527 ms vlib/v/checker/tests/custom_comptime_define_if_debug.vv

Summary for comptime define tests: 8 passed, 8 total. Elapsed time: 2848 ms.
ctk@EXWMBPROM301 nv %

[ctkjose]

Code was pushed to my repo:
https://github.com/ctkjose/vlang/tree/fix_mod_resolve

[ctkjose]

This fixes this issue 24430

[jorgeluismireles]

Very comprehensive your text about modules. Something to study carefully.

Even though I think few is documented or nothing at all about the fact (taken from Golang) that prevents importing cycles (not a problem in Java as far as I recall). So when the number of modules start to grow one need a good plan to prevent biting own's tail.
Example:

builder error: error: import cycle detected between the following modules: 
 * main -> company.lines -> company.all -> company.all.cfg -> company.all

[ctkjose]

Saludos Jorge!

We are working on a community book here. I started documenting modules and that's when I found the current issues.

[ctkjose]

Adding this here for ref.
module: fix bug with submodule import #10318 PR

[spytheman]

There are several places, where you ask "Should we force vlib/?" .
Can you please expand on what do you mean by forcing it?

[spytheman]

Modules can be siblings in the parent folder of the file being parsed. This is here for vlib/v/type_resolver where we find import ast instead of import v.ast. Should we allow this?.

No. Using v.ast everywhere is more consistent.

[spytheman]

My opinion, as I mentioned in the comments in vlib/v/util/module.v, the whole file vlib/v/util/module.v should not exist. The logic there is a very convoluted in my view.
I also think that using the current working folder at all, outside of v.pref, is a mistake, since it makes the behavior of the compiler harder to predict, and IDEs/editors can not be relied to set it consistently, with results that are hard to diagnose, by just the output.

[spytheman]

Lastly we search relative to the project folder (if pref.path is a valid folder, else we use os.getwd() in lieu of the project folder).

If pref.path is a file, instead of a folder, then os.dir(pref.path) should be used instead of the project folder (for relative lookups). Using os.getwd() will give inconsistent results, when you do v /abs/path/file.v, when the current folder is say ~/ vs /tmp/ vs ~/.vmodules/ .

[spytheman]

Another test, that you can do is:

1. v install ui
2. cd ~
3. v ~/.vmodules/ui/examples/users.v ### should pass
4. cd ~/.vmodules
5. v ~/.vmodules/ui/examples/users.v ### should pass
6. cd ~/.vmodules/ui
7. v ~/.vmodules/ui/examples/users.v ### should pass, but it does not on master
8. cd ~/.vmodules/ui/examples
9. v ~/.vmodules/ui/examples/users.v ### should pass

[spytheman]

Rescue and investigate daa5be4. What was the issue? seems related to using os.path_separator in Windows.

If I remember correctly, yes, it was related to the / vs \ - the output of vfmt oscillated on windows. It may no longer be the case now.

[ctkjose]

I need further guidance about the modules folder

If we say: Inside a project folder we can organize modules inside a folder named modules/. The user can also reference modules relative to the modules/ folder. For example in a project file we have import a where a is in /prj/moduels/a.

Then we don't make distinctions between a project and a module. A project is just the main module being build. Therefore a module can contain a modules/ folder. But then we could have these scenarios:

import a.b where b is in .vmodules/a/modules/b.

import a.b.c.d where d is in .vmodules/a/modules/b/c/modules/d or any other combination...

Do we have further guidance on the use of the modules/ folder?

[ctkjose]

@spytheman, should I re-enable looking in the parent folder or fix the cases were I find them?

I modified the logic so we will not resolve siblings, @spytheman and I agree that is not a proper use case of the "import" statement.

I call resolving siblings when we have the following:

./mod1/
    |-- mod1.v
./mod2/
    |-- mod2.v

The in mod1.v we do import mod2, to find mod2 we have to go to the parent of the file currently being parse and search there. For example:

• In vlib/v/type_resolver we find import ast instead of import v.ast (already changed to v.ast).
• In cmd/tools/vdoc/document/doc_test.v we have import document, but the document/ folder is the module. In this case we are executing a program doc_test.v from inside a module. In this case I simply think we need to structure our tests better, doc_test.v have no reason to be inside the module.

As a side project we could do some clean-up of our tests:

• add a tests folder per tools and organize their tests there
• add a global tests folder in the root of the project and organize tests there by folders like compiler/,language/,vlib/, and tools/.

[antono2]

@ctkjose
As a user I was surprised when creating the first module, because there were special directories.
Usually a user can place his files in any dir or all his libraries into one file.
It's obvious there needs to be some structure to limit the chaos.

• So, you have to put the module name in the file it belongs to and can not mix modules into one file.

• Next, all libs might be thrown into the same directory, which is solved by one directory per module. The name of the directory should not matter, because the module name is in each file.

• From there I was surprised that users also have to put their files into another special directory modules and some v files can not go in there. If they are in the main module, they have to go into src, another special directory. (At least that's how I remember it)

This is to give a perspective. As a user I would see any kind of special requirement as limiting my project, other than mandated by the need for a unique libray path/identifier for a package manager or some feature of the language, like v.mod.

I like the approach here, but can't add to the details.

[JalonSolov]

The modules directory was added to V very early. I have argued many times that it should be removed as a special directory name. The same for when src was added... I argued against it, and the decision has now be made to remove src as anything special.

[ctkjose]

Im a bit frustrated, sadly as is, I would not even call what we have modules, is just a search that stops at one of the roots (.vmodules/, vlib/) or whenever v.mod is found. @JalonSolov mention to me many times v.mod is just an anchor, sadly that's all it is.

I still have 26 failed tests on test_self. Most are just the no matching outputs, but with others I would have to replicate the old behavior, and I honestly think we should fix the tests instead.

Right now @VMODROOT will be substituted with the folder where the nearest v.mod file is (as a string). Im ok with that definition in the context of modules, but v.mod is being used all over the place for things that are clearly not packages.

For me a module is the organizational unit of code. The main module is the executable or library getting build. A module is always a folder. Just like go/rust and others a package is module meant to be reused in other projects, be internal or over VPM and thus v.mod should only exist in the scope of a package. The v.mod files holds the information about the package and its dependencies.

But then we have things like this cmd/tools/vdoc/document/doc_private_fn_test.v. This is a test that declares itself part of the document module. But then they put a v.mod in cmd/tools/vdoc/ which by no means is a module or a package.

Since the old code will just walk eternally until it finds a v.mod thats ok, right now v.mod is just a marker of a root for whatever intentions.

In the new code we scope the search of v.mod. In .vmodules/, vlib/ we dont search pass the root folder of the module. In the project space we do not look pass the project folder.

Executing cmd/tools/vdoc/document/doc_private_fn_test.v tells me that cmd/tools/vdoc/document/ is the project. The folder vdoc has no relationship to you.

If we want a means to mark what is a project then we should come up with something else, but right now we have so many ambiguities.

Sadly I dont now how to proceed, we need a discussion and consensus.

[antono2]

The tests don't matter in how to proceed, I would say.
Decide for a good structure. The one above seems good to me. Then adapt the tests as needed.
@VMODROOT could still be used as is; like an anchor in some parent dir. If it's missing information that is required to be an organizational unit of code, the v install tool or whatever needs that information could show an error about it not being a valid package.

Also, adding something like
-L@FILEDIR/../lib and pub const some_parent_dir = "@FILEDIR../.."
would remove the use of @VMODROOT in tools or where it's not an organizational unit of code.

v.mod can then be used for all the things needed to describe a package, which can be installed like v install [globalid/url].
Also, for each new v.mod with v new I would generate a globally unique identifier, that can be used to distinguish between things with the same name.

import same_name also has to distinguish between duplicates, so there must be a way to set an import path/guid in import and in v.mod dependencies. There is the problem with GUIDs, that they are not human readable, but maybe fmt can help with a commented line describing the imported module.

So that's basically my vote

[Wajinn]

Of the opinion that making the way V has done modules into something else, would not be helpful. Kind of like if someone was familiar with how Python does things, then tries to bring that over to V, where they get frustrated that V is not Python versus being more open-minded that it is something different and proceed from there. In the reverse, would be the same if bringing V or Python ways into other languages.

The other aspect is that I always thought of V's way of modules was simple and flexible. For someone new to programming, would see it as even more intuitive to use, in terms of them creating their own project folders. They could do v new basics (put in desired directory). Then from the command line (fill info for v.mod).

But they don't have do that and can just make their own folder (and their own v.mod), put .v files in there, and if doing so can make the module name be the same as the folder name. Thus module basics is in the folder \basics. This can create less confusion, because the module being the same as the folder name, means it's very clear what going on. If you misplace files of that module name, it's quite clear what folders to put them back in.

Maybe points of confusion is that the file names (freaky.v) can be anything, but you can have the module name (module basics) matching the folder name (basics). The other point of concern, is having many modules named "main", with file name of main.v, in the src folder. This is a simple structure to understand, but the programmer needs to be organized to prevent confusing themselves. I can see arguments for and against, and this will be the case for any other system, but think it comes down to just being organized, as there are easy ways to avoid that (as shown earlier).

As for, .vmodules, we can choose to use or not, and it can be seen as the standard location for downloading packages. Particularly when using VPM.

[JalonSolov]

v.mod is more than just an anchor point for finding modules, or at least it can be.

It can also identify your module, the version, etc.

And if you actually fill out the dependencies: block, you can get all necessary modules to import just by typing v install when the current directory has the v.mod file.

It could possibly be used for more, in the future, those are just the features so far.

[ctkjose]

@Wajinn I totally agree and I love how easy is to align V modules with your personal intuition to organize your code. The issue that started this fix is that sadly modules are broken on the user projects.

The idea here is just to fine tune the current module implementation, check the first entry of this discussion where I trow some ideas of what a more formal definition of modules and basic rules the code could follow.

Internally V has four different functions (or five if you include the one in vdoc) related to resolving: the qualified name of a module, the folder of a module and the v.mod file of a module. There are various issues (apart from inefficiencies) with these functions, the main ones being:

• The qualified name of a module is not always the same, this is the main bug that makes sub modules not work on projects.
• v.mod is used for two different things as an anchor and as a module definition file for VPM. Perhaps on of the reasons we depend so much on v.mod as an anchor is how the current logic resolves relative to the file being parsed.
• The files parsed/compiled may not be the ones you intended to use.

The current fix doesn't change how modules work, the folder structure is arbitrary with the exception that module <name> must be in a folder with the same name. The real change is on how we anchor the search for a module.

[spytheman]

@VMODROOT is already documented as:

- `@VMODROOT` => will be substituted with the *folder*,
  where the nearest v.mod file is (as a string).

It has no relation to module lookup bugs and the fully qualified module name mismatches.
Its expansion is well defined, and there is no need to change it.

[spytheman]

@spytheman, should I re-enable looking in the parent folder or fix the cases were I find them?

I can not give you a general answer, without considering each case separately.

...

cmd/tools/vdoc/document/doc_test.v

doc_test.v can be changed if needed, in order to import the document module in whatever way will work. It currently does import document as doc, but could also import cmd.tools.vdoc.document as doc or whatever other scheme is suitable.

[spytheman]

As a side project we could do some clean-up of our tests:

imho that could be done separately, to avoid losing the focus of the discussion.

[spytheman]

-L@FILEDIR/../lib

what will that expand to?

[antono2]

Current file directory

~/.vmodules/mymodule/lib
~/.vmodules/mymodule/lib/mylib.a
~/.vmodules/mymodule/anothermodule/filethatneedslib.c.v

#flag -L@FILEDIR/../lib -> #flag -L/home/user/.vmodules/mymodule/anothermodule/../lib
#flag -l :mylib.a

[ctkjose]

I think is appropriate to maybe reset the discussion and try to conceptually define modules in V. I leave this here as a starting point...

Edits

4/JUN/2025, Edited to include @spytheman comments

Lets revisit modules in V

The main purpose of modules are to:

• allow intuitive organization and structure of code
• allow for organic means to reuse code
  VPM is an added layer outside the modules infrastructure that facilitates using others people's modules among other features.

Conceptually a module is a logical aggregate of source code with a shared module name that becomes a namespace.

A module is just a folder with *.v files. The folder is named after the module it represents. The name of a module is short, concise and it must follow the snake_case naming convention.

All the *.v files in the module's folder must have the same module name.

This rule does not apply to test files of the module which could use the main module or belong to the module when testing the private functionality of the module.

Note: Importing a module will parse every file in the folder and add them to builder.parsed_files ( builder.parse_imports/builder.v_files_from_dir) regardless of the module in the file. But this is by no means an intended feature.

Not every folder needs to be a module. We can organized our modules intuitively using folders just to have a logical structure that is more natural and organic to the developer. For example we have modules to use Google's API, we have a module for the directory api and one for the drive api. Instead of having these modules in the googles/ folders we can add a folder google/services/ and add the directory and drive modules there. The folder services is not a module is just a folder added to organize our code better.

A folder with *.v files is a module, but not every folder in a module is a module.

A module folder can have sub modules which are simply folders that we use to further divide and organize the code/functionality provided by the module.

In all of these operations we are very conscious of io operations, we limit ourselves to stats with os.exists and reduce the walking.

Packages and v.mod

A v.mod file is strictly used as a source of meta data for VPM.

Adding a v.mod file to a module, turns the module into a VPM package. The v.mod file is used to provide information to VPM.

If module is a VPM package, then the root folder (ie: google/) is the package folder which is also the repo and module name. VPM will continue to treat package as a single unit, regardless of its folder hierarchy and sub-modules.

No distinctions are made between a package and a module.

Evaluate the need for an empty v.mod.

Using an empty v.mod creates a dummy package that can not be used with VPM, it is just a regular module.

In the project folder a v.mod is irrelevant, so a v.mod could be used when a module is include in another module. In this case the presence of a v.mod will be used to anchor the qualified module name (QMN) (defines the module boundary) and sets the path of @VMODROOT. But in practice it is not necessary because now we have @FILE_DIR and @MOD_DIR.

VLIB

VLIB is the standard library, a collection of modules that are delivered with V. The folder vlib/ is the special location where we find the modules of the standard library.

Root folders

A root folder is a location where we spect to find modules, including the modules of the standard library. There are two default root folders in V, the vlib (standard library) and the default user's modules folder usually created at ~/.vmodules (configured with the env variable VMODULES). The user can specify the root folders to use with the option -path. Both the -path options and the environment variable VMODULES allow the user two specify multiple paths. A root folder will never contain a modules/ folder. As root folders, these folders themselves are NOT consider part of a module or belonging to a module.

The paths of the root folders are available under pref.lookup_path which may include @vlib, @vmodules but not necessary. When using -path we expect users to at least include @vlib like in "@vlib|@vmodules|~/my_modules", if they don't compilation will fail.

The compilation artifact

The subject used for the compilation artifact (executable, object, library, dll, etc) is the target project. For example when we run v run ~/myproject, then ~/myproject is the target project producing the artifact. It's the same for a command like v run ~/myproject/cli/simple_cli_tool.v, here ~/myproject/cli/ if the target project. We will call this folder the artifact folder or project.

The project folder is pref.path (Builder.compiled_dir) (which could also be the working directory) and also available with the new @PRJ_DIR.

The project folder can have local modules used by the project in sub folders along side the main module's files.

A module named main is a special module that has the entry point function main() use when building traditional executables.

The artifact folder becomes by context another root folder in the search for modules. The project's modules are then expected to be relative to the artifact folder.

Sibling Resolution We add an exception (see the section module resolution) to allow for related artifacts (and modules) to be organized inside a folder and easily be able to use common modules shared between artifacts or for one artifact to use modules of another. For example I have a solution that is compromised of three components a server, a client, and an admin app. Following this example a folder mysolution/ will include three artifact folders ./server/, ./client/, './console/', we never intend to build mysolution/ instead we build three separate artifacts. By including the parent folder of an artifact in the resolution of modules we could for example in the client code do import server.query, or we could have a fourth folder named ./core/ and import from any of the three artifacts using import core.query. This also allows for us to be able to put our tests in a ./tests/ folder and still being able to resolve modules without the need of current v.mod hacks.

Modules folder

Optionally a modules/ folder could be added to organize the target's modules. When used, the modules/ folder becomes a root folder and imports can use the module's name relative to the modules folder, so import a resolves to ./modules/a/.

While this is not intended for modules meant to be reuse/shared, the module resolution does not make a distinction between the main module and a regular module. So the resolution code could indeed allow for things like import a.b resolving to .vmodules/a/modules/b, import a.b.c.d resolving to .vmodules/a/modules/b/c/modules/d and all other possible combinations.

Qualified Module Name

The Qualified Module Name (QMN) is a more specific form (unambiguous) of qualifiers for a module name.

To avoid ambiguity the QMN is always anchored to a root folder (that includes the artifact folder). For example:

• import os.cmdline is os.cmdline
• import asset in vlib/os/somefile.v is os.asset
• import nlp in .vmodules/vsl/ml/somefile.v is vsl.ml.nlp
• import a.b.c in ~/myproject/somefile.v is a.b.c
• import b in ~/myproject/a/somefile.v is a.b
• import a.b.c when ~/myproject/modules/a/b/c is a.b.c

We currently do this for vlib/ and .vmodules/ but not for the artifact.

Every folder in the module's path is part of the module's qualified name, regardless if the folder is a module itself or not. For example in one of the previous examples we had the module google/services/directory, the QMN is google.services.directory. The exception is the special modules/ folder that is never used in the QMN.

The QMN is used to scope the imported symbols in the table and ast.

Note: One of the reasons submodules are broken right now is because qualify_import() and qualify_module() may not agree on the QMN assigned to a module.

Resolving modules

The module source name is the name specified with the import or module keyword.

The alias (mod.shortname) is the last segment of the import name or as specified with the as keyword. The alias is the name used in code.

The module's source name is converted to a relative path ( . replaced with os.path_separator).

The root folders become the initial search paths. In addition we look in: the artifact folder, the folder of the file being parsed.

To allow to resolve siblings we add the parent folder of both the artifact folder and the folder of the file being parsed.

For every search root we walk the segments of the module's source name forward disqualifying a search root while testing valid instances of modules/. This steps resolves the module's directory and the anchor is the search root used.

If the resolution is called from a module name declaration we take a shortcut assuming that we are already in the module's folder.

With the module directory found we check the cache to avoid further steps and return the cache info for the module.

At this stage we already have a valid Qualified Name anchored at the search root.

We move to find a v.mod, this time we only walk back the paths segments minus the root path (the anchor). (reduce the walk, reduce the io,dont ls files, only stat with os.exists)

If v.mod is found we adjust our QMN accordingly. Adjusting the QMN is ok, we do not create ambiguity because we only resolve a module once. Changing the QMN relative to the v.mod doesn't really add anything (we may save a few bytes) but it could inadvertently create a collision of modules. If we adjust the QMN is because we found a v.mod before we reached the root folder of the module, the only pattern I can think of is a module with a sub module (package) that has a v.mod. My gut says there is no need to adjust the QMN. I discuss the v.mod file more later, but for now my question is what are the use-cases of a package with sub-packages?

The path of the module directory and the v.mod file are saved in the ast.Module and ast.Import node. Builder.parse_imports and others now use this info without having to resolve a module again.

The new resolve_module() function also handles src/ as an edge case.

Further discussions

v.mod use

The QMN anchored at a root folder solves all ambiguities required for proper symbol resolution of the table and ast imported symbol references.

For convenience we add three new at expressions @FILE_DIR, @PRJ_DIR, and @MOD_DIR, we already have means to reliably find v's folder (@VEXEROOT, VEXE).

We really do not have a need to search for v.mod while parsing:

• v.mod could be exclusive for VPM's packages.
• For a package we can safely assume there is one v.mod file at the root of the mod's repo. In what instances we would have a sub package with its own v.mod, that is not a dependency? VPM has no notion of sub packages and it will not touch it.
• For current use cases where we use @VMODROOT relaying on v.mod as a marker we have other @ expressions. Legit packages have no issues using @VMODROOT.

NOTE: We need to revise how we use @VMODROOT in vlib and its tests. Looking up @VMODROOT is expensive and in almost all cases @VEXEROOT is the correct answer. Here the cache is fairly useless because checker.at_expr() will use the current file path, so is searching all the time.

the modules folder

I don't have any particular opinion about supporting the modules/ folder, other than removing the search for it and all its possible combinations simplifies and speeds up the resolution.

Single point for module resolution

Saving the resolution in ast.Module and ast.Import allows to simplify a lot of the code.

Having resolve_module() as a single point for module resolution simplifies future improvements like:

• Allowing multiple versions of a module to exist and the developer to specify a version.
• Allowing users to alias a module to a different path/version. (Currently possible using -path and forcing a different search order.)

[spytheman]

Note: Importing a module will parse every file in the folder and add them to builder.parsed_files ( builder.parse_imports/builder.v_files_from_dir) regardless of the module in the file. But this is by no means an intended feature.

I am not sure what do you mean by this is by no means an intended feature. It is fully intended that all .v files in a module/folder are parsed, when it is imported. It is also intended that for top level projects/prototypes, you can:

### Special considerations for project folders

For the top level project folder (the one, compiled with `v .`), and *only*
that folder, you can have several .v files, that may be mentioning different modules
with `module main`, `module abc` etc

This is to ease the prototyping workflow in that folder:
- you can start developing some new project with a single .v file
- split functionality as necessary to different .v files in the same folder
- when that makes logical sense to be further organised, put them into their own directory module.

Note that in ordinary modules, all .v files must start with `module name_of_folder`.

which is documented in doc/docs.md , and it was also purposefully designed that way, to ease prototyping - starting from a single file, then a single folder with multiple files, and only then multiple modules/files.

[ctkjose]

Fixed the text about main, that was mainly meant to establish what a project/artifact folder which later is used as an anchor.

[ctkjose]

project/
   folder/os/x.v
   main.v

with main.v doing import folder.os as fos and import os, and x.v having module os .

If you take a shortcut, so to speak, what will happen to the module resolution of x.v, will that be considered part of the vlib's os module, just because it declared module os at its top?

As expected import folder.os will be resolved relative to the project, and import os to vlib/os because folder/ disambiguate the resolution, also the precedence/search order comes into play.

Inside the file folder/os/x.v when we parse the module declaration module os the path of the file parsed gave us a unique absolute path for example ~/project/folder/os/x.v thus we are at the module's directory. I mean as far as I know we do not allow arbitrary files to join modules declared elsewhere, did you mean you want x.v to join vlib.os?

Note: The shortcut of using the file path when we have a module declaration is just to skip the walking, from there we just need to find the search root to anchor everything to it which is faster than the walk.

The new resolve_module() is pretty much the same logic we had, the core difference is that we anchor the traversal, and use these anchors to generate a constant qualified name. It just bundles the logic in one place.

[ctkjose]

It is fully intended that all .v files in a module/folder are parsed, when it is imported. It is also intended that for top level projects/prototypes, you can:

I see, but I think we still need to look at why this works and what you would teach new users.

I have to check the code again, but as far as I remember I don't think qualify_import() or qualify_module() will do anything in these cases, is just that when they fail to resolve they return the name as enter by the user. Then since builder.v_files_from_dir() drags everything in, a module declared in a file like project/module_a.v is available as imported symbols. Like you mention for quick prototyping this works. But import a will never resolve. Lets look at another case where we have project/mymod/mod.v and project/mymod/anothermod.v; in mod.v we may get away with making anothermod available but everywhere else would need to do import mymod to get anothermod imported, every import anothermod is just quietly failing.

My main point here is that we are letting import statements fail. I think is worth thinking a bit more about this.

[antono2]

Right now it can not be a sub module, because import imgui.implot prints an error about not public - had this issue just yesterday, but didn't have the time to write a ticket.

Yes, it can be, and you can later import imgui.implot @antono2 . Please file an issue, if you can not, and make sure to add the working folder for your compilation (or retry it after say cd ~, with v /abs/path/to/your/usage.v , to see if the problem will go away).

@spytheman Yes, it works! Tested on minimal example. Thanks for the correction

[JalonSolov]

V is 6 years old, and the current module import system has been used the entire time it has existed. Yes, there are a few flaws, but does that mean we need to completely toss it out and start over? Or just fix the current problems and move on?

[ctkjose]

I'll post a draft PR soon, so you guys can look at the new resolve_module() function.

The logic is pretty much the same, it just consolidates the work of various functions. The changes are: how it internally sets the qualified name of a module which now anchors the QMN, but this is an internal thing that doesn't affect how the user use modules, it just ensures a module at a given path always gets the same QMN regardless of where it is import from. The other is when it stops traversing the path.

The use of v.mod and @VMODROOT are just philosophical questions, thinking that perhaps it is a good time to finally settle the ambiguities and following the one-way spirit of V, we could document a consistent explanation of these features.

Then we have the minute stuff, additional topics like special folders (modules/, src/), etc.

[JalonSolov]

Alex already decided src will no longer be treated special. It will simply be a directory name. v new has already been modified so that it won't create the src subdir if you use it to start a new V project.

I keep trying to get modules removed as anything special, as well. It should also just be a subdir name that you can create if you want it.