fix: parallel file validation for a datapackage #1722
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context
Resource
was imported as a type, but then used as a class to retrieve a class method (Resource.from_descriptor
).Resource
(not as a type) intovalidator.py
leads to a circular import. A small workaround would have been to import Resource locally, but this was not very satisfactory.Suggested changes and fixes
Validator.validate_resource
andValidator.validate_package
methods to theResource.validate
andPackage.validate
methods. This solves the circular import problem, and removes duplication (method signatures).Validator
class, keeping the methods (forwarding input parameters) as they are part of the public API. No warning, only a deprecation message in the docstring, and a mention that there is no plan to remove the class in the future.Note
I could not produce any significant performance gain in my tests, I wonder if there is some shared resources' lock that prevents the validation from effectively running the validation in parallel (but I have not spent much time on this yet).
Archive of explorations
validate_parallel
function was actually never executed in the tests.Resource
andDataPackage
directly, but I may have missed something.