Tools are absent from the generated tsv even if the category is included in the file

[scorreard]

A first version of the codex biodiversity was generated using the following categories for tools :

Assembly
Ecology
Genome annotation
Entomology
GIS

However, users identified missing tools that are associated to the categories Entomology and GIS:

netcdf_handler (GIS)
extract_ipm_date_interval (Entomology)
insect_phenology_model (Entomology)

[scorreard]

After checking the file communities/all/resources/tools.tsv that includes all the tools (before filtering by the categories), the tools mentionned above are absent from the all tools file meaning that the issue is with the tools extraction and not filtering

[scorreard]

The code line run to extract the tools is
bash sources/bin/extract_all_tools.sh "${{ matrix.subset }}"
with

        subset:
         - repositories01.list
         - repositories02.list
         - repositories03.list
         - repositories04.list

In details, I think it runs:

python sources/bin/extract_galaxy_tools.py \
                        extract \
                        --api $GITHUB_API_KEY \
                        --all-tsv $tsv_output \
                        --all $json_output \
                        --planemo-repository-list $1 \
                        --avoid-extra-repositories

The missing tools are likely not included in the repositories 1 to 4.
Do we need to add them to a repo or would removing the option --avoid-extra-repositories enough?

[paulzierep]

If the tools are in a repository which is not monitored by the planemo monitor (https://github.com/galaxyproject/planemo-monitor), the repository can be added to the https://github.com/galaxyproject/galaxy_codex/blob/main/sources/data/conf.yml as extra-repositories adn the codex will parse them

[scorreard]

I'm not sure what link to include in the conf file, here are the links to the 3 tools (in ToolShed) :
https://toolshed.g2.bx.psu.edu/repository?repository_id=75234542f3ddee40
https://toolshed.g2.bx.psu.edu/repository?repository_id=fb47b535881d43b7
https://toolshed.g2.bx.psu.edu/repository?repository_id=dde9d8447a5859cb

How do I find the GitHub link to their galaxy tool?

[paulzierep]

I'm not sure what link to include in the conf file, here are the links to the 3 tools (in ToolShed) : https://toolshed.g2.bx.psu.edu/repository?repository_id=75234542f3ddee40 https://toolshed.g2.bx.psu.edu/repository?repository_id=fb47b535881d43b7 https://toolshed.g2.bx.psu.edu/repository?repository_id=dde9d8447a5859cb

How do I find the GitHub link to their galaxy tool?

Thats one of the many reasons we developed the codex, the toolshed does not allow to find the dev repository of a tool. Via the codex you can find it, since we directly parse the code from there ...
For the tools here we need to find the development repository first. Do you know the devs ? They should know :)

[scorreard]

No I don't know the devs, it was a community request during the All Hands meeting

I found the following github repo :

https://github.com/OPENDAP/netcdf_handler
https://github.com/gregvonkuster/galaxy_tools/tree/master/tools/entomology/extract_ipm_date_interval
https://github.com/gregvonkuster/galaxy_tools/tree/master/tools/entomology/insect_phenology_model

So I guess for the last two, I can add https://github.com/gregvonkuster/galaxy_tools to https://github.com/galaxyproject/galaxy_codex/blob/main/sources/data/conf.yml ?

[paulzierep]

yes that should work