Now that I have a custom dataset and a trained tracker, how can I use vot toolkit to calculate some evaluation metrics?

[hhhyyyqqq]

Now that I have a custom dataset and a trained tracker, how can I use vot toolkit to calculate some evaluation metrics like f_curve ?

that is how to change votchallenge dataset to my own custom dataset or how to customize the configuration file .yaml?

thank you for your answer.

[lukacu]

Hi, thanks for the question, and sorry for the late reply; I was on vacation. There is no streamlined process that would make importing a new dataset very easy - at the moment. The first thing that you have to do is to convert sequences in one of the toolkit-compatible formats. Place them in a sequences subdirectory of your to-be workspace. Then, manually create a new config file in the workspace directory and specify a custom stack of experiments.

I will try to make this more accessible in the future, but that is the only way. It helps if you have an existing workspace somewhere and you can inspect it.

[lparolari]

@hhhyyyqqq I don't know if you solved your issue, but I will try to explain what I did after exporting my custom dataset in VOT format and trained a tracker:

I've setup a workspace, i.e. a directory with config.yaml, stack.yaml, trackers.ini files and a sequences folder.

The sequences folder of course should contain the dataset (more info can be found in the docs).

Then, you have the config file, which is the one that defines workspace's experiments and trackers:

config.yaml

registry:
- ./trackers.ini
stack: ./my_stack.yaml

The trackers.ini file defines where to find and how to execute a tracker:

[mytracker]
label = mytracker
protocol = traxpython
command = run_mytracker
paths = /path/to/mytracker

Please note that the command could be a python module, e.g. trackers.mytracker. This means that in /path/to/mytracker you should find a python package called "trackers" with a file inside called mytracker.py.

Then you have the experiments stack:

my_stack.yaml

title: my-stack
experiments:
  longterm:
    type: unsupervised
    repetitions: 1
    analyses:
      - type: average_tpr
        name: average_tpr
      - type: pr_curve
      - type: f_curve

It defines the experiments the toolkit should run with given tracker. I've looked through vot challenges https://github.com/votchallenge/toolkit/tree/master/vot/stack and borrowed a stack of experiments for the longterm tracking, but many other stacks could be built, it depends on your needs. Be aware that some experiments cannot be run if you do not have the appropriate annotations.

[hhhyyyqqq]

Thank you all for your help. Your suggestions are really useful.

[lukacu]

Thank you, @lparolari, for helping out 👍