Audio Retrieval Dataset: JLCorpus #2927
Conversation
| class JLCorpusA2TRetrieval(AbsTaskAny2AnyRetrieval): | ||
| metadata = TaskMetadata( | ||
| name="JLCorpusA2TRetrieval", | ||
| description=( |
There was a problem hiding this comment.
So from a speech segment retrieve the emotion (e..g text label "angry"?)
There was a problem hiding this comment.
This dataset contains transcriptions that have been said with different emotions. Supposedly, we should retrieve the transcription invariant to emotion. Maybe that can be the evaluation. Only problem here could be if evaluation is done with ids rather than on unique "text" columns.
Processed Dataset: https://huggingface.co/datasets/mteb/JL-Corpus_a2t
Original Dataset: https://huggingface.co/datasets/CLAPv2/JL-Corpus
There was a problem hiding this comment.
Right, so from the n versions of a text, it should retrieve the one that best matches the emotion. However, doesn't the task retrieve over all possible pairs?
There was a problem hiding this comment.
In the way it is currently done (text column has no emotion info), I figured we should just see if audio segments are able to consistently retrieve the correct text column even if audio has varying emotions associated with it.
There was a problem hiding this comment.
Sounds like something that is easier to test prior to merging. Seems like from the results that they can't.
Results on
laion/clap-htsat-fused:JLCorpusT2ARetrieval.json
JLCorpusA2TRetrieval.json
P.S This task could be problematic if evaluation is done on uniqueness of query-id rather than actual query.