Create an overview of cleaning taggers

[KennethEnevoldsen]

Agreed with @peterbjorgensen that it would be a great idea to create over overview of what taggers might be relevant for cleaning.

Outlining

• Create a .md table with relevant taggers + a short description
• Check what filters were used for existing cleaning strategies and at least try to match them (see here)
• potentially some estimate on speed (time to process danish gigaword Wikipedia section ~55M tokens)

[TTTTao725]

@KennethEnevoldsen @peterbjorgensen

These are taggers that might be relevant for cleaning:

• Dataset: a subset of DAGW which is only consist of documents from 'Wiki & Books'
• 1 process

#	Dolma Tagger	Description	Relation to Filtering Criteria	Process Time
1	char_length_v1	Computes document length in characters	Relates to character count	30s
2	char_length_with_paragraphs_v1	Computes document and paragraph length in characters	Relates to character count	1m27s
3	cld2_en_doc_v2	Detects document language using cld2	Indirectly relates to language and possibly stopwords	1m44s
4	olmo_pretokenizer_v1	Counts number of tokens using OLMo v1 pre-tokenizer	Relates to token count	8m08s
5	olmo_pretokenizer_with_paragraphs_v1	Counts tokens in document and paragraphs using OLMo v1 pre-tokenizer	Relates to token count	8m39s
6	whitespace_tokenizer_v1	Counts whitespace-separated tokens in document	Relates to token count	1m49s
7	whitespace_tokenizer_with_paragraphs_v1	Counts whitespace-separated tokens in document and paragraphs	Relates to token count	2m03s
8	random_number_v1	Assigns a random number to each document	Facilitates dataset splitting	24s
9	ft_lang_id_en_doc_v2	Uses fastText to detect the language of the document	Indirectly relates to language and possibly stopwords	3m20s
10	ft_lang_id_en_paragraph_v2	Uses fastText to detect the language of each paragraph	Indirectly relates to language and possibly stopwords	6m38s
11	ft_lang_id_en_paragraph_with_doc_score_v2	Uses fastText to detect the language of each paragraph and assigns a score based on the fraction of English paragraphs	Indirectly relates to language and possibly stopwords	5m32s

[KennethEnevoldsen]

Seems like we are missing the gopher filters, PII and c4. Will also add this table as a PR with a a brief introduction on how to use the dolma tagger (it can just be a reference to their documentation)

I would also like to check which existing were ignored (e.g. we discussed stopwords).

Will you also add the taggers implemented in our github (see codebase)

[peterbjorgensen]

Yes, we have these taggers implemented:
https://github.com/centre-for-humanities-computing/danish-foundation-models/tree/main/src/dfm/common/data_cleaning/dolma_taggers

Maybe we should also include the remaining Germanic languages (nl, de)

[github-actions]

This issue is stale because it has been open for 14 days with no activity. Feel free to either 1) remove the stale label or 2) comment. If nothing happens, this will be closed in 7 days.

[peterbjorgensen]

@TTTTao725 How did you measure the processing times of the taggers in the table. Did you create a script to do this?

[TTTTao725]

@peterbjorgensen No, they have a build-in timer so you check it out right after one execution of a tagger, I'll make a PR these days, I have tested more taggers including ones for scandi languages :)