Inconsistent IDs

[thammegowda]

Hi,

Thanks for your efforts in creating/curating these datasets! These are priceless and greatly advance NLP for Indian languages.

I tried adding them into mtdata thammegowda/mtdata#81
Since the README says your datasets are still growing, I am wondering whats the best long-term strategy is for keeping in sync.

For now, I can grep -i -o 'http[^ ]*zip' README.md, but the immediate concern is about consistency in determining name, version, and languages of datasets from URL.

The way current files are named (which act as ID for corpus) is a bit inconsistent. For example, consider these:

1) https://anuvaad-parallel-corpus.s3-us-west-2.amazonaws.com/oneindia_20210320_en_ml.zip
2) https://anuvaad-parallel-corpus.s3-us-west-2.amazonaws.com/pibarchives_2014_2016_en_ml.zip
3) https://anuvaad-parallel-corpus.s3-us-west-2.amazonaws.com/wikipedia-en-ml-20210201.zip

• item (1), we can easily split by _ and get (name, version, lang1, lang2), so this is great. we can see oneindia is the name, 20210320 is the version, and en_ml are langs.
• item (2), seems okay we can call 2014_2016 as version, though it would have been nice to have 2014to2016v1 as version. so splitting by _ would give exactly (name, version, lang1, lang2) as in item 1.
• item (3) seems abnormal as it doesn't fit (name, version, lang1, lang2). There are more datasets matching item (1) than item (3) pattern, so I am inclined to call this abnormal.

Could you please consider having a consistent format in dataset IDs? It'd greatly help the automated downloaders such as mtdata.
^{Otherwise, do you really want your users to manually download 196 zip files via browser, and extract and merge them? :)}

Thanks.

P.S https://github.com/thammegowda/mtdata#dataset-id

[thammegowda]

Also, note that there is a bit of inconsistency inside zipping of item (3) as well.

$ unzip oneindia_20210320_en_ml.zip
Archive:  oneindia_20210320_en_ml.zip
   creating: en-ml/
  inflating: en-ml/oneindia_train.ml
  inflating: en-ml/oneindia_train.en
$ unzip pibarchives_2014_2016_en_ml.zip
Archive:  pibarchives_2014_2016_en_ml.zip
  inflating: en-ml/.DS_Store
  inflating: __MACOSX/en-ml/._.DS_Store
  inflating: en-ml/pib_arch_train.en
  inflating: en-ml/pib_arch_train.ml

$ unzip wikipedia-en-ml-20210201.zip
Archive:  wikipedia-en-ml-20210201.zip
  inflating: en-ml/ml.txt
  inflating: en-ml/en.txt

en-ml/wikipedia_train.{en,ml} could have made scripts/automation tools simple to write.

[thammegowda]

I added these datasets to v0.3.2

pip install -I mtdata==0.3.2

See mappings

mtdata list | grep Anuvaad  | sed 's;https://anuvaad-parallel-corpus.s3-us-west-2.amazonaws.com;<BASE>;' 
2021-12-05 13:35:24 __init__.get_instance:48 INFO:: Loading index from cache /Users/tg/.mtdata/mtdata.index.0.3.2.pkl
2021-12-05 13:35:25 main.list_data:19 INFO:: Found 182787
Anuvaad-ik_2021-v1-eng-hin      <BASE>/ik-2021-v1-en-hi.zip     en-hi/*.en,en-hi/*.hi
Anuvaad-internal_judicial_2021-v1-eng-hin       <BASE>/internal-judicial-2021-v1-en-hi.zip      en-hi/*.en,en-hi/*.hi
Anuvaad-legal_terms_2021-v1-eng-hin     <BASE>/legal-terms-2021-v1-en-hi.zip    en-hi/*.en,en-hi/*.hi
Anuvaad-nouns_dict_2021-v1-eng-hin      <BASE>/nouns-dict-2021-v1-en-hi.zip     en-hi/*.en,en-hi/*.hi
Anuvaad-pib_2017-2020-eng-hin   <BASE>/pib_2017_2020_en_hi.zip  en-hi/*.en,en-hi/*.hi
Anuvaad-pibarchives_2009-2016-eng-hin   <BASE>/pibarchives_2009_2016_en_hi.zip  en-hi/*.en,en-hi/*.hi
Anuvaad-newsonair_2019-2020-eng-hin     <BASE>/newsonair_2019_2020_en_hi.zip    en-hi/*.en,en-hi/*.hi
Anuvaad-wikipedia-20210201-eng-hin      <BASE>/wikipedia-en-hi-20210201.zip     en-hi/en.txt,en-hi/hi.txt
Anuvaad-drivespark-20210303-eng-hin     <BASE>/drivespark-en-hi-20210303.zip    en-hi/*.en,en-hi/*.hi
Anuvaad-dd_national-20210320-eng-hin    <BASE>/dd-national-en-hi-20210320.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-dd_sports-20210320-eng-hin      <BASE>/dd-sports-en-hi-20210320.zip     en-hi/*.en,en-hi/*.hi
Anuvaad-nativeplanet-20210315-eng-hin   <BASE>/nativeplanet-en-hi-20210315.zip  en-hi/*.en,en-hi/*.hi
Anuvaad-catchnews-20210320-eng-hin      <BASE>/catchnews-en-hi-20210320.zip     en-hi/*.en,en-hi/*.hi
Anuvaad-dwnews_2008-2020-eng-hin        <BASE>/dwnews_2008_2020_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-oneindia-20210320-eng-hin       <BASE>/oneindia_20210320_en_hi.zip      en-hi/*.en,en-hi/*.hi
Anuvaad-mk-20210320-eng-hin     <BASE>/mk_20210320_en_hi.zip    en-hi/*.en,en-hi/*.hi
Anuvaad-goodreturns-20210320-eng-hin    <BASE>/goodreturns_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-ie_sports-20210320-eng-hin      <BASE>/ie_sports_20210320_en_hi.zip     en-hi/*.en,en-hi/*.hi
Anuvaad-ie_tech-20210320-eng-hin        <BASE>/ie_tech_20210320_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-ie_news-20210320-eng-hin        <BASE>/ie_news_20210320_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-ie_lifestyle-20210320-eng-hin   <BASE>/ie_lifestyle_20210320_en_hi.zip  en-hi/*.en,en-hi/*.hi
Anuvaad-ie_general-20210320-eng-hin     <BASE>/ie_general_20210320_en_hi.zip    en-hi/*.en,en-hi/*.hi
Anuvaad-ie_entertainment-20210320-eng-hin       <BASE>/ie_entertainment_20210320_en_hi.zip      en-hi/*.en,en-hi/*.hi
Anuvaad-ie_education-20210320-eng-hin   <BASE>/ie_education_20210320_en_hi.zip  en-hi/*.en,en-hi/*.hi
Anuvaad-ie_business-20210320-eng-hin    <BASE>/ie_business_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-toi-20210320-eng-hin    <BASE>/toi_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-fin_express-20210320-eng-hin    <BASE>/fin_express_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-thewire-20210320-eng-hin        <BASE>/thewire_20210320_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-tribune-20210320-eng-hin        <BASE>/tribune_20210320_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-zeebiz-20210320-eng-hin <BASE>/zeebiz_20210320_en_hi.zip        en-hi/*.en,en-hi/*.hi
Anuvaad-pa_govt-20210320-eng-hin        <BASE>/pa_govt_20210320_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-betterindia-20210320-eng-hin    <BASE>/betterindia_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-jagran_news-20210320-eng-hin    <BASE>/jagran-news_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-jagran_tech-20210320-eng-hin    <BASE>/jagran-tech_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-jagran_education-20210320-eng-hin       <BASE>/jagran-education_20210320_en_hi.zip      en-hi/*.en,en-hi/*.hi
Anuvaad-jagran_entertainment-20210320-eng-hin   <BASE>/jagran-entertainment_20210320_en_hi.zip  en-hi/*.en,en-hi/*.hi
Anuvaad-jagran_business-20210320-eng-hin        <BASE>/jagran-business_20210320_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-jagran_sports-20210320-eng-hin  <BASE>/jagran-sports_20210320_en_hi.zip en-hi/*.en,en-hi/*.hi
Anuvaad-jagran_lifestyle-20210320-eng-hin       <BASE>/jagran-lifestyle_20210320_en_hi.zip      en-hi/*.en,en-hi/*.hi
Anuvaad-asianetnews-20210320-eng-hin    <BASE>/asianetnews_20210320_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-business_standard-20210320-eng-hin      <BASE>/business_standard_20210320_en_hi.zip     en-hi/*.en,en-hi/*.hi
Anuvaad-pranabmukherjee-20210320-eng-hin        <BASE>/pranabmukherjee_20210320_en_hi.zip       en-hi/*.en,en-hi/*.hi
Anuvaad-lokmat_entertainment-20210501-eng-hin   <BASE>/lokmat-entertainment_20210501_en_hi.zip  en-hi/*.en,en-hi/*.hi
Anuvaad-lokmat_news-20210501-eng-hin    <BASE>/lokmat-news_20210501_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-lokmat_lifestyle-20210501-eng-hin       <BASE>/lokmat-lifestyle_20210501_en_hi.zip      en-hi/*.en,en-hi/*.hi
Anuvaad-lokmat_sports-20210501-eng-hin  <BASE>/lokmat-sports_20210501_en_hi.zip en-hi/*.en,en-hi/*.hi
Anuvaad-lokmat_tech-20210501-eng-hin    <BASE>/lokmat-tech_20210501_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-lokmat_financial-20210501-eng-hin       <BASE>/lokmat-financial_20210501_en_hi.zip      en-hi/*.en,en-hi/*.hi
Anuvaad-lokmat_healthcare-20210501-eng-hin      <BASE>/lokmat-healthcare_20210501_en_hi.zip     en-hi/*.en,en-hi/*.hi
Anuvaad-zee-30042021-eng-hin    <BASE>/zee_30042021_en_hi.zip   en-hi/*.en,en-hi/*.hi
Anuvaad-ik_2021-v1-eng-ben      <BASE>/ik-2021-v1-en-bn.zip     en-bn/*.en,en-bn/*.bn
Anuvaad-internal_judicial_2021-v1-eng-ben       <BASE>/internal-judicial-2021-v1-en-bn.zip      en-bn/*.en,en-bn/*.bn
Anuvaad-pib_2017-2020-eng-ben   <BASE>/pib_2017_2020_en_bn.zip  en-bn/*.en,en-bn/*.bn
Anuvaad-pibarchives_2015-2016-eng-ben   <BASE>/pibarchives_2015_2016_en_bn.zip  en-bn/*.en,en-bn/*.bn
Anuvaad-prothomalo_2014-2020-eng-ben    <BASE>/prothomalo_2014_2020_en_bn.zip   en-bn/*.en,en-bn/*.bn
Anuvaad-dwnews_2004-2020-eng-ben        <BASE>/dwnews_2004_2020_en_bn.zip       en-bn/*.en,en-bn/*.bn
Anuvaad-oneindia-20210320-eng-ben       <BASE>/oneindia_20210320_en_bn.zip      en-bn/*.en,en-bn/*.bn
Anuvaad-mk-20210320-eng-ben     <BASE>/mk_20210320_en_bn.zip    en-bn/*.en,en-bn/*.bn
Anuvaad-ie_sports-20210320-eng-ben      <BASE>/ie_sports_20210320_en_bn.zip     en-bn/*.en,en-bn/*.bn
Anuvaad-ie_tech-20210320-eng-ben        <BASE>/ie_tech_20210320_en_bn.zip       en-bn/*.en,en-bn/*.bn
Anuvaad-ie_news-20210320-eng-ben        <BASE>/ie_news_20210320_en_bn.zip       en-bn/*.en,en-bn/*.bn
Anuvaad-ie_general-20210320-eng-ben     <BASE>/ie_general_20210320_en_bn.zip    en-bn/*.en,en-bn/*.bn
Anuvaad-ie_entertainment-20210320-eng-ben       <BASE>/ie_entertainment_20210320_en_bn.zip      en-bn/*.en,en-bn/*.bn
Anuvaad-ie_education-20210320-eng-ben   <BASE>/ie_education_20210320_en_bn.zip  en-bn/*.en,en-bn/*.bn
Anuvaad-ie_business-20210320-eng-ben    <BASE>/ie_business_20210320_en_bn.zip   en-bn/*.en,en-bn/*.bn
Anuvaad-ie_lifestyle-20210320-eng-ben   <BASE>/ie_lifestyle_20210320_en_bn.zip  en-bn/*.en,en-bn/*.bn
Anuvaad-asianetnews-20210320-eng-ben    <BASE>/asianetnews_20210320_en_bn.zip   en-bn/*.en,en-bn/*.bn
Anuvaad-sentinel-20210320-eng-ben       <BASE>/sentinel_20210320_en_bn.zip      en-bn/*.en,en-bn/*.bn
Anuvaad-toi-20210320-eng-ben    <BASE>/toi_20210320_en_bn.zip   en-bn/*.en,en-bn/*.bn
Anuvaad-zee-30042021-eng-ben    <BASE>/zee_30042021_en_bn.zip   en-bn/*.en,en-bn/*.bn
Anuvaad-anuvaad_general-corpus-eng-ben  <BASE>/anuvaad_general_corpus_en_bn.zip en-bn/*.en,en-bn/*.bn
Anuvaad-ik_2021-v1-eng-tam      <BASE>/ik-2021-v1-en-ta.zip     en-ta/*.en,en-ta/*.ta
Anuvaad-internal_judicial_2021-v1-eng-tam       <BASE>/internal-judicial-2021-v1-en-ta.zip      en-ta/*.en,en-ta/*.ta
Anuvaad-legal_terms_2021-v1-eng-tam     <BASE>/legal-terms-2021-v1-en-ta.zip    en-ta/*.en,en-ta/*.ta
Anuvaad-nouns_dict_2021-v1-eng-tam      <BASE>/nouns-dict-2021-v1-en-ta.zip     en-ta/*.en,en-ta/*.ta
Anuvaad-pib_2017-2020-eng-tam   <BASE>/pib_2017_2020_en_ta.zip  en-ta/*.en,en-ta/*.ta
Anuvaad-pibarchives_2014-2016-eng-tam   <BASE>/pibarchives_2014_2016_en_ta.zip  en-ta/*.en,en-ta/*.ta
Anuvaad-wikipedia-20210201-eng-tam      <BASE>/wikipedia-en-ta-20210201.zip     en-ta/en.txt,en-ta/ta.txt
Anuvaad-nativeplanet-20210315-eng-tam   <BASE>/nativeplanet-en-ta-20210315.zip  en-ta/*.en,en-ta/*.ta
Anuvaad-oneindia-20210320-eng-tam       <BASE>/oneindia_20210320_en_ta.zip      en-ta/*.en,en-ta/*.ta
Anuvaad-mk-20210320-eng-tam     <BASE>/mk_20210320_en_ta.zip    en-ta/*.en,en-ta/*.ta
Anuvaad-drivespark-20210303-eng-tam     <BASE>/drivespark-en-ta-20210303.zip    en-ta/*.en,en-ta/*.ta
Anuvaad-goodreturns-20210320-eng-tam    <BASE>/goodreturns_20210320_en_ta.zip   en-ta/*.en,en-ta/*.ta
Anuvaad-ie_sports-20210320-eng-tam      <BASE>/ie_sports_20210320_en_ta.zip     en-ta/*.en,en-ta/*.ta
Anuvaad-ie_tech-20210320-eng-tam        <BASE>/ie_tech_20210320_en_ta.zip       en-ta/*.en,en-ta/*.ta
Anuvaad-ie_news-20210320-eng-tam        <BASE>/ie_news_20210320_en_ta.zip       en-ta/*.en,en-ta/*.ta
Anuvaad-ie_lifestyle-20210320-eng-tam   <BASE>/ie_lifestyle_20210320_en_ta.zip  en-ta/*.en,en-ta/*.ta
Anuvaad-ie_entertainment-20210320-eng-tam       <BASE>/ie_entertainment_20210320_en_ta.zip      en-ta/*.en,en-ta/*.ta
Anuvaad-ie_education-20210320-eng-tam   <BASE>/ie_education_20210320_en_ta.zip  en-ta/*.en,en-ta/*.ta
Anuvaad-ie_business-20210320-eng-tam    <BASE>/ie_business_20210320_en_ta.zip   en-ta/*.en,en-ta/*.ta
Anuvaad-asianetnews-20210320-eng-tam    <BASE>/asianetnews_20210320_en_ta.zip   en-ta/*.en,en-ta/*.ta
Anuvaad-toi-20210320-eng-tam    <BASE>/toi_20210320_en_ta.zip   en-ta/*.en,en-ta/*.ta
Anuvaad-zee-30042021-eng-tam    <BASE>/zee_30042021_en_ta.zip   en-ta/*.en,en-ta/*.ta
Anuvaad-ik_2021-v1-eng-mal      <BASE>/ik-2021-v1-en-ml.zip     en-ml/*.en,en-ml/*.ml
Anuvaad-internal_judicial_2021-v1-eng-mal       <BASE>/internal-judicial-2021-v1-en-ml.zip      en-ml/*.en,en-ml/*.ml
Anuvaad-legal_terms_2021-v1-eng-mal     <BASE>/legal-terms-2021-v1-en-ml.zip    en-ml/*.en,en-ml/*.ml
Anuvaad-nouns_dict_2021-v1-eng-mal      <BASE>/nouns-dict-2021-v1-en-ml.zip     en-ml/*.en,en-ml/*.ml
Anuvaad-pib_2017-2020-eng-mal   <BASE>/pib_2017_2020_en_ml.zip  en-ml/*.en,en-ml/*.ml
Anuvaad-pibarchives_2014-2016-eng-mal   <BASE>/pibarchives_2014_2016_en_ml.zip  en-ml/*.en,en-ml/*.ml
Anuvaad-wikipedia-20210201-eng-mal      <BASE>/wikipedia-en-ml-20210201.zip     en-ml/en.txt,en-ml/ml.txt
Anuvaad-oneindia-20210320-eng-mal       <BASE>/oneindia_20210320_en_ml.zip      en-ml/*.en,en-ml/*.ml
Anuvaad-mk-20210320-eng-mal     <BASE>/mk_20210320_en_ml.zip    en-ml/*.en,en-ml/*.ml
Anuvaad-drivespark-20210303-eng-mal     <BASE>/drivespark-en-ml-20210303.zip    en-ml/*.en,en-ml/*.ml
Anuvaad-nativeplanet-20210315-eng-mal   <BASE>/nativeplanet-en-ml-20210315.zip  en-ml/*.en,en-ml/*.ml
Anuvaad-goodreturns-20210320-eng-mal    <BASE>/goodreturns_20210320_en_ml.zip   en-ml/*.en,en-ml/*.ml
Anuvaad-ie_sports-20210320-eng-mal      <BASE>/ie_sports_20210320_en_ml.zip     en-ml/*.en,en-ml/*.ml
Anuvaad-ie_tech-20210320-eng-mal        <BASE>/ie_tech_20210320_en_ml.zip       en-ml/*.en,en-ml/*.ml
Anuvaad-ie_news-20210320-eng-mal        <BASE>/ie_news_20210320_en_ml.zip       en-ml/*.en,en-ml/*.ml
Anuvaad-ie_general-20210320-eng-mal     <BASE>/ie_general_20210320_en_ml.zip    en-ml/*.en,en-ml/*.ml
Anuvaad-ie_entertainment-20210320-eng-mal       <BASE>/ie_entertainment_20210320_en_ml.zip      en-ml/*.en,en-ml/*.ml
Anuvaad-ie_business-20210320-eng-mal    <BASE>/ie_business_20210320_en_ml.zip   en-ml/*.en,en-ml/*.ml
Anuvaad-marketfeed-20210320-eng-mal     <BASE>/marketfeed_20210320_en_ml.zip    en-ml/*.en,en-ml/*.ml
Anuvaad-asianetnews-20210320-eng-mal    <BASE>/asianetnews_20210320_en_ml.zip   en-ml/*.en,en-ml/*.ml
Anuvaad-toi-20210320-eng-mal    <BASE>/toi_20210320_en_ml.zip   en-ml/*.en,en-ml/*.ml
Anuvaad-zee-30042021-eng-mal    <BASE>/zee_30042021_en_ml.zip   en-ml/*.en,en-ml/*.ml
Anuvaad-ik_2021-v1-eng-tel      <BASE>/ik-2021-v1-en-te.zip     en-te/*.en,en-te/*.te
Anuvaad-internal_judicial_2021-v1-eng-tel       <BASE>/internal-judicial-2021-v1-en-te.zip      en-te/*.en,en-te/*.te
Anuvaad-legal_terms_2021-v1-eng-tel     <BASE>/legal-terms-2021-v1-en-te.zip    en-te/*.en,en-te/*.te
Anuvaad-nouns_dict_2021-v1-eng-tel      <BASE>/nouns-dict-2021-v1-en-te.zip     en-te/*.en,en-te/*.te
Anuvaad-pib_2017-2020-eng-tel   <BASE>/pib_2017_2020_en_te.zip  en-te/*.en,en-te/*.te
Anuvaad-pibarchives_2015-2016-eng-tel   <BASE>/pibarchives_2015_2016_en_te.zip  en-te/*.en,en-te/*.te
Anuvaad-wikipedia-20210201-eng-tel      <BASE>/wikipedia-en-te-20210201.zip     en-te/en.txt,en-te/te.txt
Anuvaad-oneindia-20210320-eng-tel       <BASE>/oneindia_20210320_en_te.zip      en-te/*.en,en-te/*.te
Anuvaad-mk-20210320-eng-tel     <BASE>/mk_20210320_en_te.zip    en-te/*.en,en-te/*.te
Anuvaad-drivespark-20210303-eng-tel     <BASE>/drivespark-en-te-20210303.zip    en-te/*.en,en-te/*.te
Anuvaad-nativeplanet-20210315-eng-tel   <BASE>/nativeplanet-en-te-20210315.zip  en-te/*.en,en-te/*.te
Anuvaad-goodreturns-20210320-eng-tel    <BASE>/goodreturns_20210320_en_te.zip   en-te/*.en,en-te/*.te
Anuvaad-sakshi-20210320-eng-tel <BASE>/sakshi_20210320_en_te.zip        en-te/*.en,en-te/*.te
Anuvaad-asianetnews-20210320-eng-tel    <BASE>/asianetnews_20210320_en_te.zip   en-te/*.en,en-te/*.te
Anuvaad-toi-20210320-eng-tel    <BASE>/toi_20210320_en_te.zip   en-te/*.en,en-te/*.te
Anuvaad-zee-30042021-eng-tel    <BASE>/zee_30042021_en_te.zip   en-te/*.en,en-te/*.te
Anuvaad-ik_2021-v1-eng-kan      <BASE>/ik-2021-v1-en-kn.zip     en-kn/*.en,en-kn/*.kn
Anuvaad-legal_terms_2021-v1-eng-kan     <BASE>/legal-terms-2021-v1-en-kn.zip    en-kn/*.en,en-kn/*.kn
Anuvaad-nouns_dict_2021-v1-eng-kan      <BASE>/nouns-dict-2021-v1-en-kn.zip     en-kn/*.en,en-kn/*.kn
Anuvaad-pib_2017-2020-eng-kan   <BASE>/pib_2017_2020_en_kn.zip  en-kn/*.en,en-kn/*.kn
Anuvaad-pibarchives_2015-2016-eng-kan   <BASE>/pibarchives_2015_2016_en_kn.zip  en-kn/*.en,en-kn/*.kn
Anuvaad-oneindia-20210320-eng-kan       <BASE>/oneindia_20210320_en_kn.zip      en-kn/*.en,en-kn/*.kn
Anuvaad-mk-20210320-eng-kan     <BASE>/mk_20210320_en_kn.zip    en-kn/*.en,en-kn/*.kn
Anuvaad-drivespark-20210303-eng-kan     <BASE>/drivespark-en-kn-20210303.zip    en-kn/*.en,en-kn/*.kn
Anuvaad-nativeplanet-20210315-eng-kan   <BASE>/nativeplanet-en-kn-20210315.zip  en-kn/*.en,en-kn/*.kn
Anuvaad-goodreturns-20210320-eng-kan    <BASE>/goodreturns_20210320_en_kn.zip   en-kn/*.en,en-kn/*.kn
Anuvaad-asianetnews-20210320-eng-kan    <BASE>/asianetnews_20210320_en_kn.zip   en-kn/*.en,en-kn/*.kn
Anuvaad-toi-20210320-eng-kan    <BASE>/toi_20210320_en_kn.zip   en-kn/*.en,en-kn/*.kn
Anuvaad-zee-30042021-eng-kan    <BASE>/zee_30042021_en_kn.zip   en-kn/*.en,en-kn/*.kn
Anuvaad-ik_2021-v1-eng-mar      <BASE>/ik-2021-v1-en-mr.zip     en-mr/*.en,en-mr/*.mr
Anuvaad-legal_terms_2021-v1-eng-mar     <BASE>/legal-terms-2021-v1-en-mr.zip    en-mr/*.en,en-mr/*.mr
Anuvaad-nouns_dict_2021-v1-eng-mar      <BASE>/nouns-dict-2021-v1-en-mr.zip     en-mr/*.en,en-mr/*.mr
Anuvaad-pib_2017-2020-eng-mar   <BASE>/pib_2017_2020_en_mr.zip  en-mr/*.en,en-mr/*.mr
Anuvaad-pibarchives_2015-2016-eng-mar   <BASE>/pibarchives_2015_2016_en_mr.zip  en-mr/*.en,en-mr/*.mr
Anuvaad-ie_sports-20210320-eng-mar      <BASE>/ie_sports_20210320_en_mr.zip     en-mr/*.en,en-mr/*.mr
Anuvaad-ie_news-20210320-eng-mar        <BASE>/ie_news_20210320_en_mr.zip       en-mr/*.en,en-mr/*.mr
Anuvaad-ie_lifestyle-20210320-eng-mar   <BASE>/ie_lifestyle_20210320_en_mr.zip  en-mr/*.en,en-mr/*.mr
Anuvaad-ie_entertainment-20210320-eng-mar       <BASE>/ie_entertainment_20210320_en_mr.zip      en-mr/*.en,en-mr/*.mr
Anuvaad-ie_business-20210320-eng-mar    <BASE>/ie_business_20210320_en_mr.zip   en-mr/*.en,en-mr/*.mr
Anuvaad-toi-20210320-eng-mar    <BASE>/toi_20210320_en_mr.zip   en-mr/*.en,en-mr/*.mr
Anuvaad-thewire-20210320-eng-mar        <BASE>/thewire_20210320_en_mr.zip       en-mr/*.en,en-mr/*.mr
Anuvaad-lokmat_entertainment-20210501-eng-mar   <BASE>/lokmat-entertainment_20210501_en_mr.zip  en-mr/*.en,en-mr/*.mr
Anuvaad-lokmat_news-20210501-eng-mar    <BASE>/lokmat-news_20210501_en_mr.zip   en-mr/*.en,en-mr/*.mr
Anuvaad-lokmat_lifestyle-20210501-eng-mar       <BASE>/lokmat-lifestyle_20210501_en_mr.zip      en-mr/*.en,en-mr/*.mr
Anuvaad-lokmat_sports-20210501-eng-mar  <BASE>/lokmat-sports_20210501_en_mr.zip en-mr/*.en,en-mr/*.mr
Anuvaad-lokmat_tech-20210501-eng-mar    <BASE>/lokmat-tech_20210501_en_mr.zip   en-mr/*.en,en-mr/*.mr
Anuvaad-lokmat_financial-20210501-eng-mar       <BASE>/lokmat-financial_20210501_en_mr.zip      en-mr/*.en,en-mr/*.mr
Anuvaad-lokmat_healthcare-20210501-eng-mar      <BASE>/lokmat-healthcare_20210501_en_mr.zip     en-mr/*.en,en-mr/*.mr
Anuvaad-zee-30042021-eng-mar    <BASE>/zee_30042021_en_mr.zip   en-mr/*.en,en-mr/*.mr
Anuvaad-ik_2021-v1-eng-pan      <BASE>/ik-2021-v1-en-pa.zip     en-pa/*.en,en-pa/*.pa
Anuvaad-internal_judicial_2021-v1-eng-pan       <BASE>/internal-judicial-2021-v1-en-pa.zip      en-pa/*.en,en-pa/*.pa
Anuvaad-legal_terms_2021-v1-eng-pan     <BASE>/legal-terms-2021-v1-en-pa.zip    en-pa/*.en,en-pa/*.pa
Anuvaad-nouns_dict_2021-v1-eng-pan      <BASE>/nouns-dict-2021-v1-en-pa.zip     en-pa/*.en,en-pa/*.pa
Anuvaad-pib_2017-2020-eng-pan   <BASE>/pib_2017_2020_en_pa.zip  en-pa/*.en,en-pa/*.pa
Anuvaad-pibarchives_2015-2016-eng-pan   <BASE>/pibarchives_2015_2016_en_pa.zip  en-pa/*.en,en-pa/*.pa
Anuvaad-tribune-20210320-eng-pan        <BASE>/tribune_20210320_en_pa.zip       en-pa/*.en,en-pa/*.pa
Anuvaad-pa_govt-20210320-eng-pan        <BASE>/pa_govt_20210320_en_pa.zip       en-pa/*.en,en-pa/*.pa
Anuvaad-jagran_news-20210320-eng-pan    <BASE>/jagran-news_20210320_en_pa.zip   en-pa/*.en,en-pa/*.pa
Anuvaad-jagran_tech-20210320-eng-pan    <BASE>/jagran-tech_20210320_en_pa.zip   en-pa/*.en,en-pa/*.pa
Anuvaad-jagran_education-20210320-eng-pan       <BASE>/jagran-education_20210320_en_pa.zip      en-pa/*.en,en-pa/*.pa
Anuvaad-jagran_entertainment-20210320-eng-pan   <BASE>/jagran-entertainment_20210320_en_pa.zip  en-pa/*.en,en-pa/*.pa
Anuvaad-jagran_business-20210320-eng-pan        <BASE>/jagran-business_20210320_en_pa.zip       en-pa/*.en,en-pa/*.pa
Anuvaad-jagran_sports-20210320-eng-pan  <BASE>/jagran-sports_20210320_en_pa.zip en-pa/*.en,en-pa/*.pa
Anuvaad-jagran_lifestyle-20210320-eng-pan       <BASE>/jagran-lifestyle_20210320_en_pa.zip      en-pa/*.en,en-pa/*.pa
Anuvaad-ik_2021-v1-eng-guj      <BASE>/ik-2021-v1-en-gu.zip     en-gu/*.en,en-gu/*.gu
Anuvaad-legal_terms_2021-v1-eng-guj     <BASE>/legal-terms-2021-v1-en-gu.zip    en-gu/*.en,en-gu/*.gu
Anuvaad-nouns_dict_2021-v1-eng-guj      <BASE>/nouns-dict-2021-v1-en-gu.zip     en-gu/*.en,en-gu/*.gu
Anuvaad-pib_2017-2020-eng-guj   <BASE>/pib_2017_2020_en_gu.zip  en-gu/*.en,en-gu/*.gu
Anuvaad-pibarchives_2015-2016-eng-guj   <BASE>/pibarchives_2015_2016_en_gu.zip  en-gu/*.en,en-gu/*.gu
Anuvaad-oneindia-20210320-eng-guj       <BASE>/oneindia_20210320_en_gu.zip      en-gu/*.en,en-gu/*.gu
Anuvaad-toi-20210320-eng-guj    <BASE>/toi_20210320_en_gu.zip   en-gu/*.en,en-gu/*.gu
Anuvaad-gu_govt-20210320-eng-guj        <BASE>/gu-govt_20210320_en_gu.zip       en-gu/*.en,en-gu/*.gu
Anuvaad-zee-30042021-eng-guj    <BASE>/zee_30042021_en_gu.zip   en-gu/*.en,en-gu/*.gu
Anuvaad-pib_2017-2020-eng-asm   <BASE>/pib_2017_2020_en_as.zip  en-as/*.en,en-as/*.as
Anuvaad-pibarchives_2015-2016-eng-asm   <BASE>/pibarchives_2015_2016_en_as.zip  en-as/*.en,en-as/*.as
Anuvaad-sentinel-20210320-eng-asm       <BASE>/sentinel_20210320_en_as.zip      en-as/*.en,en-as/*.as
Anuvaad-pib_2017-2020-eng-urd   <BASE>/pib_2017_2020_en_ur.zip  en-ur/*.en,en-ur/*.ur
Anuvaad-pibarchives_2015-2016-eng-urd   <BASE>/pibarchives_2015_2016_en_ur.zip  en-ur/*.en,en-ur/*.ur
Anuvaad-thewire-20210320-eng-urd        <BASE>/thewire_20210320_en_ur.zip       en-ur/*.en,en-ur/*.ur
Anuvaad-pib_2017-2020-eng-ori   <BASE>/pib_2017_2020_en_or.zip  en-or/*.en,en-or/*.or
Anuvaad-pibarchives_2015-2016-eng-ori   <BASE>/pibarchives_2015_2016_en_or.zip  en-or/*.en,en-or/*.or

P.S.
https://github.com/thammegowda/mtdata/blob/master/mtdata/index/anuvaad.py