how to load diffusers metadata.jsonl ?

[intersoulest]

I want to know,how to load metadata.jsonl file.will I need to fix my datafile?metadata.jsonl Format

[bghira]

you will have to convert it to a parquet file

[intersoulest]

my multidatabackend.json setting:
[ { "id": "metadata", "type": "local", "caption_strategy": "parquet", "parquet_path":"/usr/src/sd/pattern_40w/metadata.parquet", "parquet_filename_column":"file_name", "parquet_caption_column": "text", "parquet_fallback_caption_column": "text", "disable": false, "instance_data_dir": "/usr/src/sd/pattern_40w", "crop": false, "crop_style": "corner", "crop_aspect": "preserve", "resolution": 1024, "resolution_type": "pixel", "minimum_image_size": 600, "prepend_instance_prompt": false, "instance_prompt": null, "only_instance_prompt": false, "cache_dir_vae": "/workspace/image/vaecache", "vae_cache_clear_each_epoch": true, "probability": 1.0, "repeats": 1, "skip_file_discovery": "vae,aspect,text,metadata", "preserve_data_backend_cache": false, "text_embeds": "alt-embed-cache" }, { "id": "alt-embed-cache", "dataset_type": "text_embeds", "default": false, "type": "local", "cache_dir": "/workspace/image/textembed_cache" } ]

How can I change it?

[intersoulest]

this is my new multidatabackend.json, i solve the problem,but i meet a new problem.

[bghira]

because your filename field contains the file extension (.png) it is necessary to set additional value:

        "caption_strategy": "parquet",
        "metadata_backend": "json",
        "parquet": {
            "path": "/usr/src/lattice.parquet",
            "caption_column": "text",
            "filename_column": "filename",
            "identifier_includes_extension": true
        }

[bghira]

be sure to use metadata_backend: json so that your width/heights are auto-discovered, as your jsonl file does not contain them

[bghira]

the latest main branch has parquet backend updated to also support jsonl files - however, it's experimental and untested, as i don't really have any qualifying datasets ready to test locally.

i will note that it requires width and height keys added for each sample in the jsonl backend, however you can see the /documentation/DATALOADER.md document for more information on configuring JSONL files.