Enhancements for flle naming and directors

[hschellman]

Would be good to have the ability to expand environmental and add timestamps and/or tags in filenames and directories

{$ENV} --> ENV

{DIRTIMESTAMP} -> insert a timestamp that is a member of the (scheduler class? into the directory to avoid mixing jobs

{FILETIMESTAMP} -> make it so you can put the timestamp where you want in the output filename.

{JOBTAG} -> add an optional tag on the command line so you can figure out where stuff is without editing config.

[hschellman]

actually - it would be good if main established a global time stamp (or hash) for a run of merge that could go into output directories and possibly the tag for the run. That way every run of merge has a unique identified.

A combination of tag and that hash/timestamp would then be used for the tmp// for submission and also for the local output directory and possibly filenames.

that way any run of merge has a unique identified.

I do not understand the syntax for setting a global variable that all of the code could understand - it would go into merge.tag in the output metadata but right now I would have to access it as (for example). I think it might be better to just add it to the tag and use that tag+stamp in a lot of places where you are dealing with multiple files, including the job name.

if 'overrides' in config.metadata and 'merge.tag' in config.metadata['overrides']:
out_dir = os.path.join(out_dir, config.metadata['overrides']['merge.tag'])

since I don't know where to put it in config.

[EthanMuldoon]

I’m putting ’skip’, ‘limit’, ’tag’, and ‘comment’ as options in config.input, and they’ll get added to the output metadata with the ‘merge.’ prefix. I think a comment field might be useful for adding in a human-readable explanation of what we’re doing, especially for some of these jobs that aren’t strictly merging. I made the timestamp a global variable and let you put ’{TIMESTAMP}’ into file name specifications. But maybe we should just put a ‘merge.timestamp’ into the output metadata instead? From: Heidi Schellman ***@***.***> Date: Tuesday, November 11, 2025 at 10:20 AM To: DUNE/merge-utils ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [DUNE/merge-utils] Enhancements for flle naming and directors (Issue #25) [https://avatars.githubusercontent.com/u/33669005?s=20&v=4]hschellman left a comment (DUNE/merge-utils#25)<#25 (comment)> actually - it would be good if main established a global time stamp (or hash) for a run of merge that could go into output directories and possibly the tag for the run. That way every run of merge has a unique identified. A combination of tag and that hash/timestamp would then be used for the tmp// for submission and also for the local output directory and possibly filenames. that way any run of merge has a unique identified. I do not understand the syntax for setting a global variable that all of the code could understand - it would go into merge.tag in the output metadata but right now I would have to access it as (for example). I think it might be better to just add it to the tag and use that tag+stamp in a lot of places where you are dealing with multiple files, including the job name. if 'overrides' in config.metadata and 'merge.tag' in config.metadata['overrides']: out_dir = os.path.join(out_dir, config.metadata['overrides']['merge.tag']) since I don't know where to put it in config. — Reply to this email directly, view it on GitHub<#25 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA7Y7EIZYE7YKRZ66RFZPLL34ISIJAVCNFSM6AAAAACLWDLBFSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKMJYGIYTGMZTHA>. You are receiving this because you are subscribed to this thread.

[hschellman]

The main idea here is to be able to produce unique output directories and tags for each run of the merging process. Because people are going to run them multiple times (especially as they may fail) and we need to be able to separate them. A combo of tag and timestamp likely works. The global timestamp is useful for things like local output directories and as a tag for a run over a dataset. It may not belong in file names as they need their unique timestamp for other purposes. So local output dir = dir from config + tag + timestamp Local submitdir = tmp/tag+timestamp But I just had to run in 2 chunks because rucio fails so often and they need to share the same tag for me to be able to merge the outputs later. (Maybe we need to store the input dataset in some way, wonder if they have ID’s in the same way files do? - looks like no. If datasets had unique short id's we could store those instead of the dataset itself. ) Ie merge.datasetid= …. Could be used to recover the dataset without storing 200 characters in each file. That’s a metacat improvement. Heidi Schellman Oregon State Physics Chair, APS Division of Particles and Fields Executive Committee On Nov 11, 2025, at 6:23 PM, Ethan Muldoon ***@***.***> wrote: [This email originated from outside of OSU. Use caution with links and attachments.] [https://avatars.githubusercontent.com/u/4165521?s=20&v=4]EthanMuldoon left a comment (DUNE/merge-utils#25)<#25 (comment)> I’m putting ’skip’, ‘limit’, ’tag’, and ‘comment’ as options in config.input, and they’ll get added to the output metadata with the ‘merge.’ prefix. I think a comment field might be useful for adding in a human-readable explanation of what we’re doing, especially for some of these jobs that aren’t strictly merging. I made the timestamp a global variable and let you put ’{TIMESTAMP}’ into file name specifications. But maybe we should just put a ‘merge.timestamp’ into the output metadata instead? From: Heidi Schellman ***@***.***> Date: Tuesday, November 11, 2025 at 10:20 AM To: DUNE/merge-utils ***@***.***> Cc: Subscribed ***@***.***> Subject: Re: [DUNE/merge-utils] Enhancements for flle naming and directors (Issue #25) [https://avatars.githubusercontent.com/u/33669005?s=20&v=4]hschellman left a comment (DUNE/merge-utils#25)<#25 (comment)> actually - it would be good if main established a global time stamp (or hash) for a run of merge that could go into output directories and possibly the tag for the run. That way every run of merge has a unique identified. A combination of tag and that hash/timestamp would then be used for the tmp// for submission and also for the local output directory and possibly filenames. that way any run of merge has a unique identified. I do not understand the syntax for setting a global variable that all of the code could understand - it would go into merge.tag in the output metadata but right now I would have to access it as (for example). I think it might be better to just add it to the tag and use that tag+stamp in a lot of places where you are dealing with multiple files, including the job name. if 'overrides' in config.metadata and 'merge.tag' in config.metadata['overrides']: out_dir = os.path.join(out_dir, config.metadata['overrides']['merge.tag']) since I don't know where to put it in config. — Reply to this email directly, view it on GitHub<#25 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA7Y7EIZYE7YKRZ66RFZPLL34ISIJAVCNFSM6AAAAACLWDLBFSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKMJYGIYTGMZTHA>. You are receiving this because you are subscribed to this thread. — Reply to this email directly, view it on GitHub<#25 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AIA37DJMLHIWTABELJ3ADPD34KK25AVCNFSM6AAAAACLWDLBFSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTKMJZGU4DQNZQHE>. You are receiving this because you were assigned.Message ID: ***@***.***>