.net 6/7 branch and new record types

[mgoldste1]

I figured we should probably move over here to chat.

Links for anyone else who finds this.

• stdf 2007 rev draft - https://docplayer.net/127928234-Background-statement-for-semi-draft-document-4782-new-standard-specification-for-standard-test-data-format-stdf-memory-fail-datalog.html
• Probably the official 2007 revision's spec - https://github.com/lantianjialiang/stdf_spec/blob/master/stdf_V4-2007_spec.pdf
• A paper that goes into detail on the changes from STDF V4-2007. https://ieeexplore.ieee.org/document/6252114

I'm still trying to get approvals to push my changes to the repo. We aren't used to using industry standards so management doesn't really know how to handle this kind of thing from a liability standpoint. I showed them the microsoft public license and I've been waiting to hear back.

I switched to your .net 7 branch and made it work for .net 6. We aren't allowed to use .net 7 in prod yet. Not much needed to change there. It was just removing some required modifiers.

I added...

1. Some converters so it can write to json/xml and added a couple ToSTDF methods/extensions to make writing stdfs a little easier. Problem with this is that it required me to litter the record types with jsonignore/xmlignore attributes to make it output better and it does required newtonsoft.json now.
2. Some summary methods which I don't know if they'd be useful in the main library or not. Probably more of a sample app kinda thing. It makes CSVs with a summary of what kind of records are in it and the general structure.
3. A full name to each record type so we can pull a friendly name out of the record.
4. A lot of documentation, but haven't finished that yet.
5. VUR record parsing
6. A dirty PSR record parsing class... The way I did it, it doesn't use the flagged field layout because it doesn't appear to support the contents of PSR records. I think it needs a flagged array field layout attribute. The spec for these records also shows some bits in the optional flag that aren't associated to another field. I think that is different than the other flagged field layouts used in the project. My dirty PSR reading class showed that while our stdf files have PSR records in them, they aren't really used for anything so this might be a waste of time.

One thing I noticed is that when I read the 200mb stdf file in and immediately write it back out, the file size shrinks by 10kb. Since my deliverables don't require me to write stdf files I may or may not do an audit to see what changed.

[mgoldste1]

I think I got PSR records working the correct way. It reads and writes the data in my Psr records but since they are mostly empty I can't really tell if it actually works.

All I did was make a FlaggedArrayFieldLayoutAttribute class and inherit from ArrayFieldLayoutAttribute. Then move the Flagged field attribute variables to an interface. The FlaggedArrayFieldLayoutAttribute and the FlaggedFieldLayoutAttribute implement that interface. Then in the converters/unconverters, for anything checking if the type was a FlaggedFieldLayoutAttribute, I changed it to the interface. Surprisingly it compiled first try and I am shocked tbh. It's almost like I know what I'm doing.

[marklio]

Cool! A couple of questions/comments:

• Why would you transcode STDF to/from JSON? I'd like to understand that scenario better. This reminds me of the use of ATDF (ASCII version of STDF spec). Human readable, but MUCH larger.
• I'd prefer to keep summary/formatting stuff out of the library. There's not alot of internals that prevents implementing these external to the parsing library.
• Full name sounds like a good improvement for logging/UI scenarios. I definitely support that.

In terms of the 10 size shrink, I'd be interested to see what's missing, but this could be totally expected. When writing, my library makes full use of not writing empty fields at the end of a record. That said, writing isn't well-tested. I would be very interested in getting my hands on a 200MB STDF file for testing purposes. I'd be happy to sign any NDA or other agreement to not share it.

In my IO improvement branch, I'm finding myself really gutting things, so that's going slow. I'm also resisting the urge to rewrite the ref.emit codegen since I want to move that to source generators. That means I'm making lots of intermediate concessions. Also, the modern .NET world is async, but all the nice LINQ stuff is synchronous, so I'm trying to find a model that plays nicely in both worlds. It probably means a more intention split between "record streaming" scenarios where records are processed ephemerally, and true "query" scenarios where the structure of the file is traversed.

[mgoldste1]

Turning it into json/xml was just so I could see what is in the records in a more human readable way. My task with STDF files is to convert them to our own format so I'm going to need to match data from STDF files to where they would be located in our own files. Exporting an STDF file to JSON just makes my life easier when I'm trying to find something. I am not reading the json/xml back into stdf files.

I can ask about the NDA stuff, but seeing as I'm having difficulty getting permission to contribute stuff that has zero IP in it, it's probably not going to happen. If they do let me contribute I can strip the json/xml stuff out before sending it on over.

[marklio]

All makes sense. Totally understand the reluctance to share data or engage in open source. I'm totally open to choosing a different software license if that helps. The one that is there is fairly old. It was just what MS was recommending years ago when I first published the code.

[mgoldste1]

Still no word on if I can publish it. I think I'm having the same problems you do with the whole testing aspect. Without stdf files with each of the record types used to the fullest, it's quite hard to make sure it works properly...

I finished the documentation today. It took literal days but every record type and variable has an explanation for what it's for. It's formatted in a way where the tooltips look nice and you can still read it from classes. About 8000 lines were added to the repo when I checked it into my personal one...

Regarding the license... Do you think we could switch to the MIT License? It seems to be the standard these days and the text very straight forward. Also could we put your copyright stuff in the readme or license file instead of at the top of every document? When you add a license to the project it will prompt you for your name and automatically put it in there. You can add the license by adding a file to the project from the git repo's webpage with the name "LICENSE" in all caps. It then knows what you are trying to do and will let you pick one from a list.

[marklio]

Yes. This seems very reasonable. MIT is fine, and doing it the way everyone does it these days is good.

[mgoldste1]

I submitted the request to my company's open source group so I can contribute, but they requested this license change request to be done before they review it further.

[marklio]

Understood. I support this. Given holidays and other stuff I'm doing, I'm unlikely to make much progress till the new year.

[mgoldste1]

I found the right group to review my request to contribute here. I just don't have the time to do what they are asking yet. Hopefully I can get the approvals after I get a prototype of my converter out.

BTW, reading in a 1.4gb stdf file with 14.3 million records took 32 seconds and ate up 8gb of ram. Lot of ram but the parsing time is pretty good imo. I'm designing the config mappings between the stdf file and our custom format now so hopefully I will be able to get some more metrics soon. I have to take a text based "WIR.WAFER_ID" kinda config and get the data out of it so hopefully reflection doesnt slow things down too much.

[marklio]

Cool. What API are you using to read? I'll bet you're in a situation where it's using one of the "indexing" strategies that caches the entire file, and my approaches 15 years ago were not terribly compact :). There are patterns that allow direct streaming of the records, which should be much lighter weight on RAM. I don't have time to find the right pattern. I know you can specify a non-caching strategy directly, but I think there's an API that will just stream the records to you. My experiments with new IO approaches allow REALLY high speed parsing (and they do away with the Reflection.emit stuff), and are low-allocation. THe problem right now is that you have to buy into async. I'm chatting with some folks to see if I can get the IO benefits without forcing async. All our new IO stuff is async-first, which makes sense for a web server, but is silly for an essentially single-threaded tool.

[mgoldste1]

I've played around with the indexing strategies more with the stdf to json code. A getRecordsEnumerable() with a non cache strat is a bit faster than the default one and only uses 29mb of ram. That went from 168 seconds to 138 seconds. When I was filtering to exclude the 14 million dtr records, the time to convert to json with noncache went down to 20 seconds.

For reading it into ram I used the getRecordsEnumerable().ToList() and also getRecords.ToList(). Same result. I also did a .Consume() with both indexing strategies to see the difference there. Noncache was about 20 seconds and cache was still in the 32seconds to 35 second range. When I pulled just the specific information I needed out of the ptr records, that alone took 25 seconds with the non-cache and ate a couple gigs of ram. After I get the configs ironed out I'll start playing around more with performance related tests.

And yea I'm completely with you on the whole async stuff. It's great for my discord bots but for this? It doesn't seem necessary.