Compliance tests and performance criteria for benchmarks

[mrasquinha-g]

Copying this over from the post mortem document ---

• Testing for GPT-J – Should we enable Test01 and/or relax the submission checker checks for the benchmark since it takes very long to run.

The problem here is not of TEST01 - rather the offline scenario is needing 24756 queries. As I understand this is not a proper requirement and we should choose this number specific to each benchmark. For example,
ResNet50 - 1024 (performance sample count) is enough as all inputs take the same time.
RetinaNet, bert, 3d-unet, gpt-j - dataset size as the processing time varies with the input