preprocess.py script does not create subset entries if default values are used

[willu47]

When using the preprocess.py script on a datafile which has been converted using otoole, we found that subsets were not being generated for entries in InputActivityRatio and OutputActivityRatio whose value is equal to the default value (zero in most cases). This causes an error downstream when using glpsol to create the model as there are inconsistent subsets.

In my mind - if InputActivityRatio and OutputActivityRatio are zero, then the equations should not be generated anyway - the technologies will not consume or produce anything and can reduce to zero.

On the other hand, if we need subsets produced for zero valued combinations, then how to make this robust?

I'm not sure if this is something that should be address in i) the model formulation ii) otoole iii) our model definition

[willu47]

I note that in the fast code, the modex sets are not created for zero-valued entries.

[abhishek0208]

Hi @willu47, I don't think I quite understand the issue. The script creates subsets in 'MODExTECHNOLOGYperFUELin and 'MODExTECHNOLOGYperFUELout only for TECHNOLOGY-FUEL combinations that exist (i.e. InputActivtyRatio and OutputActivityRatio <> 0, respectively). MODEperTECHNOLOGY is a union of the above two MODEx* sets, where modes for each TECHNOLOGY that have either an InputActivityRatio or OutputActivityRatio <>0 are listed. Am I missing something?

[abhishek0208]

Following on from that, are there cases when we'd need subsets produced for zero valued combinations?

[willu47]

Hi @willu47, I don't think I quite understand the issue. The script creates subsets in 'MODExTECHNOLOGYperFUELin and 'MODExTECHNOLOGYperFUELout only for TECHNOLOGY-FUEL combinations that exist (i.e. InputActivtyRatio and OutputActivityRatio <> 0, respectively). MODEperTECHNOLOGY is a union of the above two MODEx* sets, where modes for each TECHNOLOGY that have either an InputActivityRatio or OutputActivityRatio <>0 are listed. Am I missing something?

I don't think that's the case? For the momani code (which I haven't tested), then line 156 writes to output_table, but doesn't seem to be used later. I see now checks for the values of output and input activity ratios. For the otoole code, I've added these in #69.

Following on from that, are there cases when we'd need subsets produced for zero valued combinations?

No, we never need subsets for zero valued combinations. This issue is that because there was no check of zero values, it created empty subsets (which was confusing). The main breaking error was the problem of emission sets.

[abhishek0208]

Ah, I figured out what's causing this issue. I had removed a line of code as part of an earlier bug fix that introduced this new bug. The script is now fixed, tested, and pushed up to the repo (fe98533) Thanks @willu47.
output_table is a remnant from when the preprocessing script was used linked to results processing. output_table is used to calculate ProductionByTechnology from RateOfActivity. So it's not strictly needed here.