Update swmm-python to latest OWA SWMM #94

karosc · 2022-01-07T14:36:26Z

This pull request updates the swmm-solver submodule to the latest OWA SWMM commit, incoperating the addition of potential evaporation system attribute and enhanced pollutant setters from @abhiramm7 and @bemason

Addresses #92, #91, and #87.

- add P_EVAP_RATE to output_enum and metadata - update swmm-solver submodule to latest commit on develop branch

- define EXPORT_TOOLKIT in solver.i

- change swmm_getVersionInfo -> swmm_getSemVersion

- needed to add swig exception handling as with SMO_init and SMO_close

- Update shared enum with changes from toolkit_enums.h - update solver rename for two new pollutant setter methods - add PEP8 swmm_getBuildId

michaeltryby

@karosc Thanks for creating this PR. Looks great! I recommend a few minor changes.

swmm-toolkit/src/swmm/toolkit/shared_enum.py

michaeltryby · 2022-01-07T16:34:00Z

swmm-toolkit/src/swmm/toolkit/shared_enum.py

@@ -447,6 +450,8 @@ class NodePollutant(Enum):
    :attr:`~QUALITY`
    """
    QUALITY = 0
+    INF_CONC = 1


Make more descriptive - INFLOW_CONC

Thanks @karosc for making these changes! We also need to add the reactor concentration -- we added both inflow concentration and reactor concentration to SWMM:
SM_NODECIN = 1, /< Inflow Concentration */
SM_NODEREACTORC = 2 /< Reactor Concentation */
See the old pull request for more details: pyswmm/Stormwater-Management-Model#326.

And then do we need to write tests to test all three of these additions?

Yes, we should add tests, but I haven't thought about how to design the tests yet. I still have not actually used the new setters in any context. Should we just start a simulation with pollutants at 0, set high concentrations at runtime, then check that they are not zero after a few time steps? What would be your approach here?

I haven't exactly figured out the best way to do this either. I didn't write the tests for these additions when we edited SWMM, @abhiramm7 did.

Here is how inflow concentration was tested: https://github.com/OpenWaterAnalytics/Stormwater-Management-Model/blob/76b8ae7e39f07a659b51284effafe668394d7f29/tests/solver/test_pollut.cpp#L151-L183

Here is how reactor concentration was tested: https://github.com/OpenWaterAnalytics/Stormwater-Management-Model/blob/76b8ae7e39f07a659b51284effafe668394d7f29/tests/solver/test_pollut.cpp#L185-L222
Couldn't find where hrt was tested. Not sure if this helps of now? Hopefully @abhiramm7 has some ideas!

swmm-toolkit/src/swmm/toolkit/solver.i

swmm-toolkit/tests/test_metadata.py

- make shared_enum vars more descriptive - move EXPORT_TOOLKIT define to CMakeLists - fix some formatting and docstrings

michaeltryby

@karosc Looks good!

jennwuu · 2022-01-10T16:15:25Z

@michaeltryby Is this the current mapping for branches in OWA/SWMM and OWA/swmm-python?

OWA/swmm-python dev => OWA/SWMM develop
OWA/swmm-python master => OWA/SWMM master

michaeltryby · 2022-01-10T16:36:57Z

@jennwuu yes that is correct. Releases belong in the master branch. Ongoing development occurs in the develop branch.

jennwuu · 2022-01-10T16:40:41Z

Thank you for confirming!

@karosc I think this PR is ready for merge whenever you're ready.

karosc · 2022-01-10T18:43:52Z

I'm okay to merge unless we want to finalize unit tests for the pollutant setters first. I have not even tried them myself yet, so I don't actually know if they work...probably something we want. However, swig does the heavy lifting here, so if there are bugs, I image they are most likely on the SWMM API side 😬.

jennwuu · 2022-01-10T18:50:56Z

@karosc Maybe let's hold off until we finalize the unit tests first.

abhiramm7 · 2022-01-10T19:24:09Z

Thanks for all your work on this @karosc. I can review the PR and get back to you by tomorrow if that is ok.

abhiramm7

Hi @karosc, this looks great! Thanks for all your work :)

The only thing I would recommend is adding unit tests. We have unit tests for pollutant getters. For completion and best practice, it would be better to have some tests in swmm-python to detect if something breaks in owa-swmm.

I think that we could add a few tests

Pollutant Getters: Set pollutant in node and get the pollutant in the downstream node after a couple of iterations. We could repeat the same thing for links.
HRT and other params: We could add a test to see if they are being pulled. We don't have to check the value as they are computed in SWMM.

Let me know what you think 😁

karosc · 2022-01-16T06:37:48Z

@abhiramm7, I took a crack at writing some unit tests, but am having some trouble getting the node pollutant setter to work, the link pollutant setter seems to work fine. If I set a very high link pollutant concentration upstream, I can see that large mass show up upstream and downstream in the simulation. However, if I try the same thing with the node setter, it doesn't work. For some reason, it doesn't look like I can get python to inject pollutant mass into a node. Am I missing something here? The sample scripts below can be run from the swmm-toolkit/tests directory.

Also while debugging this, I noticed that the swmm_setNodePollut takes 3 args and the swmm_setLinkPollut takes 4, was that intentional? Also, the type argument to the swmm_setLinkPollut function is an int rather than an SM_LinkPollut enum, consequently if I try to use the python shared_enum.LinkPollutant.QUALITY for that arg, I get a type error and need to instead just use 0. Thoughts?

Link setter success (links show super high concentration)

In [12]: import os
    ...: from swmm.toolkit import solver, shared_enum
    ...: 
    ...: DATA_PATH="data"
    ...: INPUT_FILE_EXAMPLE_1 = os.path.join(DATA_PATH, 'test_Example1.inp')
    ...: REPORT_FILE_TEST_1 = os.path.join(DATA_PATH, 'temp_Example.rpt')
    ...: OUTPUT_FILE_TEST_1 = os.path.join(DATA_PATH, 'temp_Example.out')
    ...: 
    ...: 
    ...: solver.swmm_open(INPUT_FILE_EXAMPLE_1, REPORT_FILE_TEST_1, OUTPUT_FILE_TEST_1)
    ...: us = 0 # link 1
    ...: ds = 3# link 6
    ...: leadidx = 1
    ...: three_hours = 60 * 3
    ...: solver.swmm_start(0)
    ...: print(solver.link_get_pollutant(us,shared_enum.LinkPollutant.QUALITY))
    ...: 
    ...: for i in range(three_hours):
    ...:     solver.link_set_pollutant(us,0,leadidx,1000000.0)
    ...:     solver.swmm_step()
    ...:     print(solver.link_get_pollutant(us,shared_enum.LinkPollutant.QUALITY))
    ...: 
    ...: 
    ...: solver.swmm_end()
    ...: solver.swmm_close()
    ...: 

 o  Retrieving project data[0.0, 0.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[0.0, 1000000.0]
[15.8082717941059, 1000000.0]
[15.8082717941059, 1000000.0]
[15.8082717941059, 1000000.0]
[15.8082717941059, 1000000.0]
[15.8082717941059, 1000000.0]
[15.8082717941059, 1000000.0]
[15.8082717941059, 1000000.0]
[15.8082717941059, 1000000.0]
[15.8082717941059, 1000000.0]
[15.8082717941059, 1000000.0]
[15.8082717941059, 1000000.0]
[15.808271794105899, 1000000.0]
[15.8082717941059, 1000000.0]
[15.808271794105899, 1000000.0]
[15.808271794105899, 1000000.0]
[15.807531255216857, 1000000.0]
[15.80648979352922, 1000000.0]
[15.805404986609942, 1000000.0]
[15.804398301034363, 1000000.0]
[15.803509580784624, 1000000.0]
[15.802746461979801, 1000000.0]
[15.802092781256203, 1000000.0]
[15.801535884766382, 1000000.0]
[15.801061368425138, 1000000.0]
[15.800656130135943, 1000000.0]
[15.800306857416365, 1000000.0]
[15.800003019077405, 1000000.0]
[15.79973773887079, 1000000.0]
[15.799504543214486, 1000000.0]
[15.799298444119998, 1000000.0]
[15.797895629705238, 1000000.0]
[15.795855931839371, 1000000.0]
[15.79348218978959, 1000000.0]
[15.79096156373117, 1000000.0]
[15.78839834339483, 1000000.0]
[15.785855927822857, 1000000.0]
[15.783352660056128, 1000000.0]
[15.780911707951697, 1000000.0]
[15.77854099673189, 1000000.0]
[15.776247681095096, 1000000.0]
[15.774019746081109, 1000000.0]
[15.771866952930775, 1000000.0]
[15.769776661852093, 1000000.0]
[15.767752493757696, 1000000.0]
[15.765792023146195, 1000000.0]
[15.763286499640362, 1000000.0]
[15.760478253633321, 1000000.0]
[15.757508201208289, 1000000.0]
[15.754453321188564, 1000000.0]
[15.75135978108337, 1000000.0]
[15.74825393520306, 1000000.0]
[15.745150606366186, 1000000.0]
[15.74205806221766, 1000000.0]
[15.738980866816068, 1000000.0]
[15.735921495999268, 1000000.0]
[15.732881249638426, 1000000.0]
[15.729860766260673, 1000000.0]
[15.726860313178173, 1000000.0]
[15.723879949967134, 1000000.0]
[15.720919620532989, 1000000.0]
[15.716623777496311, 1000000.0]
[15.711884770276779, 1000000.0]
[15.707169593516017, 1000000.0]
[15.702675267157014, 1000000.0]
[15.698474945427195, 1000000.0]
[15.694597422155036, 1000000.0]
[15.691025982226918, 1000000.0]
[15.687738681179328, 1000000.0]
[15.68471053566842, 1000000.0]
[15.681916012896105, 1000000.0]
[15.679330667583795, 1000000.0]
[15.67693040274615, 1000000.0]
[15.674702019176205, 1000000.0]
[15.672622828127277, 1000000.0]
[15.670682711556902, 1000000.0]
[15.666553222549584, 1000000.0]
[15.661330650091852, 1000000.0]
[15.655564264104518, 1000000.0]
[15.64954897753535, 1000000.0]
[15.643437181956106, 1000000.0]
[15.637306937671037, 1000000.0]
[15.631197946540574, 1000000.0]
[15.625130157962479, 1000000.0]
[15.61911336317999, 1000000.0]
[15.613152137644844, 1000000.0]
[15.607248384970184, 1000000.0]
[15.60140264530985, 1000000.0]
[15.595614693574168, 1000000.0]
[15.589883987836092, 1000000.0]
[15.58420966102963, 1000000.0]
[15.578096990015862, 1000000.0]
[15.571764154378544, 1000000.0]
[15.565324304528882, 1000000.0]
[15.558837823449561, 1000000.0]
[15.552335633868262, 1000000.0]
[15.545833501044962, 1000000.0]
[15.539339443927124, 1000000.0]
[15.532857525590423, 1000000.0]
[15.526389789465432, 1000000.0]
[15.519937248131352, 1000000.0]
[15.513500388177935, 1000000.0]
[15.50707942793217, 1000000.0]
[15.500674449000694, 1000000.0]
[15.494285463395734, 1000000.0]
[15.487912447782783, 1000000.0]
[15.481407536539889, 1000000.0]
[15.474841526779207, 1000000.0]
[15.468250644179774, 1000000.0]
[15.461653522034396, 1000000.0]
[15.455059641699265, 1000000.0]
[15.448473813834843, 1000000.0]
[15.44189846945804, 1000000.0]
[15.435334827556371, 1000000.0]
[15.42878348992133, 1000000.0]
[15.42224474413335, 1000000.0]
[15.415718717868378, 1000000.0]
[15.409205457473107, 1000000.0]
[15.402704967970827, 1000000.0]
[15.396217233427087, 1000000.0]

Node setter error (nodes don't show super high concentration)

In [14]: import os
    ...: from swmm.toolkit import solver, shared_enum
    ...: 
    ...: DATA_PATH="data"
    ...: INPUT_FILE_EXAMPLE_1 = os.path.join(DATA_PATH, 'test_Example1.inp')
    ...: REPORT_FILE_TEST_1 = os.path.join(DATA_PATH, 'temp_Example.rpt')
    ...: OUTPUT_FILE_TEST_1 = os.path.join(DATA_PATH, 'temp_Example.out')
    ...: 
    ...: 
    ...: solver.swmm_open(INPUT_FILE_EXAMPLE_1, REPORT_FILE_TEST_1, OUTPUT_FILE_TEST_1)
    ...: us = 1 # node 10
    ...: ds = 9 # node 21
    ...: leadidx = 1
    ...: three_hours = 60 * 3
    ...: solver.swmm_start(0)
    ...: print(solver.node_get_pollutant(us,shared_enum.NodePollutant.QUALITY))
    ...: 
    ...: for i in range(three_hours):
    ...:     solver.node_set_pollutant(us,leadidx,1000000.0)
    ...:     solver.swmm_step()
    ...:     print(solver.node_get_pollutant(us,shared_enum.NodePollutant.QUALITY))
    ...: 
    ...: 
    ...: solver.swmm_end()
    ...: solver.swmm_close()
    ...: 

 o  Retrieving project data[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[0.0, 0.0]
[50.60756816992596, 10.121513633985193]
[49.58849133158248, 9.917698266316497]
[45.9346759177738, 9.18693518355476]
[42.15699919616713, 8.431399839233425]
[39.41944460183664, 7.88388892036733]
[37.64281457101643, 7.5285629142032855]
[36.514315425559644, 7.302863085111929]
[35.7852485907924, 7.15704971815848]
[35.29706349785949, 7.059412699571897]
[34.95554943750723, 6.991109887501446]
[34.70588891658058, 6.941177783316116]
[34.51603570256265, 6.90320714051253]
[34.36683449997846, 6.873366899995693]
[34.24643662931581, 6.849287325863162]
[34.12514166079317, 6.825028332158636]
[33.23635335570479, 6.6472706711409595]
[32.162271115381316, 6.432454223076264]
[31.12525557456024, 6.225051114912049]
[30.246634319095477, 6.049326863819096]
[29.396321864452766, 5.879264372890554]
[28.83330295161961, 5.766660590323922]
[28.37933750523137, 5.675867501046275]
[27.996958603944808, 5.5993917207889625]
[27.600468595012373, 5.5200937190024755]
[27.286657599464206, 5.4573315198928425]
[27.102760657635347, 5.42055213152707]
[26.901648756385335, 5.380329751277067]
[26.7139424717934, 5.342788494358681]
[26.483342796263965, 5.296668559252794]
[26.314006771903447, 5.26280135438069]
[26.0807072022937, 5.21614144045874]
[26.02104838762927, 5.204209677525855]
[25.933597393028208, 5.186719478605641]
[25.844596742015213, 5.168919348403043]
[25.70410200351494, 5.140820400702989]
[25.665996997226372, 5.133199399445275]
[25.59220164201166, 5.1184403284023325]
[25.514690070867793, 5.102938014173559]
[25.386387905748432, 5.077277581149688]
[25.357678874616646, 5.071535774923331]
[25.239648213328568, 5.047929642665713]
[25.21842403986121, 5.043684807972242]
[25.158393051229343, 5.031678610245869]
[25.09510891185337, 5.019021782370674]
[25.00761150280915, 5.001522300561829]
[24.940186212652687, 4.988037242530538]
[24.887767235359487, 4.977553447071899]
[24.862483059081352, 4.97249661181627]
[24.840825689080898, 4.968165137816179]
[24.819734142639664, 4.9639468285279325]
[24.798821199124614, 4.959764239824923]
[24.778044616118372, 4.955608923223675]
[24.757403270962428, 4.951480654192486]
[24.73689897591452, 4.947379795182904]
[24.7165325305408, 4.9433065061081605]
[24.696303818460127, 4.939260763692025]
[24.676212168733674, 4.935242433746735]
[24.65625660007548, 4.931251320015095]
[24.636435963744223, 4.927287192748844]
[24.61674902463763, 4.923349804927526]
[24.528438186804205, 4.905687637360841]
[24.289032197271037, 4.857806439454208]
[24.05412508342486, 4.810825016684973]
[23.865342074255672, 4.773068414851133]
[23.653299816361326, 4.730659963272265]
[23.47294501467478, 4.694589002934957]
[23.311019425385183, 4.662203885077036]
[23.164226664297008, 4.632845332859402]
[23.028209582611197, 4.605641916522239]
[22.90051281604452, 4.580102563208904]
[22.802791689089258, 4.5605583378178505]
[22.676759293667008, 4.535351858733401]
[22.593006157542725, 4.518601231508544]
[22.47994126015107, 4.495988252030213]
[22.389786574726642, 4.4779573149453284]
[22.300626367053706, 4.460125273410741]
[22.269880124313683, 4.453976024862736]
[22.24207529300394, 4.448415058600786]
[22.2144307943557, 4.44288615887114]
[22.18692196198932, 4.437384392397864]
[22.15961182264838, 4.431922364529676]
[22.132535132862508, 4.4265070265725015]
[22.105708061729622, 4.421141612345924]
[22.079137017783747, 4.415827403556748]
[22.052823410143105, 4.410564682028621]
[22.026766112222777, 4.405353222444554]
[22.000873125898806, 4.40017462517976]
[21.97479330098877, 4.394958660197754]
[21.949686816827523, 4.389937363365504]
[21.924637313176056, 4.384927462635211]
[21.901526686434245, 4.380305337286849]
[21.886091764065746, 4.3772183528131485]
[21.870897561791082, 4.374179512358217]
[21.85568920715755, 4.371137841431509]
[21.840483972460518, 4.368096794492103]
[21.82529672571536, 4.365059345143073]
[21.81013525702953, 4.3620270514059065]
[21.79500349921715, 4.35900069984343]
[21.77990340523417, 4.355980681046834]
[21.764835916844458, 4.352967183368892]
[21.74980145976363, 4.349960291952726]
[21.734800196524006, 4.346960039304802]
[21.719832155578253, 4.343966431115651]
[21.704897297202074, 4.340979459440415]
[21.689995547128305, 4.337999109425661]
[21.675139883301778, 4.335027976660356]
[21.660774665597373, 4.332154933119475]
[21.646418296688424, 4.329283659337685]
[21.63206752678528, 4.326413505357057]
[21.617730781955903, 4.32354615639118]
[21.603412744874873, 4.320682548974976]
[21.589115789771107, 4.317823157954222]
[21.57484109819632, 4.314968219639264]
[21.560589243508687, 4.312117848701738]
[21.546360489335083, 4.309272097867016]
[21.53215494164818, 4.306430988329636]
[21.517972626231398, 4.30359452524628]
[21.50381352813161, 4.300762705626322]
[21.489677611749297, 4.29793552234986]

bemcdonnell · 2022-01-16T14:52:42Z

@karosc ans @abhiramm7, I just looked at how set node pollut was done in swmm. There might be a better method. Take a look at how I wrote set_node_inflow

https://github.com/OpenWaterAnalytics/Stormwater-Management-Model/blob/76b8ae7e39f07a659b51284effafe668394d7f29/src/solver/toolkit.c#L2800

I think you can accomplish inflows with the same design. My suspicion is that overriding the mass as you are doing it in the Node structure might be missing a flow rate to accompany the mass rate.

abhiramm7 · 2022-01-16T17:11:39Z

Looking at the code for setting node pollutants, I think a couple of things might be happening:

Node pollutant setter uses swmm index to determine where to set the pollutant. You seem to be directly passing the ID, unless I am mistaken.
In the example you are using, there are a lot of inter-connections and pollutant sources. So I would recommend a simple test case to pinpoint issues.

I've attached a python script and input file in the zip file. I used to test the set node pollutant feature. It is the same file used in the owa-swmm tests.

import numpy as np
import matplotlib.pyplot as plt
from swmm.toolkit import solver, shared_enum

INP = "./data/node_constantinflow_constanteffluent.inp"
RPT = "./data/node_constantinflow_constanteffluent.rpt"
OUT = "./data/node_constantinflow_constanteffluent.out"

mean = 10.0
steps = 200

solver.swmm_open(INP, RPT, OUT)
solver.swmm_start(0)

upstream_node = "Tank"
downstream_node = "Outfall"
pollutant = "P1"

upstream_node = solver.project_get_index(shared_enum.ObjectType.NODE, upstream_node)
downstream_node = solver.project_get_index(shared_enum.ObjectType.NODE, downstream_node)
pollutant = solver.project_get_index(shared_enum.ObjectType.POLLUT, pollutant)

data = {
    "upstream_node": [],
    "downstream_node": []
}

influent = np.linspace(0, 2, steps) * np.pi

for i in range(0, steps):
    solver.node_set_pollutant(upstream_node, pollutant, 10.0 + np.sin(influent[i]))
    solver.swmm_step()
    data["upstream_node"].append(solver.node_get_pollutant(upstream_node, shared_enum.NodePollutant.QUALITY)[pollutant])
    data["downstream_node"].append(solver.node_get_pollutant(downstream_node, shared_enum.NodePollutant.QUALITY)[pollutant])

solver.swmm_end()
solver.swmm_close()

plt.figure(dpi=100)
plt.plot(data["upstream_node"], label="Up stream Node")
plt.plot(data["downstream_node"], label="Down stream Node")
plt.ylabel("Pollutant Concentration")
plt.legend()
plt.show()

test_pollutants.zip

Hope this helps :)

abhiramm7 · 2022-01-16T17:38:10Z

The issue is coming from the function declaration for the link pollutant function call. Right now, it is

EXPORT_TOOLKIT int swmm_setLinkPollut(int index, int type, int pollutant_index, double pollutant_value)

it should be

EXPORT_TOOLKIT int swmm_setLinkPollut(int index, SM_LinkPollut type, int pollutant_index, double pollutant_value)

Until we patch it, it should be very minor fix, we can do this

solver.link_set_pollutant(link, shared_enum.LinkPollutant.QUALITY.value, pollutant, 10.0)

abhiramm7 · 2022-01-16T17:40:15Z

@karosc ans @abhiramm7, I just looked at how set node pollut was done in swmm. There might be a better method. Take a look at how I wrote set_node_inflow

https://github.com/OpenWaterAnalytics/Stormwater-Management-Model/blob/76b8ae7e39f07a659b51284effafe668394d7f29/src/solver/toolkit.c#L2800

I think you can accomplish inflows with the same design. My suspicion is that overriding the mass as you are doing it in the Node structure might be missing a flow rate to accompany the mass rate.

Thanks @bemcdonnell, I'll look into it.

karosc · 2022-01-16T17:45:52Z

@abhiramm7, so what I'm getting from your post is that we cannot use the pollutant setters to create new inflows of pollutants. There must already be a pollutant inflow prescribed in the inp file and the setter can just augment the values?

abhiramm7 · 2022-01-16T17:50:50Z

@abhiramm7, so what I'm getting from your post is that we cannot use the pollutant setters to create new inflows of pollutants. There must already be a pollutant inflow prescribed in the inp file and the setter can just augment the values?

Just to make sure, we can modify any of the existing pollutants, but not create new ones that are not defined in the SWMM INP file. For example, say there is no TSS in the INP file, you cannot inject TSS into a node. From what I understand how SWMM works, we need a variable assigned for pollutants before the sim starts. We cannot create one in between yet. Did I answer your question?

karosc · 2022-01-16T21:00:46Z

I think I found the issue, my example is working now, but I had to add in a treatment for the node that I'm trying to modify. Without the node being specified in the treatment section, the pollutant setter doesn't work.

I added the following Treatment section to test_Example1.inp:

[TREATMENT]
;;Node           Pollutant        Function  
;;-------------- ---------------- ----------
24               TSS              R = 0.0

I then started looking at two nodes without any inflows between them and things starting making sense. Concentration at us and ds nodes are basically identical except when flow rate is 0, then the downstream node has no concentration. Thanks for the example @abhiramm7, it really helped! 🙏

import numpy as np
import matplotlib.pyplot as plt
from swmm.toolkit import solver, shared_enum

INP = "./data/test_Example1.inp"
RPT = "./data/test_Example1.rpt"
OUT = "./data/test_Example1.out"

mean = 10.0
steps = 36 * 60  # 36 hours simulation

solver.swmm_open(INP, RPT, OUT)
solver.swmm_start(0)

upstream_node = "24"
downstream_node = "17"
link_between = "16"
pollutant = "TSS"

upstream_node = solver.project_get_index(shared_enum.ObjectType.NODE, upstream_node)
downstream_node = solver.project_get_index(shared_enum.ObjectType.NODE, downstream_node)
pollutant = solver.project_get_index(shared_enum.ObjectType.POLLUT, pollutant)
link_between = solver.project_get_index(shared_enum.ObjectType.LINK, link_between)

data = {"upstream_node": [], "downstream_node": [], "flow": []}


for i in range(0, steps):

    # set node pollutant to high value
    solver.node_set_pollutant(upstream_node, pollutant, 1000)

    solver.swmm_step()

    # pull us and ds node concentrations and flow from the link that connects them
    data["upstream_node"].append(
        solver.node_get_pollutant(upstream_node, shared_enum.NodePollutant.QUALITY)[
            pollutant
        ]
    )
    data["downstream_node"].append(
        solver.node_get_pollutant(downstream_node, shared_enum.NodePollutant.QUALITY)[
            pollutant
        ]
    )

    data["flow"].append(
        solver.link_get_result(link_between, shared_enum.LinkResult.FLOW)
    )


solver.swmm_end()
solver.swmm_close()

fig, ax = plt.subplots(dpi=200)

ax.plot(data["upstream_node"], label="Up stream Node")
ax.plot(data["downstream_node"], label="Down stream Node")

axf = ax.twinx()
axf.plot(data["flow"], label="flow", color="red")

ax.set_ylabel("Pollutant Concentration")
axf.set_ylabel("Flow Rate")
fig.legend()
fig.show()

karosc · 2022-01-16T21:44:28Z

Moving on to HRT, I am also having trouble getting it to show up using your example inp. The tank object is a storage node, so it should have an HRT associated with it, right?

import numpy as np
import matplotlib.pyplot as plt
from swmm.toolkit import solver, shared_enum

INP = "./data/node_constantinflow_constanteffluent.inp"
RPT = "./data/node_constantinflow_constanteffluent.rpt"
OUT = "./data/node_constantinflow_constanteffluent.out"

mean = 10.0
steps = 200

solver.swmm_open(INP, RPT, OUT)
solver.swmm_start(0)

upstream_node = "Tank"
downstream_node = "Outfall"
pollutant = "P1"

upstream_node = solver.project_get_index(shared_enum.ObjectType.NODE, upstream_node)
downstream_node = solver.project_get_index(shared_enum.ObjectType.NODE, downstream_node)
pollutant = solver.project_get_index(shared_enum.ObjectType.POLLUT, pollutant)

data = {"upstream_node": [], "downstream_node": [], "hrt": [], "vol": []}

influent = np.linspace(0, 2, steps) * np.pi

for i in range(0, steps):
    solver.node_set_pollutant(upstream_node, pollutant, 10.0 + np.sin(influent[i]))
    solver.swmm_step()
    data["upstream_node"].append(
        solver.node_get_pollutant(upstream_node, shared_enum.NodePollutant.QUALITY)[
            pollutant
        ]
    )
    data["downstream_node"].append(
        solver.node_get_pollutant(downstream_node, shared_enum.NodePollutant.QUALITY)[
            pollutant
        ]
    )

    data["hrt"].append(
        solver.node_get_result(upstream_node, shared_enum.NodeResult.HYD_RES_TIME)
    )

    data["vol"].append(
        solver.node_get_result(upstream_node, shared_enum.NodeResult.VOLUME)
    )

solver.swmm_end()
solver.swmm_close()

fig, ax = plt.subplots(dpi=200)

ax.plot(data["upstream_node"], label="Up stream Node")
ax.plot(data["downstream_node"], label="Down stream Node")


axf = ax.twinx()
axf.plot(data["hrt"], label=" Up stream hrt", color="red")
axf.plot(data["vol"], label="Up stream volume", color="blue")
ax.set_ylabel("Pollutant Concentration")
axf.set_ylabel("hrt;volume")
fig.legend()
fig.show()

karosc · 2022-01-17T00:14:15Z

Looking at how HRT is initialized in node.c, where the node's subindex (storage index) is used to set the HRT in the storage struct, I think we need to implement that in the toolkit node result getter

I propose revising the C code to something like:

case SM_HRT:
      int k;
      k = Node[index].subIndex;
      *result = Storage[k].hrt; break;
  default: error_code_index = ERR_API_OUTBOUNDS; break;

Making that revision solves my issues with the above code:

I'll put together a PR soon for this and include the getLinkPollut fix as well

bemason · 2022-03-31T13:58:45Z

@karosc how do I point your python-swmm to my changes in swmm so that I can write new swmm-python tests using my changes in swmm?

karosc · 2022-03-31T14:18:22Z

I think this will work:

When you install swmm-python, you need to initialize the swmm submodule with git submodule update --init --recursive. This will clone the OpenWaterAnalytics/Stormwater-Management-Model into a folder called ./swmm-toolkit/swmm-solver.

Change directory into swmm-solver, add your fork as a remote, then pull your changes:

git remote add bemason https://github.com/bemason/Stormwater-Management-Model
git pull bemason develop

Then build swmm-python with python setup.py bdist_wheel. You can either symlink the build directory to your site-packages folder (my preferred approach), or install the wheel in the _dist folder whenever you change swmm and rebuild.

1. Added test to get node pollutant REACTOR_QUAL. Passed test. 2. Added test to get node pollutant INFLOW_QUAL. Passed test. 3. Added test to get link pollutant REACTOR_QUAL. Passed test.

bemason · 2022-04-04T16:30:37Z

@karosc I added the required tests to swmm-python. I just submitted a pull request to your branch. Can you add these changes to this pull request? Once done, I think we are again ready to finish this pull request and move onto pyswmm.

karosc · 2022-04-05T01:38:07Z

Great work @bemason, thank you!!! 🥳👏🙏 Once your changes get merged into the OWA SWMM repo, I can accept your swmm-python PR and update the submodule to make sure tests complete successfully. No reviewers have been assigned to the SWMM PR yet though, so it might be some time before the maintainers have some free time to approve.

bemcdonnell · 2022-04-05T04:28:30Z

@karosc, I’d be happy to review the owa-swmm code themes as long as you are all happy with the numerical aspects. Just point me to the pr

bemason · 2022-04-05T12:43:20Z

Thank you @bemcdonnell and @karosc. The swmm pull resquest is here.

bemason · 2022-04-05T13:18:44Z

swmm-toolkit/src/swmm/toolkit/solver_docs.i

@@ -664,6 +664,24 @@ flowrate: double
 ) swmm_setNodeInflow;


+%feature("autodoc", 
+"Set node inflow pollutant concentration


@karosc This doesn't set the inflow concentration. It sets the current node concentration. We should modify the description in line 668. It also says link index on like 673 instead of node index.

Add new required parameter to node_set_pollutant in test_solver.py.

bemason · 2022-04-14T15:55:44Z

@karosc swmm has been successfully merged! 🥳👏 Next swmm-python! Let me know if you need anything from me. :)

karosc · 2022-04-14T16:22:49Z

Thanks for your hard work @bemason, I'll merge your pull request and update the swmm submodule later today and ping you if anything comes up!

…ason-dev

- Updated windows action runner to 2022 - Revised cmake args to use VS 2022 - Update cmake to 3.21 for VS 2022 generator support - Deprecate Python 3.6 by removing from build and test scripts - Add Python 3.10 support by adding to build and test scripts - Update numpy to 1.21.5 for Python 3.10 support - Update pytest to 7.1.1 for Python 3.10 support

karosc · 2022-04-14T19:39:01Z

@bemason, I was a bit of a headache dealing with some merge conflicts and github action runner issues, but everything seems to be working! 🎉

In addition to adding pollutant changes and updating the swmm submodule, I was also able to deprecate python 3.6 and added python 3.10 support per #95, so we can check that box with this PR too.

Once the maintainers are able to look through this and approve things, we should be good to merge. The next step in this journey after merging this PR will be for SWMM and swmm-python develop branches to be merged in to main branches and for new releases to be generated.

- add type map for swmm_getVersion - add custom exception handling for swmm_getVersion - add test for legacy version getter

karosc added 6 commits January 6, 2022 10:41

update to latestest SWMM

d31395d

- add P_EVAP_RATE to output_enum and metadata - update swmm-solver submodule to latest commit on develop branch

fix swig error by

d6f1abf

- define EXPORT_TOOLKIT in solver.i

fix solver_rename.i

605766c

- change swmm_getVersionInfo -> swmm_getSemVersion

fix error with swmm_getSemVersion&swmm_getBuildId

2e5973a

- needed to add swig exception handling as with SMO_init and SMO_close

add new pollutant treatment to shared_enum

d994036

- Update shared enum with changes from toolkit_enums.h - update solver rename for two new pollutant setter methods - add PEP8 swmm_getBuildId

Add system get_system_attribute tester

3547795

karosc mentioned this pull request Jan 7, 2022

Pollutant treatment enhancements. #92

Closed

jennwuu requested a review from michaeltryby January 7, 2022 14:42

michaeltryby suggested changes Jan 7, 2022

View reviewed changes

Address comments

9b5ca78

- make shared_enum vars more descriptive - move EXPORT_TOOLKIT define to CMakeLists - fix some formatting and docstrings

michaeltryby approved these changes Jan 7, 2022

View reviewed changes

abhiramm7 approved these changes Jan 15, 2022

View reviewed changes

abhiramm7 requested changes Jan 15, 2022

View reviewed changes

abhiramm7 mentioned this pull request Jan 16, 2022

Link and Node Pollutant Function Call pyswmm/Stormwater-Management-Model#363

Closed

Add pollutant tests

2e4f638

1. Added test to get node pollutant REACTOR_QUAL. Passed test. 2. Added test to get node pollutant INFLOW_QUAL. Passed test. 3. Added test to get link pollutant REACTOR_QUAL. Passed test.

bemason reviewed Apr 5, 2022

View reviewed changes

bemason added 3 commits April 5, 2022 10:44

Updated node setter pollutant description in solver_docs.i

b3ee118

Merge branch 'dev' of https://github.com/bemason/swmm-python into dev

e1b3755

Update test_solver.py

e2e0dfe

Add new required parameter to node_set_pollutant in test_solver.py.

update swmm-solver to latest develop

1450374

karosc force-pushed the dev branch from 263d505 to 1450374 Compare April 14, 2022 16:48

karosc added 3 commits April 14, 2022 12:53

Merge branch 'dev' of https://github.com/bemason/swmm-python into bem…

246b27f

…ason-dev

Merge branch 'dev' of https://github.com/bemason/swmm-python into dev

c129f18

fix error in swmm_setNodePollut docstring

7ae4411

karosc force-pushed the dev branch 5 times, most recently from 6137be5 to 7ae4411 Compare April 14, 2022 18:03

abhiramm7 approved these changes Apr 15, 2022

View reviewed changes

Debug swmm_get_version()

40c80d8

- add type map for swmm_getVersion - add custom exception handling for swmm_getVersion - add test for legacy version getter

karosc mentioned this pull request Apr 25, 2022

Draft for OWA_v5.1.14 release pyswmm/Stormwater-Management-Model#372

Merged

abhiramm7 merged commit 34f504e into pyswmm:dev May 21, 2022

karosc mentioned this pull request Aug 29, 2022

function node_set_pollutant does not work properly. #104

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update swmm-python to latest OWA SWMM #94

Update swmm-python to latest OWA SWMM #94

karosc commented Jan 7, 2022

michaeltryby left a comment

michaeltryby Jan 7, 2022

bemason Jan 7, 2022 •

edited

Loading

karosc Jan 7, 2022

bemason Jan 7, 2022

michaeltryby left a comment

jennwuu commented Jan 10, 2022

michaeltryby commented Jan 10, 2022

jennwuu commented Jan 10, 2022

karosc commented Jan 10, 2022

jennwuu commented Jan 10, 2022

abhiramm7 commented Jan 10, 2022 •

edited

Loading

abhiramm7 left a comment •

edited

Loading

karosc commented Jan 16, 2022 •

edited

Loading

bemcdonnell commented Jan 16, 2022

abhiramm7 commented Jan 16, 2022 •

edited

Loading

abhiramm7 commented Jan 16, 2022

abhiramm7 commented Jan 16, 2022

karosc commented Jan 16, 2022

abhiramm7 commented Jan 16, 2022

karosc commented Jan 16, 2022 •

edited

Loading

karosc commented Jan 16, 2022

karosc commented Jan 17, 2022

bemason commented Mar 31, 2022

karosc commented Mar 31, 2022 •

edited

Loading

bemason commented Apr 4, 2022

karosc commented Apr 5, 2022

bemcdonnell commented Apr 5, 2022

bemason commented Apr 5, 2022

bemason Apr 5, 2022 •

edited

Loading

bemason commented Apr 14, 2022

karosc commented Apr 14, 2022

karosc commented Apr 14, 2022

Update swmm-python to latest OWA SWMM #94

Update swmm-python to latest OWA SWMM #94

Conversation

karosc commented Jan 7, 2022

michaeltryby left a comment

Choose a reason for hiding this comment

michaeltryby Jan 7, 2022

Choose a reason for hiding this comment

bemason Jan 7, 2022 • edited Loading

Choose a reason for hiding this comment

karosc Jan 7, 2022

Choose a reason for hiding this comment

bemason Jan 7, 2022

Choose a reason for hiding this comment

michaeltryby left a comment

Choose a reason for hiding this comment

jennwuu commented Jan 10, 2022

michaeltryby commented Jan 10, 2022

jennwuu commented Jan 10, 2022

karosc commented Jan 10, 2022

jennwuu commented Jan 10, 2022

abhiramm7 commented Jan 10, 2022 • edited Loading

abhiramm7 left a comment • edited Loading

Choose a reason for hiding this comment

karosc commented Jan 16, 2022 • edited Loading

Link setter success (links show super high concentration)

Node setter error (nodes don't show super high concentration)

bemcdonnell commented Jan 16, 2022

abhiramm7 commented Jan 16, 2022 • edited Loading

abhiramm7 commented Jan 16, 2022

abhiramm7 commented Jan 16, 2022

karosc commented Jan 16, 2022

abhiramm7 commented Jan 16, 2022

karosc commented Jan 16, 2022 • edited Loading

karosc commented Jan 16, 2022

karosc commented Jan 17, 2022

bemason commented Mar 31, 2022

karosc commented Mar 31, 2022 • edited Loading

bemason commented Apr 4, 2022

karosc commented Apr 5, 2022

bemcdonnell commented Apr 5, 2022

bemason commented Apr 5, 2022

bemason Apr 5, 2022 • edited Loading

Choose a reason for hiding this comment

bemason commented Apr 14, 2022

karosc commented Apr 14, 2022

karosc commented Apr 14, 2022

bemason Jan 7, 2022 •

edited

Loading

abhiramm7 commented Jan 10, 2022 •

edited

Loading

abhiramm7 left a comment •

edited

Loading

karosc commented Jan 16, 2022 •

edited

Loading

abhiramm7 commented Jan 16, 2022 •

edited

Loading

karosc commented Jan 16, 2022 •

edited

Loading

karosc commented Mar 31, 2022 •

edited

Loading

bemason Apr 5, 2022 •

edited

Loading