proxy for wps outputs

[cehbrecht]

Twitcher is used as a proxy for WPS (and other services) but the wps outputs still come from the original service. At least in some cases we would like to have them also available through the proxy because the wps is not directly available. Currently we use work-arounds in phoenix.

[tomLandry]

@dbyrns we worked on this right? Can we update Carsten?

[dbyrns]

In fact we are working on #21 and #24 : Adding a rest proxy and a monitoring tool for submitted jobs. We'are starting to get something, needs some doc and a PR should be on its way. I think that the output proxy could be added to that pretty easily. @cehbrecht if you want to take a look and give your impression : https://github.com/Ouranosinc/twitcher/tree/issue-24-rest-api/twitcher/wps_restapi. Just keep in mind that the implementation / api are still a draft.

[cehbrecht]

@dbyrns thanks for the info :) I will have a look at it.

[cehbrecht]

@dbyrns Good to see an example of Swagger ... my colleagues already asked for an example and we need to provide an internal talk for it. Do you use any of the Swagger tools? I have seen you are using the Pyramid/Cornice/Swagger library.

Do you try to make your WPS api "somehow" compliant to existing approches?
https://52north.github.io/tamis-rest-api/
https://www.slideshare.net/TheodorFoerster/restful-web-processing-service

The rest-api will also replace the existing xmlrpc interfaces (or jsonrpc)? But I suppose they currently co-exist.

A PR is very welcome :)

[dbyrns]

@cehbrecht Yes we are using swagger with cornice to generate the api documentation along with the live instance (e.g. work in progress here https://colibri.crim.ca/twitcher/api).

The API has been inspired by the 52north one but we are trying to move more and more to what is discussed in the OGC testbed 14. They are really close to each other but the 52north one gives less information for most requests which does not really suit our needs.

The rpc interface has been left intact to avoid breaking something. Once the new interface suits your need I think that we will be able to remove it.

[cehbrecht]

@dbyrns have you done already something on the output proxy?

[dbyrns]

We have not work specifically on this issue yet. However we are pretty much done with the restapi. We have been slowed down by others projects so we are just starting to use it in place of Phoenix and are still finding some minors glitches. Also we are not done yet with the tests writing.

As for the output proxy I think that we could achieve it pretty easily with our current implementation. The output would be cached from the execute/monitoring function :
https://github.com/Ouranosinc/twitcher/blob/issue-24-rest-api/twitcher/wps_restapi/processes/processes.py#L120

The get output route would then retrieve the cache version rather than the remote one :
https://github.com/Ouranosinc/twitcher/blob/issue-24-rest-api/twitcher/wps_restapi/jobs/jobs.py#L178

If you want we could start a feature branch on your side so you can test our work and maybe add the proxy thing yourself if you are on a rush?

[cehbrecht]

It would be nice to have the "output proxy" ... otherwise I need to use my quirky workaround. You are pointing me to the rest-api ... about I can't use it yet from my clients. I could try to figure out if I can add this to the simple wps proxy. If not a wait for you :)

[dbyrns]

Indeed! I did not think that you will still submit and retrieve your result via the WPS interface. In that case twitcher will not monitor the jobs and thus my idea doesn't apply. I guess that we will have to add an extra check in the owxproxy.py and cache outputs the first time they are acceesed? If this is that kind of solution you are looking for maybe we could help down the road but this is further from our primary goal.

[cehbrecht]

I can't say yet how the solution could like :)