Configure storage_copy_function to False

[huard]

Description

PyWPS 4.4.1 has new option to link output files instead of copying them. This has the potential to speed server response.

• What are the implications in terms of data retention policies ?
• What is the performance improvement ?

[tlvu]

Docs how to activate this option?

[huard]

https://github.com/geopython/pywps/blob/pywps-4.4/docs/configuration.rst

Search for storage_copy_function

[huard]

@aulemahal Could you check if you're able to benchmark this option and tell if this makes a difference ?

[aulemahal]

Oh boy that was a long long long way to get an answer... I did learn more about debugging docker container though!

1. This option needs pywps 4.4.1, which is incompatible with current finch. I started a pywps-4.4.4 branch solving this issue (it also relaxes the dependency to pywps~=4.4).
2. We can't make hard links because the working directory /tmp lies on another "device" than wpsoutputs. I.e. in the docker container, wpsoutputs is on /data which is a volume shared between containers and /tmp is local to the finch container.

So I don't know if it makes a real difference sorry...

(P.S.: pywps defaults to copy when link doesn't work, but doesn't print any info about the error, which was a truly frustrating "bug" and source of very dodgy and hacky workarounds. Might open a PR to add a logging call.)

[huard]

@tlvu Do you see a solution to this ?

This is not a problem per say, but an opportunity to reduce IO and improve performance. Instead of making copies of outputs, we could create symbolic links, but as Pascal showed above, it doesn't work out of the box.

[tlvu]

@tlvu Do you see a solution to this ?

This is not a problem per say, but an opportunity to reduce IO and improve performance. Instead of making copies of outputs, we could create symbolic links, but as Pascal showed above, it doesn't work out of the box.

A naive answer would be to configure in wps.cfg to use /data/wpsoutputs/{birdname}/tmp as temp location ... if such a config exists !

[tlvu]

@tlvu Do you see a solution to this ?
This is not a problem per say, but an opportunity to reduce IO and improve performance. Instead of making copies of outputs, we could create symbolic links, but as Pascal showed above, it doesn't work out of the box.

A naive answer would be to configure in wps.cfg to use /data/wpsoutputs/{birdname}/tmp as temp location ... if such a config exists !

Oh wait, looking at https://github.com/geopython/pywps/blob/pywps-4.4/docs/configuration.rst, it looks like server.temp_path config exists ! Hope it does what I think it does.

Will try to prototype this tomorrow.

[tlvu]

@huard

Update from the prototyping:

• I think you meant storage_copy_function = link and not storage_copy_function = False as in the title of this issue?
• Here's the change bird-house/birdhouse-deploy@78749e3 and it's live on my server https://lvupavics.ouranos.ca/twitcher/ows/proxy/finch
• But when I do not set server.temp_path, it stills works so there's some automatic fall-back to copy so I do not really have any confirmation that we are really leveraging hardlinks or not.

[tlvu]

I am into blind fix right now so I also set server.workdir to be on the same partition as server.outputdir see bird-house/birdhouse-deploy@9a42ffc. Again it's live on my server if you want to test https://lvupavics.ouranos.ca/twitcher/ows/proxy/finch

[tlvu]

Still ongoing, will be tested together with bird-house/birdhouse-deploy#11