Better handling of form edits and versioning.

[panterz]

The COBWEB design depends on the premise that there is one schema (data-model) per survey. This causes problems for Survey Coordinators that change their survey after submitting records in an old survey. The workflow is:

1. Coordinator creates a survey
2. Users join the survey and submit data exposed through the WFS & Portal
3. Survey is changed / updated by the coordinator by add, removing or changing existing fields
4. Users submit new data in an incompatible schema which can't be added to the database and WFS.
5. New data are fortunately still available at PCAPI which supports different schema-per-observation as it uses a filesystem-based approach

The obvious solution for this is to assume that changes in surveys are actually different surveys (i.e. survey versioning). This complies with the COBWEB data-model (WP7) and the whole design of the architecture as endorsed by COBWEB partners.

[xmichael]

This is mostly portal work i.e. the ability to clone surveys. Discussion in Basecamp is here

Until then, we should make sure Survey Coordinators know don't change surveys that already have data submitted to them. They should create new surveys at the portal as a workaround instead.

[MasonJohnDavis]

Perhaps we need to discuss a little more. I feel that this also has
connection with the contingency viewer also?

On 12 June 2015 at 12:35, M. Koutroumpas [email protected] wrote:

This is mostly portal work i.e. the ability to clone surveys. Discussion
in Basecamp is here
https://basecamp.com/2255772/projects/2799919/messages/40276555#comment_285203954

Until then, we should make sure Survey Coordinators know don't change
surveys that already have data submitted to them. They should create new
surveys at the portal as a workaround instead.

—
Reply to this email directly or view it on GitHub
#150 (comment).

[Delawen]

So, the idea is that once the user clicks on the authoring link tool, no one will be able to modify the survey. But they will have instead a "duplicate" button.

Is that right?

[xmichael]

That's correct

[pvgenuchten]

i think the default 'duplicate content' offers the required functionality; duplicate the survey with a new uuid; on which a new app configuration can be coupled; Note that this survey is duplicated in the same group, so the same users can access it

[pvgenuchten]

Let's be clear on the fact that with this workaround we're solving a limitation of the pcapi data explode scripts that currently aren't able to detect if the model has been changed, so an update on the postgres table structure would have been required, and from a user perspective this workaround is not optimal

[xmichael]

@pvgenuchten this is not true. PCAPI is very much able to detect changes and it tries really hard to overcome this limitation. In particular the table schema will be updated if there are no observations already uploaded.

However, if there are already observations published with the old schema there is nothing PCAPI can do to magically transform irrelevant data models to each other.

[Delawen]

What no-SQL databases do is add the new columns with a null or empty value for the previous records.
Also, the columns "removed" can be really removed or just add null or empty values for future rows.

But we can implement the duplicate button and not add more complexity to PCAPI, who knows what other implications this may bring.

[xmichael]

@Delawen I am happy to implement this in PCAPI if you explain how. Complexity is not a problem if it is the right thing to do. However I am still not convinced that this is the right thing.

Could you please explain this by providing an example how an existing Feature with data model (Text, Decision Tree, Image 1, Image 2, Geom) gets converted to a new data model with (Text, Audio, Date) ?

If the answer is to concatenate the two data models into a huge one while adding NULLs in the middle then we need an explanation for the Co-design users when they get a 100 Column database of NULLs after a couple of revisions.

We also need to make sure that existing OGC implementation like Geoserver, SOS, WPSs don't mind a featuretype that can change in every second.

In any case NoSQL databases do solve this but not with NULLs. They are just embedding the schema in every observation, which is exactly what PCAPI already does.

[Delawen]

Well, the idea is not to concatenate "as is", but detect the fields that have changed and add them. And then detect the fields that are no longer there and just add like a "ghost" field with an empty value. If a field has changed the type, then it counts as a removal and new add.

About OGC services, as long as we don't cache them, it is not a problem. Maybe we will have to tell GeoServer to refresh the schema, but that's all.

But I don't know what are the implications of doing this on PCAPI, that's why if you say it is not "easy" (easy, fast, useful, too complex), then just forget about it and we add the duplicate button.

[xmichael]

@Delawen , it is "easy", it just does not seem sensible to do so

[xmichael]

@Delawen to summarize above: if you add a parameter parent_sid=X along with the existing sid=Y when calling the survey designer it will load X and save it as Y.

[Delawen]

Thanks.

[Delawen]

On some basecamp thread I cannot find now (or was it an ITF?) someone mentioned that the clone button was no longer needed. Did I dream it or does it make sense?