Job output transform #548

OlivGruwe · 2023-08-08T14:42:35Z

Add support for various GeoTIFF formats, allowing flexible handling and representation of GeoTIFFs in outputs
(fixes #100 <https://github.com/crim-ca/weaver/issues/100>_).
Add support for GET /results/{id} and GET /outputs/{id} routes to enable direct access to individual
job result items by ID. This enhancement includes: support alternate representations based on the Accept header.
If an alternate format (e.g., YAML for a JSON source) is requested it will be automatically generated and returned.
Link headers containing all possible output formats, allowing retrieval via query parameters
(e.g., output?f=application/x-yaml). (fixes #18 <https://github.com/crim-ca/weaver/issues/18>_).

Direct Links in job output + makefile + tests

Testing transformer engine added

Remove requirements-trfm Add to main requirements

Why not pandas

Update requirements

Ajout Multi frame Images

fmigneault

There's a few linting issues that can be resolved automatically using the make fix[...] targets.
Validate as well using make check[...] targets any other flagged issues that might need some manual adjustments. I didn't comment on all of them.

I did not go in depth into the actual implementation of the transform functions.
I think there is a few tests missing to validate that the actual results from each transformation is valid (eg: pixel values from one format to another are properly rendered or expected text from one format to another is properly structured).

Makefile

tests/smoke/docker-compose.smoke-test.yml

tests/wps_restapi/test_jobs.py

tests/transform/test_tranform.py

weaver/formats.py

weaver/processes/builtin/__init__.py

weaver/wps_restapi/jobs/jobs.py

weaver/tranform/png2svg.py

weaver/tranform/tiff.py

weaver/tranform/transform.py

…ansform

…moving todos and unused import

fmigneault

rescanned quickly the changes, many comments are still open

CHANGES.rst

fmigneault · 2025-04-02T15:00:03Z

Makefile

+		echo "Binary package manager cairo not found. Attempting to install it."; \
+		$(SUDO) apt-get install libpangocairo-1.0-0 \


Might need other libraries here. https://cairosvg.org/documentation/

fmigneault · 2025-04-02T15:01:12Z

docker/Dockerfile-base

@@ -23,6 +23,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
        g++ \
        git \
        nodejs \
+        libpangocairo-1.0-0 \


We will need to validate against libraries listed in https://cairosvg.org/documentation/ that are different...
Maybe a custom docker smoke-test or something? Not sure yet.

fmigneault · 2025-04-02T15:04:20Z

requirements.txt

Since we already have a install-transforms Makefile target, I think we should consider moving the new dependencies to a dedicated requirements-transform.txt file to keep things clear.
The corresponding pip install -r requirements-transform.txt would be added to install-transforms.

Maybe another PR could explore the optional activation of transform capabilities...

fmigneault · 2025-04-02T16:31:09Z

weaver/wps_restapi/jobs/jobs.py

+                "code": "",
+                "description": "The requested output Id is not available in the job results.",
+                "cause": "The output ID is not available",
+                "error": "",
+                "value": ""


Fill the details (output_id in value, InvalidIdentifierValue for code).

Cause could indicate something more useful, such as The output ID must be one of: {} with the list retrieved from available IDs in job.results.

Use "ID" to be consistent across messages.

weaver/wps_restapi/jobs/jobs.py

fmigneault · 2025-04-02T16:49:22Z

weaver/wps_restapi/jobs/jobs.py

+    is_reference = bool(get_any_value(result, key=True, file=True))
+    _, output_format = get_job_output_transmission(job, output_id, is_reference)
+    output_format = accept or output_format or result_media_type
+
+    # To ensure consistency and avoid type mismatches, all output formats are converted to a dictionary.
+    # This standardization prevents errors when handling cases in `get_job_results_single`
+    # where some formats are dictionaries and others are different types (e.g., strings or ContentType).
+    if not isinstance(output_format, dict):
+        output_format = {"mime_type": output_format}
+
+    return get_job_results_single(job, result, output_id, output_format, headers=headers, settings=settings)


This is essentially the same as the end of get_job_results_response:

weaver/weaver/wps_restapi/jobs/utils.py

Lines 758 to 765 in 64db348

res_id = out_vals[0][0]

# check accept header

req_fmt = (request_headers or {}).get("accept")

out_fmt = out_transmissions[res_id][1]

out_type = get_field(results[res_id], "mime_type", search_variations=True, default=None) # a voir en debuggant

out_select = req_fmt or out_fmt or out_type # (resolution order/precedence)

out_fmt = out_select

return get_job_results_single(job, out_info, res_id, out_fmt, headers=headers, settings=settings)

Please define a function like resolve_result_single or similar, and invoke it in both places.

The output transmission check should already be done by the get_job_results_single call, with all corresponding logic to convert data/link and transforms.

fmigneault · 2025-04-02T16:54:01Z

weaver/wps_restapi/jobs/jobs.py

+    config.add_cornice_service(sd.process_jobs_service)
+    config.add_cornice_service(sd.process_job_service)
+    config.add_cornice_service(sd.process_results_service)
+    config.add_cornice_service(sd.process_result_value_service)
+    config.add_cornice_service(sd.process_outputs_service)
+    config.add_cornice_service(sd.process_output_service)
+    config.add_cornice_service(sd.process_inputs_service)
+    config.add_cornice_service(sd.process_exceptions_service)
+    config.add_cornice_service(sd.process_logs_service)
+    config.add_cornice_service(sd.process_stats_service)


sync order with upstream branch to avoid the diff

weaver/wps_restapi/jobs/utils.py

OlivGruwe added 8 commits June 7, 2023 10:47

Transformer Implementation

07a27e7

Transformer Implementation

3c1ef15

Transformer Implementation +

6d23b4c

Direct Links in job output + makefile + tests

Transformer Implementation

d85aa30

Testing transformer engine added

Transformer Implementation

b02bb01

Remove requirements-trfm Add to main requirements

Transformer Implementation

d7c727f

Why not pandas

Transformer Implementation

b3b4f52

Update requirements

Correction Txt -> Pdf

dcde84e

Ajout Multi frame Images

fmigneault requested changes Aug 29, 2023

View reviewed changes

This was referenced Sep 13, 2024

Support /results/{id} and /outputs/{id} routes #18

Open

Support GML for results #718

Open

Merge branch 'master' of github.com:crim-ca/weaver into job_output_tr…

d6b8c6c

…ansform

github-actions bot added feature/job Issues related to job execution, reporting and logging. feature/quotation Issues related to quotation, billing and estimators. process/builtin Issue related to builtin application processes labels Sep 24, 2024

Nazim-crim self-assigned this Sep 24, 2024

Nazim-crim added 3 commits September 24, 2024 15:57

Fixing naming typos

b3048bb

Adding libpangocairo to dockerfile

a23519e

Fixing import and linting

ee9c208

github-actions bot added the feature/cli Issues or features related to CLI operations. label Sep 25, 2024

Nazim-crim added 5 commits September 26, 2024 15:40

Fixing docker call and removing unused code

a8d4473

Removing transformer route and test in job, fixing makefile

574314c

Adding swagger and endpoint definition to support outputId and resultId

0304b3a

Adding in config output and result service

4cf0e80

Fixing test to use transformer resource path

81721dd

Nazim-crim added 4 commits November 26, 2024 10:56

Fixing comment indent, removing unused code

32bdb81

Wrap test in try finally

a7f8216

Fixing docstring, removing unused function

c017929

Fixing doc, requirements ordering and removing print

2ce3667

fmigneault added the WIP Work in progress label Dec 14, 2024

Nazim-crim and others added 17 commits December 18, 2024 14:41

Fixing cli formating, swagger def and correcting wps_package assert

f9aee2f

modifying execute and fix import

0bc4822

Merge master

a2e8eee

fixing requirements

216b7ea

Merge branch 'master' into job_output_transform

11e5546

Fixing linting and provenance test when outputs are empty

f2a87a8

Fixing test to keep outputs endpoint json only

c8e7564

Fix cli download to only unique link

ddbe770

Removing unused import, code cleanup, fixing ogc strict check

2f89e63

Fix process test to use contentType constants

e8093ec

Merge branch 'master' of github.com:crim-ca/weaver into job_output_tr…

c79a22b

…ansform

Fixing import

2852caf

Simplify possible output format lookup and removing unused function

704e217

refactoring to extract logic to extend_alternate_formats

7ec5e4f

Adding missing types to tests and adding missing docstring

f3d0d9e

Fixing import, adding test for coverage and removed unused code

e289017

Adding parameter in extend alternate format to avoid circular dep, re…

64db348

…moving todos and unused import

Nazim-crim requested a review from fmigneault April 1, 2025 21:11

fmigneault requested changes Apr 2, 2025

View reviewed changes

fmigneault and others added 6 commits April 2, 2025 13:22

Merge branch 'master' into job_output_transform

bc859d3

Fixing changelog

555ecc1

Fixing test formating, removing try catch

a2306c1

Adding pytest fixture to avoid duplicate

8b3b3c4

fixing docstring

e2b282c

Fixing format resolution and type validation

47d35cf

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Job output transform #548

Job output transform #548

OlivGruwe commented Aug 8, 2023 •

edited by Nazim-crim

Loading

fmigneault left a comment

fmigneault left a comment

fmigneault Apr 2, 2025

fmigneault Apr 2, 2025

fmigneault Apr 2, 2025

fmigneault Apr 2, 2025

fmigneault Apr 2, 2025

fmigneault Apr 2, 2025

		echo "Binary package manager cairo not found. Attempting to install it."; \
		$(SUDO) apt-get install libpangocairo-1.0-0 \

	res_id = out_vals[0][0]
	# check accept header
	req_fmt = (request_headers or {}).get("accept")
	out_fmt = out_transmissions[res_id][1]
	out_type = get_field(results[res_id], "mime_type", search_variations=True, default=None) # a voir en debuggant
	out_select = req_fmt or out_fmt or out_type # (resolution order/precedence)
	out_fmt = out_select
	return get_job_results_single(job, out_info, res_id, out_fmt, headers=headers, settings=settings)

Job output transform #548

Are you sure you want to change the base?

Job output transform #548

Conversation

OlivGruwe commented Aug 8, 2023 • edited by Nazim-crim Loading

fmigneault left a comment

Choose a reason for hiding this comment

fmigneault left a comment

Choose a reason for hiding this comment

fmigneault Apr 2, 2025

Choose a reason for hiding this comment

fmigneault Apr 2, 2025

Choose a reason for hiding this comment

fmigneault Apr 2, 2025

Choose a reason for hiding this comment

fmigneault Apr 2, 2025

Choose a reason for hiding this comment

fmigneault Apr 2, 2025

Choose a reason for hiding this comment

fmigneault Apr 2, 2025

Choose a reason for hiding this comment

OlivGruwe commented Aug 8, 2023 •

edited by Nazim-crim

Loading