GPX tracks display in different position depending on GPX track setting for NAD83 map (regression)

[sfroyen]

Steps to reproduce

1. Open a map on Android OOM and attach a GPS.
2. Go for a walk and sprinkle some point objects using GPS positioning
3. Copy the map from Android to a desktop and open in OOM
4. Observe that yhe point objects added using GPS coordinates no longer align with the GPX track from OOM Android.

Here's a screenshot from Android:

And here's one from the Mac:

Notice that the track position where I added the purple plus sign shifts to the NE by about 1m. This shift is similar to the shift between NAD83 and WGS84 GPS lat/lon coordinates. Note that the shift only happens to the GPX track, not to the map objects that were added.

(I can close and restart Android OOM and the track does not move.)

Caveats:
The map I'm using was created using NAD 83 / UTM zone 13N and then converted to EPSG:6342 / NAD83(2011) when that option was added.

Mapper Version: 0.9.3 and v20200613.1 (I've also tried 0.9.3)
Operating System: Android 7.0 and MacOS 10.15.6

[dg0yt]

There is as a "GPX" setting in GDAL > Import with GDAL/OGR. Is it different on macOS vs. Android?

[pkturner]

Could they be built with different versions of PROJ?

[dg0yt]

Could they be built with different versions of PROJ?

No, not for a single official release (v0.9.3) on Android, macOS, Windows. These binaries come out of Continuous Integration from the same source tarballs.

There is as a "GPX" setting in GDAL > Import with GDAL/OGR. Is it different on macOS vs. Android?

This is important, because this represents different code paths.

[sfroyen]

Android OOM did not have GPX checked under GDAL/OGR. Once I checked that option, Android OOM behaves like the desktop version, i.e., the (saved) track is shifted. I will repeat the exercise with a new capture and report back.

My desktop OOM had the GDAL GPX box checked.

[sfroyen]

FYI - Unselecting the GDAL GPX option on the desktop and importing the GPX track again produces the initial Android result -- as expected.

[dg0yt]

Okay, I was expecting this from line style on the screenshots.
When checked, i.e. when using GDAL, coordinate conversion is done by GDAL. The other option does not use GDAL.

[sfroyen]

Two bugs?

Collecting a track on Android with the GDAL GPX option on behaves as follows. While the track is being collected, it aligns with the position of added point objects. After saving and reopening the map, the GPS track is shifted to the NW.

Re. my previous FYI:
After importing the GPX track on the desktop with the the GDAL GPX option off, there is no way to re-import it with the GPX option on. It seems that OOM caches the previous import and uses the non-GDAL method. Clearing the Closed Template list had no effect.

[dg0yt]

After importing the GPX track on the desktop with the the GDAL GPX option off, there is no way to re-import it with the GPX option on. It seems that OOM caches the previous import and uses the non-GDAL method. Clearing the Closed Template list had no effect.

I have a GPX test file with a named waypoint. Waypoint names are displayed only by the non-GDAL implementation. With this test file, I see that the chosen implementation correctly follows the user setting when opening a map file containing a GPX template, or when adding a new GPX file.

[dg0yt]

I can reproduce a small shift of app. 1 m at 40N 105W with PROJ 7.1.

I started with a map and GPX track created with PROJ 4.9 / GDAL 2 which did not have alignment issues. When I opened the map with Mapper using PROJ 7.1, there was an immediate warning about a georeferencing mismatch of app. 2 m, i.e. the geographic reference point and the projected reference point stored in the georefencing did not match, according to PROJ 7.1. This is what I get the for the same geographic coordinates:

Geographic coordinates	40N 105W
EPSG:6342, PROJ 4.9	500000.00 4427757.22
EPSG:6342, PROJ 7.1	500000.86 4427756.50

(Sidenote: Mapper assumes "WGS 84" for geographic coordinates, but this is not accurate with regard to different realizations. And PROJ switch from "WGS 84" to "GRS 1980" for some purposes.)

Now when I fix the georeferencing, keeping geographic coordinates, I would expect that the georeferenced GPX template keeps its position relative to the map, because GPX data uses geographic coordinates. This seems to work for non-GDAL mode, but not for GDAL mode.

[dg0yt]

The table shows results from Mapper's georeferencing dialog, i.e. from the transformation which Mapper establishes with PROJ. So how does this work?

Mapper constructs a PROJ transformation from +proj=latlong +datum=WGS84 to +init=epsg:6342 (using proj_create_crs_to_crs). Note the source CRS. Changing that to EPSG:4326 (for an accurate modern use of PROJ) makes Mapper produce the original result which we had seen before.

To finally overcome these issues, I would like to fully establish modern usage of PROJ, at least for EPSG.

[sfroyen]

PROJ 4 did not support the necessary transformations and the EPSG files did not have the necessary parameters until late 2019 or early this year. Also, it appears that EPSG:4326 (or wgs84) is some form of generic specification (I have not figured out exactly what that means). Perhaps the current revision at current time?).

EPSG 7657, 7659, 7661, 7663 and the current 7665 have been used by GPS satellites between 1994 and present. A very precise GPS (without correction signals) should have returned coordinates in one of the these systems (EPSG:7665 at present). I believe their parameters include linear corrections for time so, for centimeter precision, we need a timestamp too. If you use SBAS corrections, you should get the same coordinates with parameters updated twice(?) per year. If you do not have a super accurate GPS, you can use RTK corrections from a base station to get cm precision. In this case, the lat/lon coordinates depend on the base station setting. They may be WGS based, e.g., EPSG:7665 or NAD83 based like EPSG:6318. The two are not the same:

echo 40.02020698N 105W | cs2cs -d 8 EPSG:7665 EPSG:6318
40.02020047 -104.99998987 0.88568757

My GPS was using WGS based EPSG:7665, so comments about NAD83 based lat/lon do not apply here. But any solution this issue ought to handle the NAD83 case too. Perhaps geographic coordinates should not be used internally and always be converted from/to on import/export.

I will try to investigate PROJ's behavior when using wgs84 and EPSG:7665.

[dg0yt]

EPSG:4326 is WGS 84 in terms of EPSG.

The GDAL GPX driver uses WGS 84 defined as EPSG:4326 with explicit lat,lon order for geographic 2D coordinates, via macro SRS_WKT_WGS84_LAT_LONG. Many other GDAL drivers do the same. SRS_WKT_WGS84_LAT_LONG is public API:
https://github.com/OSGeo/gdal/blob/e67def79aec4f3354b093b35ddbf956c43812ffb/gdal/ogr/ogr_srs_api.h#L70-L71

By using EPSG:4326 or WGS 84 instead of +proj=latlong +datum=WGS84, we would effectively use the same default as GDAL.

Choosing something else than the default when needed is a different thing. We don't get exaxt information from GPX files or GNSS sources. (But we do have timestamps.)

[pkturner]

Using EPSG:4326 with PROJ is inaccurate in North America, because it sloppily "converts" WGS 84 (the datum of most GPS receivers) to NAD83 with a no-op. The shift is on the order of a meter.

EPSG:7665 (3D) and EPSG:9057 (2D) are more specific and current realizations of WGS 84 that PROJ converts accurately to/from NAD83. I hope Mapper can make it easy to get that accuracy.

[pkturner]

Another consideration -- and at first I thought this was the issue at hand -- is that after Mapper first displays a template with the map, it disrupts the user if the template is opened later and is positioned differently. Should tracks and other templates be defended against changes in PROJ versions, in GDAL options, in Mapper's implementation of Geographic CRS, etc.?

[dg0yt]

Trying to avoid disruption is certainly something what we try to achieve. For this issue, switching from +proj=latlong +datum=WGS84 to EPSG:4326 or WGS 84 is just that - maybe restoring an older shortcoming. I didn't test yet with EPSG:9057, but I will give it a try.

To make the template positioning reproducible, you need to store the exact definition of the CRS of the external data, and the exact definition of the transformation to the internal representation (the projected CRS of the map.) Mapper does this only partially, and still using old PROJ syntax, which leads to a number of issues.

And our unit tests still don't cover the NAD83 world, to detect bugs related to our usage of PROJ or GDAL.