Skip to content

Vector Tiles with GeoServer Notes

Jump to bottom
Anthony Fok edited this page Sep 30, 2022 · 10 revisions

This wiki page was created by Joost van Ulden on 3 May 2022

GeoServer in a Container

$ docker pull kartoza/geoserver
$ docker run -d --name "geoserver" -e GEOSERVER_ADMIN_PASSWORD=geoserver -e GEOSERVER_ADMIN_USER=admin -p 8080:8080 kartoza/geoserver

http://localhost:8080/geoserver [Username/PW: admin/geoserver]

Stopping the application

$ docker ps (find the container name)
$ docker stop geoserver

Restarting

$ docker start geoserver

  1. Create data folder...

  2. Copy files into it:

    docker cp dsra_acm7p0_georgiastraitfault_indicators_s.gpkg \
    geoserver:/opt/geoserver/data_dir/workspaces/OpenDRR/dsra_acm7p0_georgiastraitfault_indicators_s.gpkg

  3. Generate tiles...

  4. Copy tiles to local machine out of container:

    docker cp geoserver:/opt/geoserver/data_dir/tmp/tiles/OpenDRR_dsra_acm7p0_georgiastraitfault_indicators_s .

  5. Change permissions to be able to test locally:

    chmod -R 755 mapbox-tile-leaflet

Notes from Joost’s demo on May 3, 2022:

  1. Add new workspace.

    • Can work without a workspace, but a workspace to keep things compartmentalized and is recommended
    • Don't worry about enabling any of the stuff (services, etc.)

  2. Upload data (whatever dataset) into the directory

    • For DSRA, see OpenDRR/earthquake-scenarios repo, either in release assets, or in FINISHED/geopackages/dsra_acm7p0_georgiastraitfault_indicators_csd.gpkg (stored in Git LFS)
    • For Docker, Look at Joost’s notes above: 
      docker cp dsra_acm7p0_georgiastraitfault_indicators_s.gpkg \
    geoserver:/opt/geoserver/data_dir/workspaces/OpenDRR/dsra_acm7p0_georgiastraitfault_indicators_s.gpkg

  3. Check the files are indeed in there:

    • If using Docker Desktop, click on the CLI button, which will run docker exec -it geoserver /bin/sh automatically.
      • Convenient, but /bin/sh points to /bin/dash which does not support command-line completion.
    • Run docker exec -it geoserver /bin/bash manually instead (from Windows PowerShell, macOS Terminal, etc.) for command-line completion.
    • Inside the kartoza/docker-geoserver container: 
      $ docker exec -it geoserver /bin/bash
 _  __          _                  ____             _                ____           ____                           
| |/ /__ _ _ __| |_ ___ ______ _  |  _ \  ___   ___| | _____ _ __   / ___| ___  ___/ ___|  ___ _ ____   _____ _ __ 
| ' // _` | '__| __/ _ \_  / _` | | | | |/ _ \ / __| |/ / _ \ '__| | |  _ / _ \/ _ \___ \ / _ \ '__\ \ / / _ \ '__|
| . \ (_| | |  | || (_) / / (_| | | |_| | (_) | (__|   <  __/ |    | |_| |  __/ (_) |__) |  __/ |   \ V /  __/ |   
|_|\_\__,_|_|   \__\___/___\__,_| |____/ \___/ \___|_|\_\___|_|     \____|\___|\___/____/ \___|_|    \_/ \___|_|   
                                                                                                                   
geoserveruser@75d15ea62ded:/geoserver$ ls -l
total 0
geoserveruser@75d15ea62ded:/geoserver$ ls
geoserveruser@75d15ea62ded:/geoserver$ cd /opt
geoserveruser@75d15ea62ded:/opt$ ls -l
total 20
drwxr-xr-x 1 geoserveruser geoserverusers 4096 Jan 26 06:57 fonts
drwxr-xr-x 1 geoserveruser geoserverusers 4096 Jan 26 06:57 footprints_dir
drwxr-xr-x 1 root          root           4096 Jan 26 06:57 geoserver
drwxr-xr-x 7 root          root           4096 Feb  6 18:51 libjpeg-turbo
geoserveruser@75d15ea62ded:/opt$ cd geoserver/
geoserveruser@75d15ea62ded:/opt/geoserver$ ls -l
total 8
drwxr-xr-x 1 geoserveruser geoserverusers 4096 May 21 09:56 data_dir
geoserveruser@75d15ea62ded:/opt/geoserver$ cd data_dir
geoserveruser@75d15ea62ded:/opt/geoserver/data_dir$ ls -l
total 72
-rw-r--r-- 1 geoserveruser geoserverusers  134 Apr  6 19:38 controlflow.properties
drwxr-x--- 2 geoserveruser geoserverusers 4096 Apr  6 19:38 csw
-rw-r----- 1 geoserveruser geoserverusers 2910 May 21 09:56 global.xml
drwxr-xr-x 1 geoserveruser geoserverusers 4096 May 21 01:20 gwc
-rw-r----- 1 geoserveruser geoserverusers 1597 Apr  6 19:38 gwc-gs.xml
drwxr-x--- 2 geoserveruser geoserverusers 4096 May 20 19:11 gwc-layers
drwxr-x--- 2 geoserveruser geoserverusers 4096 Apr  6 19:38 inspire
-rw-r----- 1 geoserveruser geoserverusers  147 May 20 23:29 logging.xml
drwxr-xr-x 2 geoserveruser geoserverusers 4096 May 20 23:29 logs
drwxr-xr-x 3 geoserveruser geoserverusers 4096 Apr  6 19:38 monitoring
drwxr-x--- 2 geoserveruser geoserverusers 4096 Apr  6 19:38 printing
drwxr-xr-x 8 geoserveruser geoserverusers 4096 May 20 19:11 security
drwxr-x--- 2 geoserveruser geoserverusers 4096 Apr  6 19:38 styles
drwxr-x--- 3 geoserveruser geoserverusers 4096 Apr  6 19:38 temp
drwxr-x--- 2 geoserveruser geoserverusers 4096 Apr  6 19:38 tmp
drwxr-xr-x 2 geoserveruser geoserverusers 4096 Apr  6 19:38 user_projections
drwxr-x--- 3 geoserveruser geoserverusers 4096 May 11 20:19 workspaces
-rw-r----- 1 geoserveruser geoserverusers 1745 Apr  6 19:38 wps.xml
geoserveruser@75d15ea62ded:/opt/geoserver/data_dir$ cd workspaces
geoserveruser@75d15ea62ded:/opt/geoserver/data_dir/workspaces$ ls -l
total 8
drwxr-x--- 14 geoserveruser geoserverusers 4096 May 20 19:10 OpenDRR
-rw-r-----  1 geoserveruser geoserverusers  188 May 11 20:19 default.xml
geoserveruser@75d15ea62ded:/opt/geoserver/data_dir/workspaces$ cd OpenDRR
geoserveruser@75d15ea62ded:/opt/geoserver/data_dir/workspaces/OpenDRR$ ls -l
total 1395508
-rw-rw-r-- 1 geoserveruser           1000   3158016 May  6 17:00 dsra_acm7p0_georgiastraitfault_indicators_csd.gpkg
drwxr-x--- 3 geoserveruser geoserverusers      4096 May 19 21:55 dsra_acm7p0_georgiastraitfault_shakemap_hexgrid_1km
-rw-rw-r-- 1 geoserveruser           1000  81776640 May  6 17:02 dsra_acm7p0_georgiastraitfault_shakemap_hexgrid_1km.gpkg
drwxr-x--- 3 geoserveruser geoserverusers      4096 May 19 18:29 dsra_acm7p0_georgiastraitfault_shakemap_hexgrid_5km
-rw-rw-r-- 1 geoserveruser           1000   8028160 May  6 17:04 dsra_acm7p0_georgiastraitfault_shakemap_hexgrid_5km.gpkg
-rw-rw-r-- 1 geoserveruser           1000   2998272 May  6 17:00 dsra_acm7p3_leechriverfullfault_indicators_csd.gpkg
drwxr-x--- 3 geoserveruser geoserverusers      4096 May 20 19:04 dsra_acm7p3_leechriverfullfault_shakemap_hexgrid_1km
-rw-rw-r-- 1 geoserveruser           1000  61460480 May  6 17:02 dsra_acm7p3_leechriverfullfault_shakemap_hexgrid_1km.gpkg
drwxr-x--- 3 geoserveruser geoserverusers      4096 May 19 19:56 dsra_acm7p3_leechriverfullfault_shakemap_hexgrid_5km
-rw-rw-r-- 1 geoserveruser           1000   6807552 May  6 17:04 dsra_acm7p3_leechriverfullfault_shakemap_hexgrid_5km.gpkg
-rw-rw-r-- 1 geoserveruser           1000   3067904 May  6 17:00 dsra_idm7p1_sidney_indicators_csd.gpkg
drwxr-x--- 3 geoserveruser geoserverusers      4096 May 20 19:08 dsra_idm7p1_sidney_shakemap_hexgrid_1km
-rw-rw-r-- 1 geoserveruser           1000  65261568 May  6 17:02 dsra_idm7p1_sidney_shakemap_hexgrid_1km.gpkg
drwxr-x--- 3 geoserveruser geoserverusers      4096 May 19 19:15 dsra_idm7p1_sidney_shakemap_hexgrid_5km
-rw-rw-r-- 1 geoserveruser           1000   6828032 May  6 17:04 dsra_idm7p1_sidney_shakemap_hexgrid_5km.gpkg
-rw-rw-r-- 1 geoserveruser           1000   6959104 May  6 17:01 dsra_scm7p5_valdesbois_indicators_csd.gpkg
drwxr-x--- 3 geoserveruser geoserverusers      4096 May 20 19:10 dsra_scm7p5_valdesbois_shakemap_hexgrid_1km
-rw-rw-r-- 1 geoserveruser           1000 215785472 May  6 17:03 dsra_scm7p5_valdesbois_shakemap_hexgrid_1km.gpkg
drwxr-x--- 3 geoserveruser geoserverusers      4096 May 19 19:21 dsra_scm7p5_valdesbois_shakemap_hexgrid_5km
-rw-rw-r-- 1 geoserveruser           1000  14761984 May  6 17:04 dsra_scm7p5_valdesbois_shakemap_hexgrid_5km.gpkg
-rw-rw-r-- 1 geoserveruser           1000   5013504 May  6 17:00 dsra_sim9p0_cascadiainterfacebestfault_indicators_csd.gpkg
drwxr-x--- 3 geoserveruser geoserverusers      4096 May 20 19:11 dsra_sim9p0_cascadiainterfacebestfault_shakemap_hexgrid_1km
-rw-rw-r-- 1 geoserveruser           1000 263675904 May  6 17:02 dsra_sim9p0_cascadiainterfacebestfault_shakemap_hexgrid_1km.gpkg
drwxr-x--- 3 geoserveruser geoserverusers      4096 May 19 20:58 dsra_sim9p0_cascadiainterfacebestfault_shakemap_hexgrid_5km
-rw-rw-r-- 1 geoserveruser           1000  20471808 May  6 17:04 dsra_sim9p0_cascadiainterfacebestfault_shakemap_hexgrid_5km.gpkg
-rw-r----- 1 geoserveruser geoserverusers       165 May 11 20:19 namespace.xml
drwxr-x--- 3 geoserveruser geoserverusers      4096 May 19 17:15 psra_indicators_csd
-rw-rw-r-- 1 geoserveruser           1000  33619968 May  6 17:48 psra_indicators_csd.gpkg
drwxr-x--- 3 geoserveruser geoserverusers      4096 May 19 19:31 psra_indicators_s
-rw-rw-r-- 1 geoserveruser           1000 629260288 May  6 17:48 psra_indicators_s.gpkg
-rw-r----- 1 geoserveruser geoserverusers       188 May 11 20:19 workspace.xml

  4. In GeoServer, go to Stores > Add new Store, then:

    1. On "New data source" page, under Vector Data Sources, pick GeoPackage
    2. On "New Vector Data Source" page:
      1. Under "Basic Store Info":
        1. ensure "OpenDRR" is selected as the Workspace;
        2. fill "Data Source Name" with whatever you want (e.g. "Canada_Boundary")
      2. Under "Connection Parameters":
        1. click on "Browse..." to the right of the "database *" field, go to the same directory as above (workspaces > OpenDRR), and choose e.g. Geometry_CANADA.gpkg, which would result in the value of file:workspaces/OpenDRR/Geometry_CANADA.gpkg
      3. You can just ignore the passwd.
      4. Click "Save"

GeoServer AWS EC2 instance

Start the EC2 instance (if it is stopped):

$ aws ec2 start-instances --instance-ids i-03e0603c20262c7b8

Log into the instance:

$ aws sso login
$ ssh -L 8080:localhost:8080 ubuntu@i-03e0603c20262c7b8

Once logged in, run docker start geoserver to start geoserver if it isn't running yet.

Then, thanks to the SSH port-forwarding above, GeoServer running on on AWS EC2 may be accessed from the local computer at http://localhost:8080/geoserver

See More Notes on Vector Tiles with GeoServer for more draft notes.

Compressing/archiving the vector tiles for distribution

  • Do not do it over NFS (too slow)

To include just non-empty files:

for i in OpenDRR_psra_indicators_csd; do
    time find $i -type f \! -empty \
        | env XZ_OPT="-9vv -T0" eatmydata tar --owner=1000 --group=1000 --transform="s/^OpenDRR_//" -caf /data/tmp/${i#OpenDRR_}.NOEMPTY.tar.xz -T -
done

where:

  • --owner=1000 --group=1000 is optional: sets the UID and GID that happens to be ubuntu/ubuntu on Ubuntu.
  • --transform="s/^OpenDRR_//" and ${i#OpenDRR_} both remove the OpenDRR_ prefix (which is the workspace name appended to the layer by GeoServer)
  • -T - means reading the file list from stdin

To see percentage of non-empty tiles

for i in OpenDRR_*/*; do
    all=$(find $i -type f | wc -l); nonempty=$(find $i -type f \! -empty | wc -l); percent=$(perl -E "printf(\"%.3f\%\", 100.0 * $nonempty / $all)");
    echo -e "${i#OpenDRR_}:\t$nonempty\t$all\t$percent";
done | tee /data/tmp/nonempty-stats.txt
Vector tiles directory non-empty PBF files all PBF files % non-empty
dsra_acm7p0_georgiastraitfault_shakemap_hexgrid_1km/EPSG_4326 368726 679945 54.229%
dsra_acm7p0_georgiastraitfault_shakemap_hexgrid_1km/EPSG_900913 147566 270854 54.482%
dsra_acm7p3_leechriverfullfault_shakemap_hexgrid_1km/EPSG_4326 274414 644036 42.608%
dsra_acm7p3_leechriverfullfault_shakemap_hexgrid_1km/EPSG_900913 109110 255926 42.633%
dsra_idm7p1_sidney_shakemap_hexgrid_1km/EPSG_4326 293300 651338 45.030%
dsra_idm7p1_sidney_shakemap_hexgrid_1km/EPSG_900913 116731 258881 45.091%
dsra_scm7p5_valdesbois_shakemap_hexgrid_1km/EPSG_4326 989690 2650210 37.344%
dsra_scm7p5_valdesbois_shakemap_hexgrid_1km/EPSG_900913 371041 1012006 36.664%
dsra_sim9p0_cascadiainterfacebestfault_shakemap_hexgrid_1km/EPSG_4326 1210119 3234668 37.411%
dsra_sim9p0_cascadiainterfacebestfault_shakemap_hexgrid_1km/EPSG_900913 515937 1395935 36.960%
psra_indicators_csd/EPSG_4326 12920124 40508757 31.895%
psra_indicators_csd/EPSG_900913 7929562 28162150 28.157%
psra_indicators_s/EPSG_4326 939041 39844423 2.357%
psra_indicators_s/EPSG_900913 451787 25104295 1.800%

This can also be used as a quick check to see the vector tiles are actually completely generated. (E.g., the percentages for EPSG_4326 and EPSG_900913 should not differ too much.)

Time logs

$ for i in ?sra_*; do echo $i; time eatmydata find $i -type f \! -empty | tar -I 'xz -9vv -T0' -cf /data/FINISHED/no-empty/$i.NOEMPTY.tar.xz -T -; done                                      
psra_indicators_csd                                            
xz: Filter chain: --lzma2=dict=64MiB,lc=3,lp=0,pb=2,mode=normal,nice=64,mf=bt4,depth=0                                        
xz: Using up to 16 threads.                                    
xz: 19986 MiB of memory is required. The limiter is disabled.                                                                 
xz: Decompression will need 65 MiB of memory.                  
  100 %        512.9 MiB / 70.3 GiB = 0.007   6.2 MiB/s    3:14:15                                                            

real    194m15.825s                                            
user    100m47.036s                                            
sys     25m39.977s

ZIP:

$ for i in psra_indicators_csd; do echo $i; time find $i -type f \! -empty | eatmydata zip -9v /data/FINISHED/no-empty/$i.NOEMPTY.zip -@; done
...
  adding: psra_indicators_csd/EPSG_4326/6/31/5.pbf      (in=9387) (out=5939) (deflated 37%)
total bytes=58784524793, compressed=28246126104 -> 52% savings

real    251m23.305s
user    42m29.416s
sys     24m44.965s

GeoServer on Windows (without Docker)

https://docs.geoserver.org/latest/en/user/installation/win_installer.html

Tips on removing empty PBF files on Linux

btrfs seems a good choice:

  1. No inode limit

  2. find returns a sorted file list:

    for i in psra_indicators_csd; do echo $i; time eatmydata find $i -type f \! -empty | tar -I 'xz -9vv -T0' -cf ~/OpenDRR/2022-09-22/new/$i.tar.xz -T -; done

    With the benefit that the resulting XZ archive is smaller than when the files are in random order.

Sprint Planning | Sprint Review
Wikis: data | model-factory | opendrr-api | opendrr | python-env | riskprofiler-cms

Clone this wiki locally