Publish as vector tiles format alongside Geo/TopoJSON

[nickpeihl]

With the introduction of feature-state in the Kibana Maps app, we can consider publishing our vector layers as file-based vector tiles. This would allow us to produce much higher resolution datasets than is currently possible with GeoJSON and TopoJSON.

For example, the USA Zip Codes layer currently served as TopoJSON is dramatically simplified to reduce the file size for web map consumption. Compare the following screenshots of zip codes near New York City.

Simplified TopoJSON at zoom 10

Full resolution shapefile at zoom 10

We could theoretically publish zip codes as a directory tree of vector tiles and maintain nearly the same resolution of the shapefile.

Vector Tiles at zoom 10

Full resolution shapefile at zoom 10

GIS files can be converted into vector tiles using the Tippecanoe tool.

1. Download and extract the US Census ZCTA shapefile from ftp://ftp2.census.gov/geo/tiger/TIGER2018/ZCTA5/tl_2018_us_zcta510.zip
2. Convert the shapefile into line delimited GeoJSON using gdal.
   ogr2ogr -f GeoJSONSeq -lco RS=YES -lco ID_FIELD=ZCTA5CE10 -sql "SELECT ZCTA5CE10 as zip FROM tl_2018_us_zcta510" zcta.geojsons ~/Downloads/tl_2018_us_zcta510/tl_2018_us_zcta510.shp
3. Use tippecanoe to generate a directory of vector tiles from the GeoJSON file. This may take a while.
   tippecanoe --read-parallel --detect-shared-borders -S10 -zg -e usa_zip_codes --coalesce-densest-as-needed --no-tile-compression --generate-ids zcta.geojsons

The result vector tiles directory contains nested subdirectories for the {z}, {x}, and {y} tiles. So they can be stored and served using the same cloud storage we use for the other vector files (e.g. https://vector.maps.elastic.co/data/<layer_name>/{z}/{x}/{y}.pbf). And the vector tiles can be styled the same way we style GeoJSON or TopoJSON datasets.

In addition to higher resolution datasets, the vector tiles passed to the client can be much smaller than the GeoJSON and TopoJSON formats. Also, the Maps app in Kibana has native support for vector tiles which means the data is rendered much more quickly.

The Region Maps and Vega Maps in Kibana do not support vector tiles, so we would still maintain support for those visualizations by publishing simplified GeoJSON or TopoJSON datasets.

[nyurik]

While we could publish our regions data as a set of standalone tile layers, I think this is a bit of a dead-end approach - huge number of independently maintained layers, and higher traffic due to each layer being loaded separately. Instead, I would propose to merge our vector data into our existing tile vectors generation pipeline that we are building. This way, those detailed shapes would end up as our main tile generation pipeline, and showing them would be as simple as saying "if vector tile has an element of type X, show it with color Y". In a way, this is no different from our current pipeline on the client:

• take tile, add another layer to the tile from geojson, add tile styling to show that specific element with the new color

With the new approach, the geojson will essentially be already baked into the data tile loaded from the server.

In case of the user-data, the client would still have to do the data tile mixing (loading data from elasticsearch + base layers)