Data Package Boom Proposal

Basic proposal

We want to be able to have / create layer packages before / outside of a LmServer installation. To do this, the layers in packages would need to be identified by something other than a URL. We discussed using UUIDs or GUIDs to identify the layers. These packages would also include the metadata for the layers in the package. That way the package could be used by LmCompute to seed the data for computations and for LmServer for cataloging the layers (with or without storing the data).

LmCompute

For LmCompute, we will need to change the way that layers are cataloged in the layers.db database. Currently, layers are cataloged based on the host name, url path, and a hash of the query parameters in the URL. The directory structure for storage will need to change and file paths will not be as clear, unless we come up with another mechanism for storing the layers that is easier to understand. This would also be a good time to harden the layer code.

Resolver service

It is not clear if the UUID should point to a metadata object, or if each would point at the data. In other words, would there be one UUID for a layer, or would there be one for the layer metadata and one for each data interface. This decision will matter to the resolver service.

This service will take one of these UUIDs, at least ours and maybe others, and resolve them to a data or metadata object. We might also want to add query parameters for our services that take a UUID and return corresponding objects. This could be an alternative to our IDs.