vinyl: a record store for Clojure

Vinyl provides a facade for FoundationDB's record-layer.

The intent of the record layer is to provide a protobuf based storage engine for indexed records stored in FoundationDB.

Search queries

Queries in vinyl can be supplied using the following functions:

• exoscale.vinyl.store/list-query
• exoscale.vinyl.store/execute-query

A typical query will be executed this way:

@(store/list-query vinyl-store [:RecordType [:= :field "value"]])

Both list-query and execute-query accept different arities:

(list-query store query)
(list-query store query opts)
(list-query store query opts values)

(execute-query store query)
(execute-query store query opts)
(execute-query store query opts values)

The query argument can either be an instance of RecordQuery or a vector, in which case a RecordQuery will be built from the vector. See Query language for a reference.

The additional opts argument allows supplying options to perform modifications on the results inside the transaction in which the query is executed. See Record cursors for details on what can be supplied there.

The additional values argument is a map of bindings to be applied to a prepared query where applicable.

Query language

The data representation for queries takes the following shape:

[:RecordType optional-filter]

Select all queries

To select all fields you can provide a single member vector containing the record type:

[:RecordType] ;; for instance: @(store/list-query store [:RecordType])

Field equality

Field equality is supported in two ways, if the comparand is anything but a vector it will be treated as a fixed value:

@(store/list-query store [:RecordType [:= :id 1234]])
@(store/list-query store [:RecordType [:= :name "jane"]])

If the comparand is provided as a keyword, a prepared query is built and the corresponding keyword is expected to be found in the query's evaluation context:

(def my-query (query/build-query :RecordType [:= :user-name :name]))
@(store/list-query store my-query {} {:name "jane"})
@(store/list-query store my-query {} {:name "unknown"})

Field set membership

A field can be checked for membership in a set:

@(store/list-query store [:RecordType [:in :id [10 20 30]]])

Prepared queries are also supported for membership tests:

(def my-query (query/build-query :RecordType [:= :in :user-name :names]))
@(store/list-query store my-query {} {:names ["jane" "unknown"]})

Field inequality, null value check, and non null value checks

Fields can be checked for values that do not match a fixed comparand. Prepared queries are currently unsupported for the comparand of inequality tests:

@(store/list-query store [:RecordType [:not= :id 1234]])
@(store/list-query store [:RecordType [:some? :email]])
@(store/list-query store [:RecordType [:nil? :purge_date]])

Range comparisons

For values supporting comparisons, range operations are supported:

@(store/list-query store [:RecordType [:> :id 100]])
@(store/list-query store [:RecordType [:< :id 100]])
@(store/list-query store [:RecordType [:>= :id 100]])
@(store/list-query store [:RecordType [:<= :id 100]])

For string fields, prefix searches are supported:

@(store/list-query store [:RecordType [:starts-with? :path "/usr/local/"]])

Boolean operations

Filters can be composed with boolean operations :not, :or, :and:

@(store/list-query store [:Rec [:and [:not [:= :id 1]]
                                     [:or [:> :id 100] [:< :id 50]]]])

Nested field matches

Since the FDB record layer supports storing records which contain nested values, There needs to be support for matching those in queries.

Suppose you have the following protobuf definition:

message Info {
  string path = 1;
}

message Top {
   int64 id = 1;
   Info info = 2;
}

You can apply any field query to fields in info by using :nested:

@(store/list-query store [:Top [:nested :info [:starts-with? :path "/"]]])

Enums

Proto3 enum must start at ordinal 0. Internally, FDB record layer does not make the difference between 0 and null and thus 0 will be serialized as null.

When defining an enum, you must add a dummy entry that you must never use.

 enum Payment {
    INVALID = 0; # This value must not be used
    PREPAID = 1;
    POSTPAID = 2;
 }