Grafana Warp 10™ Datasource Plugin

Install the plugin

Using grafana-cli

sudo grafana-cli plugins install ovh-warp10-datasource
installing ovh-warp10-datasource @ 2.2.0
from: https://grafana.com/api/plugins/ovh-warp10-datasource/versions/2.2.0/download
into: /var/lib/grafana/plugins

✔ Installed ovh-warp10-datasource successfully 

Restart grafana after installing plugins . <service grafana-server restart>

Cloning the repository

Just clone the repository in the Grafana plugins folder

git clone [email protected]:ovh/ovh-warp10-datasource.git /var/lib/grafana/plugins/ovh-warp10-datasource

Grafana will use the dist/ folder by default

Grafana 7.0 troubleshooting

As mentioned in #71, "CORS" issues can occurs with a Warp10 datasource. To fix it:

• Or set x_content_type_options to false in Grafana servers configuration settings,
• Or whitelist content-type in allow headers of Warp10 with the configuration key cors.header.

Add a new Warp 10™ Datasource

• Go to the Grafana menu (top left) > "datasources" > "add data source"
• Choose a name
• Set Warp10 as type
• Paste the Warp 10™ Platform exec endpoint URL ( do not append /api/v0/... )

Add execution variables

You can define variables at datasource level (~ organisation level) which can be available for all dashboards. you can put tokens, constants, macros, ... In case of a macro definition, the variable value must start with <% and end with %>. In the query you can prepend @ to the macro name to execute it.

For example, you can store a read token here:

Make a query

On a new dashboard, in a Graph visualization, click on Query icon on the left side bar, and choose Warp10 datasource.

A user friendly query editor will appear. You can use the global variable you defined previously:

It will generate WarpScript for you. The generated WarpScript is visible in your browser console:

You will see at the beginning of the query some special variables set by the plugin.

Name	Description	Example
$end	Timestamp of the most recent point in the Grafana time window	1498038153276000
$endISO	end value in ISO-8601 format	'2017-06-21T09:42:33.276Z'
$start	Timestamp of the less recent point in the Grafana time window	1498034553276000
$startISO	start value in ISO-8601 format	'2017-06-21T08:42:33.276Z'
$interval	Difference between $end and $start	3600000000
$__interval	Could be used as a bucket span to display less datapoints on a graph, saving your browser RAM. See Grafana doc.	3600000000

Advanced users can check the WarpScript editor and write the WarpScript they need.

To graph something on Grafana you need to return some Geo Time Series™ (GTS)

Graph example

The plugin look for GTS or GTS array in your stack, all other stack entry will be ignored. Turn on the WarpScript™ editor, and use this script:

NEWGTS
'io.warp10.grafana.test' RENAME
{ 'func' 'sinus' } RELABEL
'sinus' STORE

NEWGTS
'io.warp10.grafana.testmetric' RENAME
{ 'func' 'cosinus' } RELABEL
'cosinus' STORE

100 // Not graphable -> ignored
'b' // Not graphable -> ignored

$interval 20 / TOLONG 'step' STORE

<% $step + %> 'stepMacro' STORE
<% 'index' STORE $sinus $index NaN NaN NaN $index SIN  ADDVALUE DROP %> 'execMacroSinus' STORE
<% 'index' STORE $cosinus $index NaN NaN NaN $index COS  ADDVALUE DROP %> 'execMacroCoinus' STORE

$start $end $stepMacro $execMacroSinus FORSTEP
$start $end $stepMacro $execMacroCoinus FORSTEP
$sinus $cosinus

Table example

By default, the plugin build a table with the timestamp as the first column, and one column per GTS.

You can build custom tables instead of formating GTS array, if your result stack have only 1 element and this element have columns and rows property. Then you can choose Table as Table transform in Table Options section

WarpScript™ example:

{
  'columns' [
    {
      'text' 'columnA'
      'type' 'number'
      'sort' true
      'desc' true
    }
    {
      'text' 'columnB'
      'type' 'number'
    }
  ]
  'rows' [
    [ 10 20 ]
    [ 100 200 ]
  ]
}

Worldmap example

You can use ovh-warp10-datasource as datasource for showing position data on grafana using grafana-worldmap-panel plugin.

In order to do it, you need to install the grafana-worldmap-panel plugin: Worldmap Panel.

When both ovh-warp10-datasource and the grafana-worldmap-panel installed, you can define a new Worldmap widget, with a Warp 10 datasource and json result as Location Data in the Worlmap tab:

Now in your WarpScript™ you can generate data in the JSON format supported by Worldmap, for example :

'[ { "key": "amsterdam", "latitude": 52.3702, "longitude": 4.8952, "name": "Amsterdam" }, { "key": "charleroi", "latitude": 50.4108, "longitude": 4.4446, "name": "Charleroi" }, { "key": "frankfurt", "latitude": 50.110924, "longitude": 8.682127, "name": "Frankfurt" }, { "key": "london", "latitude": 51.503399, "longitude": -0.119519, "name": "London" }, { "key": "paris", "latitude": 48.864716, "longitude": 2.349014, "name": "Paris" } ]'
JSON->

And then you can see the chosen locations in the map:

You can also give a value to each location, in order to show the locations with different sizes and colors, as Worldmap allows:

'[ { "key": "amsterdam", "latitude": 52.3702, "longitude": 4.8952, "name": "Amsterdam", "value": 9 }, { "key": "charleroi", "latitude": 50.4108, "longitude": 4.4446, "name": "Charleroi", "value": 6 }, { "key": "frankfurt", "latitude": 50.110924, "longitude": 8.682127, "name": "Frankfurt", "value": 9 }, { "key": "london", "latitude": 51.503399, "longitude": -0.119519, "name": "London", "value": 12 }, { "key": "paris", "latitude": 48.864716, "longitude": 2.349014, "name": "Paris", "value": 15 } ]'
JSON->

Use Annotations

You can add Annotation on your graph: Dashboard > "Manage dashboard" > "Annotations"

Just add you WarpScript™

/!\ You must return a single GTS on TOP of your stack

Annotation example

NEWGTS
'alerts' RENAME
{ 'a' 'b' 'c' 'd' } RELABEL
$end $interval 2 / - NaN DUP DUP 'Restart WebServer' ADDVALUE
$end $interval 3 / - NaN DUP DUP 'Update v1.0.2' ADDVALUE

Define Templating variables

You can make a WarpScript query to build the choice list of your templating variables. In the dashboard settings, select Variables, and create a new one from a Query, with Warp10 as datasource. You can write any WarpScript in the Query field to build your list of choices:

• If you let several values on the stack, each value will be added to the choice list.
• Best practice: Let a list on the stack. Each value will be added to the choice list.
• Best practice: Let a map on the stack. The map keys will be added to the choice list, the map values will be available within the pannels WarpScript query. The values will be hidden from the dashboard user. This allow to hide complex values behind user friendly labels.

Here are valid queries for variable definition:

Each value is transformed into two WarpScript variables you can use in your queries:

• A string, named as you named your variable.
• A list of strings, named as you named your variable, suffixed by _list.

• If you do not use multiple selection, variable and variable_list will contain the currently selected value
• If you use multiple selection:
  • the string will contain an optimized WarpScript regular expression
  • the list will contain each element selected
• If you defined a custom all value and checked "All", variable and variable_list will contain your customized value.

In the example below,

• testMap has a custom all value set to ~custom.*allvalue.*
• testList has no defined custom all value.

Query returning Labels

A variable can contain values for a defined Label. For example, to get all the unique values for the key hostname, you can specify a query like this in the templating variable Query setting.

[ $ReadToken '~.*' { 'hostname' '~.*' } ] FIND
<% DROP LABELS 'hostname' GET %> LMAP
UNIQUE

You can also create nested variables. For example if you had another variable, for example $region. Then you could have the hosts variable only show hosts from the current selected region with a query like this:

[ $ReadToken '~.*' { 'region' $region } ] FIND
<% DROP LABELS 'hostname' GET %> LMAP
UNIQUE

You can fetch keys for a given Class.

[ $ReadToken '<class_name>' { } ] FIND
<% DROP LABELS KEYLIST %> LMAP
FLATTEN
UNIQUE

Templating variable evaluation

To understand the variable resolution, this is how a query is built

• Inject dashboard variables ($end, $interval, etc...)
• Inject datasource execution variables (Customized by datasource)
• Inject templating variables following the configuration order (a templating variable can call the previous templating variables in its resolution)
• Inject user query (can use all previous variables)

/!\ all of the templating values are casted into strings by Grafana engine.

User friendly query builder

• Basic Fetch
• Bucketizer
• Reducer
• Renamer
• Mapper
• Filter
• Extend limits (LIMIT, MAXOPS, MAXFETCH, ...)
• Anomaly detection

Data fetching example

// you can use this public token on the https://warp.senx.io endpoint to fetch raw sensor data from a connected BeerTender.

"gCA1SVjbDkaxtmTx9ydI4TI2iGc5hFgcbCWnzMRZSt45XW8dZ53Z7VK_if28i0kXwNDLazXHgLrXUKgxLK0RbS79eJmBCpyBlIxw9US7bPfdWH4Fta51.kXN.D4Hsk5OZOwl.vLRBzMpP7F2pAMfclMXSGtCOT6F"
'rt' STORE


[ $rt '~beertender.(rawvalue|rawoffset)' {} $end $interval ] FETCH 
'rawResult' STORE

[ $rawResult [] 'beertender.rawvalue' filter.byclass ] FILTER 0 GET 'rawvalue' STORE

// do a linear interpolation to convert raw sensor value into percent.
// empty : 30.8e6
// full (cold): 30.52e6
30.8e6  $rawvalue - // you can substract GTS and constants
[ SWAP 0 mapper.max.x 0 0 0 ] MAP 0 GET // clamp min value to zero
2800 /  //divide by 2800
[ SWAP 100 mapper.min.x 0 0 0 ] MAP 0 GET // clamp max value to 100
[ SWAP bucketizer.mean 0 1 h 0 ] BUCKETIZE // keep one point per hour, the mean of each hour.


'level in percent' RENAME 

Related links

• Contribute: https://github.com/ovh/ovh-warp10-datasource/blob/master/CONTRIBUTING.md
• Report bugs: https://github.com/ovh/ovh-warp10-datasource/issues