String is getting trimmed

[mihaimyh]

I am pushing metrics such as
cpu_usage{process="dsapi_slave",pid="89543"} 0.1

But in Prometheus I see the process as dsapi_sla+ instead of the original pushed name dsapi_slave (notice trimming happening by adding the + sign):

cpu_usage{environment="DEV", exported_instance="machine", exported_job="top", instance="someServer", job="Engine", pid="89543", process="dsapi_sla+", source="push_gateway"}

Any hints about this?

[mihaimyh]

This seems to happen when the script to push the metrics is being executed from crontab or via systemd. Running it manually doesn't shows this issue.

[mihaimyh]

More info, I was debugging what the curl command is sending to push gateway:

curl -v -X POST -H "Content-Type: text/plain" --data "${var}" https://localhost:10083/metrics/job/top/instance/machine --insecure --trace-ascii /dev/stdout

1. When the script is being run manually, everything looks ok:

2e00: rker/1:0]", pid="130216"} 0.0.memory_usage{process="[kworker/1:0
2e40: ]", pid="130216"} 0.0.cpu_usage{process="[kworker/2:1]", pid="13
2e80: 0251"} 0.0.memory_usage{process="[kworker/2:1]", pid="130251"} 0
2ec0: .0.cpu_usage{process="phantom", pid="130485"} 0.0.memory_usage{p
2f00: rocess="phantom", pid="130485"} 0.1.

2. Running it with systemd/cron it gets trimmed:

1780: _usage{process="[kworker/+", pid="130216"} 0.0.memory_usage{proc
17c0: ess="[kworker/+", pid="130216"} 0.0.cpu_usage{process="[kworker/
1800: +", pid="130251"} 0.0.memory_usage{process="[kworker/+", pid="13
1840: 0251"} 0.0.cpu_usage{process="phantom", pid="130485"} 0.0.memory
1880: _usage{process="phantom", pid="130485"} 0.1

[beorn7]

Weird. I can only imagine that some processing of the ${var} content happens in some layer of variable expansion/escaping/quoting that you only have in cron or systemd and not directly on the command line.

I don't know those Unix and shell details enough to say for certain.

I usually use --data-binary rather than --data, when using curl to push, and I feed the data via STDIN (i.e. curl --data-binary @-). Not sure if you can try the same in your case.

[beorn7]

Since this is most likely not an issue with Pushgateway itself, I'll convert this issue into a discussion topic in the Q&A category.

[weibeld]

@mihaimyh It would be helpful to print out the content of $var in both scenarios (curl detects a different content length in the two cases). So, if $var is different in the two scenarios, then the issue is somewhere earlier in the chain, i.e. not related to either curl or Pushgateway.

@beorn7 For the records, using curl --data-binary instead of --data seems to be necessary because the Prometheus metrics format requires a newline at the end of the last line and --data strips trailing newlines from the input.

From the Prometheus documentation:

Prometheus' text-based format is line oriented. Lines are separated by a line feed character (\n). The last line must end with a line feed character.

From the curl man page:

[--data-binary] Data is posted in a similar manner as --data-ascii does, except that newlines are preserved and conversions are never done.

--data-binary does not strip or convert anything, so the input is submitted as is, and thus the trailing newline is preserved.