"ValueError: Objects not equal" in 'tests.test_key_triggers'

[arm4b]

Tracking here the following error message which is caused by some race condition:

................
id: 59f7685026e6f06c72999755
action.ref: tests.test_key_triggers
parameters: 
  protocol: http
  token: 315b6e30d6594c57ada2c6a61bbd6e4f
status: failed
result_task: assert_trigger_instance_payload_matches_expected_create
result: 
  exit_code: 1
  result: None
  stderr: "u'Input: \\nname:fixtures.ABCMeta:cache'\nu'Expected: \\nname:a'\nTraceback (most recent call last):\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py\", line 259, in <module>\n    obj.run()\n  File \"/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py\", line 155, in run\n    output = action.run(**self._parameters)\n  File \"/opt/stackstorm/packs/asserts/actions/object_contains.py\", line 27, in run\n    (object, expected))\nValueError: Objects not equal. Input: {u'object': {u'uid': u'key_value_pair:st2kv.system:fixtures.ABCMeta:cache', u'encrypted': True, u'value': u'000AE0ABBE7BD10AD660FD684072AC44945D50CBC7DBB0A66A17B9FFA2A2773EC9097E1E5B22331492294570F0E73FF93309DAF07E9E9BC82C', u'secret': True, u'scope': u'st2kv.system', u'id': u'59f7686826e6f06c72999765', u'name': u'fixtures.ABCMeta:cache'}}, Expected: {u'object': {u'name': u'a', u'encrypted': False, u'value': u'b', u'secret': False, u'scope': u'st2kv.system', u'uid': u'key_value_pair:st2kv.system:a'}}.\n"
  stdout: ''
error: u'Input: 
name:fixtures.ABCMeta:cache'
u'Expected: 
name:a'
Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py", line 259, in <module>
    obj.run()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py", line 155, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/asserts/actions/object_contains.py", line 27, in run
    (object, expected))
ValueError: Objects not equal. Input: {u'object': {u'uid': u'key_value_pair:st2kv.system:fixtures.ABCMeta:cache', u'encrypted': True, u'value': u'000AE0ABBE7BD10AD660FD684072AC44945D50CBC7DBB0A66A17B9FFA2A2773EC9097E1E5B22331492294570F0E73FF93309DAF07E9E9BC82C', u'secret': True, u'scope': u'st2kv.system', u'id': u'59f7686826e6f06c72999765', u'name': u'fixtures.ABCMeta:cache'}}, Expected: {u'object': {u'name': u'a', u'encrypted': False, u'value': u'b', u'secret': False, u'scope': u'st2kv.system', u'uid': u'key_value_pair:st2kv.system:a'}}.

traceback: None
failed_on: assert_trigger_instance_payload_matches_expected_create
start_timestamp: 2017-10-30T17:58:40.909907Z
end_timestamp: 2017-10-30T17:59:13.214101Z

So far seen on Ubuntu14 and Ubuntu16, our st2cicd build server.

The failing task in st2tests is:

st2tests/packs/tests/actions/chains/test_key_triggers.yaml

Lines 98 to 111 in ddb6c82

	-
	name: assert_trigger_instance_payload_matches_expected_create
	ref: asserts.object_contains
	params:
	object: "{{ retrieve_key_create_trigger_instance.stdout.payload }}"
	expected:
	object:
	name: a
	value: b
	encrypted: false
	secret: false
	scope: st2kv.system
	uid: "key_value_pair:st2kv.system:a"
	on-success: update_key