chatops connect to Slack always fail - st2 3.2.0

[dove-young]

Using hubot token

 HUBOT_SLACK_TOKEN=xoxb-my-hubot-token \
> HUBOT_ADAPTER=slack \
> HUBOT_LOG_LEVEL=debug \
> HUBOT_SLACK_RTM_CLIENT_OPTS='{ "logLevel": "debug" }' \
> HUBOT_SLACK_RTM_CLIENT_OPTS='{ "retryConfig": { "retries": 20 } }' \
> HUBOT_SLACK_RTM_START_OPTS='{ "mpim_aware": true }' \
>  ./bin/hubot --adapter slack
body-parser deprecated undefined extended: provide extended option node_modules/hubot/src/robot.js:445:21
[Wed Jun 17 2020 22:25:02 GMT-0700 (Pacific Daylight Time)] DEBUG Loading adapter slack
[Wed Jun 17 2020 22:25:03 GMT-0700 (Pacific Daylight Time)] INFO hubot-slack adapter v4.5.5
warn: SlackDataStore is deprecated and will be removed in the next major version. See project documentation for a migration guide.
warn: SlackDataStore is deprecated and will be removed in the next major version. See project documentation for a migration guide.
[Wed Jun 17 2020 22:25:03 GMT-0700 (Pacific Daylight Time)] DEBUG RtmClient initialized with options: {"retryConfig":{"retries":20},"maxRequestConcurrency":1}
[Wed Jun 17 2020 22:25:03 GMT-0700 (Pacific Daylight Time)] DEBUG RtmClient#start() with options: {"mpim_aware":true}
[Wed Jun 17 2020 22:25:04 GMT-0700 (Pacific Daylight Time)] DEBUG SlackBot#authenticated() Found self in RTM start data
[Wed Jun 17 2020 22:25:04 GMT-0700 (Pacific Daylight Time)] INFO Logged in as @hubot in workspace hubot-test-20
[Wed Jun 17 2020 22:25:05 GMT-0700 (Pacific Daylight Time)] INFO Connected to Slack RTM
[Wed Jun 17 2020 22:25:05 GMT-0700 (Pacific Daylight Time)] DEBUG Loading scripts from /opt/stackstorm/chatops/scripts
[Wed Jun 17 2020 22:25:05 GMT-0700 (Pacific Daylight Time)] DEBUG Loading scripts from /opt/stackstorm/chatops/src/scripts
[Wed Jun 17 2020 22:25:05 GMT-0700 (Pacific Daylight Time)] DEBUG Loading external-scripts from npm packages
[Wed Jun 17 2020 22:25:05 GMT-0700 (Pacific Daylight Time)] DEBUG Using slack adapter
[Wed Jun 17 2020 22:25:05 GMT-0700 (Pacific Daylight Time)] INFO Requesting a token...
[Wed Jun 17 2020 22:25:05 GMT-0700 (Pacific Daylight Time)] DEBUG Parsing help for /opt/stackstorm/chatops/node_modules/hubot-stackstorm/src/stackstorm.js
[Wed Jun 17 2020 22:25:05 GMT-0700 (Pacific Daylight Time)] DEBUG Parsing help for /opt/stackstorm/chatops/node_modules/hubot-help/src/help.coffee
[Wed Jun 17 2020 22:25:05 GMT-0700 (Pacific Daylight Time)] ERROR Failed to authenticate: Invalid or missing credentials
[Wed Jun 17 2020 22:25:05 GMT-0700 (Pacific Daylight Time)] ERROR undefined
[Wed Jun 17 2020 22:25:05 GMT-0700 (Pacific Daylight Time)] ERROR {"name":"APIError","status":401,"message":"Invalid or missing credentials"}
[Wed Jun 17 2020 22:25:05 GMT-0700 (Pacific Daylight Time)] INFO Hubot will shut down ...
[Wed Jun 17 2020 22:25:05 GMT-0700 (Pacific Daylight Time)] INFO Disconnected from Slack RTM
[Wed Jun 17 2020 22:25:05 GMT-0700 (Pacific Daylight Time)] INFO Exiting...

Using a bot user token

 HUBOT_SLACK_TOKEN=xoxb-my-bot-token \
> HUBOT_ADAPTER=slack \
> HUBOT_LOG_LEVEL=debug \
> HUBOT_SLACK_RTM_CLIENT_OPTS='{ "logLevel": "debug" }' \
> HUBOT_SLACK_RTM_CLIENT_OPTS='{ "retryConfig": { "retries": 20 } }' \
> HUBOT_SLACK_RTM_START_OPTS='{ "mpim_aware": true }' \
>  ./bin/hubot --adapter slack
body-parser deprecated undefined extended: provide extended option node_modules/hubot/src/robot.js:445:21
[Wed Jun 17 2020 22:19:27 GMT-0700 (Pacific Daylight Time)] DEBUG Loading adapter slack
[Wed Jun 17 2020 22:19:28 GMT-0700 (Pacific Daylight Time)] INFO hubot-slack adapter v4.5.5
warn: SlackDataStore is deprecated and will be removed in the next major version. See project documentation for a migration guide.
warn: SlackDataStore is deprecated and will be removed in the next major version. See project documentation for a migration guide.
[Wed Jun 17 2020 22:19:28 GMT-0700 (Pacific Daylight Time)] DEBUG RtmClient initialized with options: {"retryConfig":{"retries":20},"maxRequestConcurrency":1}
[Wed Jun 17 2020 22:19:28 GMT-0700 (Pacific Daylight Time)] DEBUG RtmClient#start() with options: {"mpim_aware":true}
[Wed Jun 17 2020 22:19:29 GMT-0700 (Pacific Daylight Time)] DEBUG SlackBot#authenticated() Found self in RTM start data
[Wed Jun 17 2020 22:19:29 GMT-0700 (Pacific Daylight Time)] INFO Logged in as @ansible-bot in workspace hubot-test-20
[Wed Jun 17 2020 22:19:30 GMT-0700 (Pacific Daylight Time)] INFO Connected to Slack RTM
[Wed Jun 17 2020 22:19:30 GMT-0700 (Pacific Daylight Time)] DEBUG Loading scripts from /opt/stackstorm/chatops/scripts
[Wed Jun 17 2020 22:19:30 GMT-0700 (Pacific Daylight Time)] DEBUG Loading scripts from /opt/stackstorm/chatops/src/scripts
[Wed Jun 17 2020 22:19:30 GMT-0700 (Pacific Daylight Time)] DEBUG Loading external-scripts from npm packages
[Wed Jun 17 2020 22:19:30 GMT-0700 (Pacific Daylight Time)] DEBUG Using slack adapter
[Wed Jun 17 2020 22:19:30 GMT-0700 (Pacific Daylight Time)] INFO Requesting a token...
[Wed Jun 17 2020 22:19:30 GMT-0700 (Pacific Daylight Time)] DEBUG Parsing help for /opt/stackstorm/chatops/node_modules/hubot-stackstorm/src/stackstorm.js
[Wed Jun 17 2020 22:19:30 GMT-0700 (Pacific Daylight Time)] DEBUG Parsing help for /opt/stackstorm/chatops/node_modules/hubot-help/src/help.coffee
[Wed Jun 17 2020 22:19:30 GMT-0700 (Pacific Daylight Time)] ERROR Failed to authenticate: Invalid or missing credentials
[Wed Jun 17 2020 22:19:30 GMT-0700 (Pacific Daylight Time)] ERROR undefined
[Wed Jun 17 2020 22:19:30 GMT-0700 (Pacific Daylight Time)] ERROR {"name":"APIError","status":401,"message":"Invalid or missing credentials"}
[Wed Jun 17 2020 22:19:30 GMT-0700 (Pacific Daylight Time)] INFO Hubot will shut down ...
[Wed Jun 17 2020 22:19:30 GMT-0700 (Pacific Daylight Time)] INFO Disconnected from Slack RTM
[Wed Jun 17 2020 22:19:30 GMT-0700 (Pacific Daylight Time)] INFO Exiting...

And the hubot token I used here is still working

Here is the env version

[root@ chatops]# st2 --version
st2 3.2.0, on Python 2.7.5
[root@ chatops]#
[root@ chatops]# bin/hubot -v
Must provide HUBOT_ADAPTER ENV variable to start.
See README.md for more information
[root@ chatops]# HUBOT_ADAPTER=slack bin/hubot -v
body-parser deprecated undefined extended: provide extended option node_modules/hubot/src/robot.js:445:21
[Wed Jun 17 2020 22:54:16 GMT-0700 (Pacific Daylight Time)] DEBUG Loading adapter slack
[Wed Jun 17 2020 22:54:17 GMT-0700 (Pacific Daylight Time)] INFO hubot-slack adapter v4.5.5
warn: SlackDataStore is deprecated and will be removed in the next major version. See project documentation for a migration guide.
warn: SlackDataStore is deprecated and will be removed in the next major version. See project documentation for a migration guide.
[Wed Jun 17 2020 22:54:17 GMT-0700 (Pacific Daylight Time)] DEBUG RtmClient initialized with options: undefined
3.1.1
[root@ chatops]#

[dove-young]

Using the same token in St 3.1.0 works well.

HUBOT_SLACK_TOKEN=xoxb-my-bot-user-token \
> HUBOT_ADAPTER=slack \
> HUBOT_LOG_LEVEL=debug \
> HUBOT_SLACK_RTM_CLIENT_OPTS='{ "logLevel": "debug" }' \
> HUBOT_SLACK_RTM_CLIENT_OPTS='{ "retryConfig": { "retries": 20 } }' \
> HUBOT_SLACK_RTM_START_OPTS='{ "mpim_aware": true }' \
>  ./bin/hubot --adapter slack
body-parser deprecated undefined extended: provide extended option node_modules/hubot/src/robot.js:445:21
[Wed Jun 17 2020 22:45:11 GMT-0700 (Pacific Daylight Time)] DEBUG Loading adapter slack
[Wed Jun 17 2020 22:45:12 GMT-0700 (Pacific Daylight Time)] INFO hubot-slack adapter v4.5.5
warn: SlackDataStore is deprecated and will be removed in the next major version. See project documentation for a migration guide.
warn: SlackDataStore is deprecated and will be removed in the next major version. See project documentation for a migration guide.
[Wed Jun 17 2020 22:45:12 GMT-0700 (Pacific Daylight Time)] DEBUG RtmClient initialized with options: {"retryConfig":{"retries":20},"maxRequestConcurrency":1}
[Wed Jun 17 2020 22:45:12 GMT-0700 (Pacific Daylight Time)] DEBUG RtmClient#start() with options: {"mpim_aware":true}
[Wed Jun 17 2020 22:45:12 GMT-0700 (Pacific Daylight Time)] ERROR Error: listen EADDRINUSE: address already in use 0.0.0.0:8081
    at Server.setupListenHandle [as _listen2] (net.js:1309:16)
    at listenInCluster (net.js:1357:12)
    at doListen (net.js:1496:7)
    at processTicksAndRejections (internal/process/task_queues.js:85:21)

(node:1907) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] DEBUG SlackBot#authenticated() Found self in RTM start data
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] INFO Logged in as @hubot_sre in workspace sre-test
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] INFO Connected to Slack RTM
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] DEBUG Loading scripts from /opt/stackstorm/chatops/scripts
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] DEBUG Loading scripts from /opt/stackstorm/chatops/src/scripts
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] DEBUG Loading external-scripts from npm packages
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] INFO Using ST2_API_KEY as authentication. Expiry will lead to bot exit.
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] DEBUG Using slack post data handler.
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] DEBUG Parsing help for /opt/stackstorm/chatops/node_modules/hubot-stackstorm/scripts/stackstorm.js
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] INFO Loading commands....
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] DEBUG Parsing help for /opt/stackstorm/chatops/node_modules/hubot-help/src/help.coffee
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] DEBUG Added command: cem find all incidents filter by <query string> - Return a list of incidents
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] DEBUG Added command: cem {{action=find}} all {{obj_type=incidents}} filter by {{filter}} in cluster {{cluster}} - Return a list of incidents
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] DEBUG Added command: cem {{action=find}} all {{obj_type=incidents}} filter by {{filter}} - Return a list of incidents
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] DEBUG Added command: cem find major incidents - Return a list of incidents with priority<=2
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] DEBUG Added command: cem find major (incidents|incident|inc) {{action=find}} {{obj_type=incidents}} {{filter=priority <= 2 and state != 'close
d'}} in cluster {{cluster}} - Return a list of incidents with priority<=2
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] DEBUG Added command: cem find major (incidents|incident|inc) {{action=find}} {{obj_type=incidents}} {{filter=priority <= 2 and state != 'close
d'}} - Return a list of incidents with priority<=2
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] DEBUG Added command: cem find incidents "owner == 'name'" - Return a list of incidents for specific owner
[Wed Jun 17 2020 22:45:13 GMT-0700 (Pacific Daylight Time)] DEBUG Added command: cem find (incidents|incident|inc) {{filter}} {{action=find}} {{obj_type=incidents}} in cluster {{cluster}} - Return a lis
t of incidents for specific owner

bin/hubot -v
body-parser deprecated undefined extended: provide extended option node_modules/hubot/src/robot.js:445:21
[Wed Jun 17 2020 22:52:08 GMT-0700 (Pacific Daylight Time)] DEBUG Loading adapter slack
[Wed Jun 17 2020 22:52:08 GMT-0700 (Pacific Daylight Time)] INFO hubot-slack adapter v4.5.5
warn: SlackDataStore is deprecated and will be removed in the next major version. See project documentation for a migration guide.
warn: SlackDataStore is deprecated and will be removed in the next major version. See project documentation for a migration guide.
[Wed Jun 17 2020 22:52:08 GMT-0700 (Pacific Daylight Time)] DEBUG RtmClient initialized with options: undefined
3.1.1

st2 --version
st2 3.1.0, on Python 2.7.5