ST --> MQTT Bridge messages not retained

[jmbinette]

Hi,

I am implementing a ESP8266 sensor with ESPHome library and everything works fine so far.
The sensor is running for 60s and enter deep sleep for 15min.

I am using a "ota_switch" via MQTT to prevent going to deep sleep. That way, I am able to update the sensor via OTA.

The problem is that the message the MQTT publishes following a state change on ST is not persistent. If I happen to "time" the message during the 60s runtime, my logic works perfectly. But if push the virtual button during deep sleep, it is not retained until next boot.

I have the same behavior using MQTT Explorer, the message is not retained.
The weird thing is "MQTT --> ST" is always retain and I have no issues at all

[sgupta999]

Just want to make sure I understand what you are saying:

But if push the virtual button during deep sleep, it is not retained until next boot.

next reboot of what? The device? ST? MBS-Server

The way I think it is supposed to work:

1. Since ST isa wake all the time if you push virtual button in ST, and MBS server is running, and retain settings are set correctly - the message should be published to MQTT broker and be retained.
2. If your ESP8266 is in deep sleep it will not receive that message from MQTT broker since the wifi for the device is off.
3. Remember MQTT will retain the last message published to that topic.
4. If now your device comes out of deep sleep and makes a wifi and MQTT connection it will receive the last retained messages from the MQTT broker.
5. If however your device itself is configured to retain last state before deep sleep and it is configured to wake up to its last state (for e.g. if you flashed it with tasmota and that attribute if default) then what ever retained message you receive from MQTT broker might be getting overwritten or ignored by device itself.

if you clarify your flow it might make more sense.

In summary:
1.Anything opening a new connection to an MQTT broker and subscribing to a topic will receive the last retained message for that topic.
2. How the device/software receiving this message reacts with that retained message will depend on the device/software specific implementation - you may choose to ignore it, override it or use it.

To check if MBS-server retained setting is working here is what i would do.

1. Power off the device and make sure no other app or software is publishing to the same topic.
2. Send command from your ST and make sure it is published by ST->MQTT in the logs.
3. Now restart MBS-server.
4. When MBS-server reconnects to MQTT broker the log will print all the retained nessages for subscribed topic from the broker that will not be printed as MQTT -> ST. Typically you dont subscribe to topics you are publishing to but another topic (like a status topic) - so this may or may not reflect the retained settings.

hope this clarifies

[sgupta999]

I don't think you can wake from deep sleep using MQTT since wifi is down during deep sleep and MQTT message will not be received. I think a D1 and RST have to be shorted (or the other way round) for it to come out of deep sleep or like you are doing waking up with a timer.

[jmbinette]

I am not trying to wake the device but prevent from going to deep sleep again

The message is published correctly to clients that are already connected.
If I send the message and connect afterwards, all the "subscribed" topics (MQTT --> ST) are present.
But no "retained" messages (ST --> MQTT) are present

Hope this is more clear

[sgupta999]

If I happen to "time" the message during the 60s runtime, my logic works perfectly

not sure I follow - if all clients that are connected are getting the message and you have retained setting set correctly under what circumstances is it working?

Are you sure message from ST is the last message published to that topic?
How are you confirming that messages are not being retained?

[sgupta999]

I have created a test server
https://github.com/sgupta999/mqtt-bridge-smartthings/blob/master/test/mbs-server-retain-test.js

try using this. Here the log will show if we are sending the right true flag to the MQTT broker - if that shows 'true' for the commands that you are sending and still cannot confirm messages are being retained the problem might be with the broker or device and we can troubleshoot those once we can eliminate MBS-server

[jmbinette]

The retain flag seems to be true :
Jan 23 09:16:09 raspberrry mqtt-bridge-smartthings[23599]: 2020-01-23 09:16:09 AM info: ST** --> MQTT: [winesensor_cellier_1][switch][on] [smartthings/raspberrry /winesensor_cellier_1/switch/state][on][Retain:true]

If I start MQTT explorer after this, I see all topics except "winesensor_cellier_1/switch/state"
If I switch the state when MQTT explorer is connected, I see the topic appear ...

[sgupta999]

You've probably already check this but just to confirm:
The topic you are publishing to is
smartthings/raspberrry /winesensor_cellier_1/switch/state

so that is what message will be retained to. But the topic you mention you are looking for when MQTT explorer starts is
winesensor_cellier_1/switch/state
just confirming the topic you are tracking in MQTT explorer is
smartthings/raspberrry /winesensor_cellier_1/switch/state
and not
winesensor_cellier_1/switch/state

so by extention when you say

If I switch the state when MQTT explorer is connected, I see the topic appear

you mean the topic
smartthings/raspberrry /winesensor_cellier_1/switch/state appears
and not
winesensor_cellier_1/switch/state

[sgupta999]

check on other thing for me if you could please:
it could be that ST publishes to
smartthings/raspberrry /winesensor_cellier_1/switch/state

the eSP8266 is subscribed to that topic so it responds to that by changing state.
then the ESP8266 publishes a state change to topic
winesensor_cellier_1/switch/state

which is is being picked by MQTT explorer

just wondering if that could be the scenario

Also noticing a space between raspberry and '/' just wondering if that could be an issue if inadvertently the subscription topic is missing that space.

[jmbinette]

Yes you are right I reduced the lenght of the topic for sake of simplifying things but the topics are the same.

The device is publishing temperature, humidity, etc and they are retained correctly.

FYI, I just did the test :
mosquitto_pub.exe -h 192.168.xx.xx -r -t smartthings/raspberrry/winesensor_cellier_1/switch/state -m "on"

The message is correctly retained and the deep sleep is prevented on my device

[jmbinette]

check on other thing for me if you could please:
it could be that ST publishes to
smartthings/raspberrry /winesensor_cellier_1/switch/state

the eSP8266 is subscribed to that topic so it responds to that by changing state.
then the ESP8266 publishes a state change to topic
winesensor_cellier_1/switch/state

which is is being picked by MQTT explorer

just wondering if that could be the scenario

The ESP doesn't publish state change, it just subscribe to the topic and execute actions when payload is "on"

[sgupta999]

Also noticing a space between raspberry and '/' just wondering if that could be an issue if inadvertently the subscription topic is missing that space.

If that is not the case then let explore some other scenarios:

1. So if the device is on and you are trying to keep it awake it should be getting the same publish message smartthings/raspberrry/winesensor_cellier_1/switch/state "on" from mbs-server
2. Not sure I understand why MQTT explorer being active or inactive would have anything to do the message going from mbs-server to mqtt broker to the esp8266.
3. if esp8266 is on and message is being received then it should not go into deep sleep depending on how you have coded it (or ESPHome - i have not worked with that)
4. How is the fact that the message is retained or not preventing deep sleep - i am assuming as long as you send the message to esp8266 while it is not in deep sleep it will stay on

[jmbinette]

No don't worry I replace the name of my server for sake of privacy lol ... there is no "typo"
But you are right asking

[sgupta999]

looks like u r using mosquitto.
Just checking for outliars

1. are you bridging mqtt brokers
2. are u using username / password

[jmbinette]

looks like u r using mosquitto.
Just checking for outliars

1. are you bridging mqtt brokers

You mean multiple brokers ? No I use the Smartthings bridge and Mosquitto on the same raspberry pi

2. are u using username / password

no

[sgupta999]

thnx could you respond to other comment above also please

[jmbinette]

Also noticing a space between raspberry and '/' just wondering if that could be an issue if inadvertently the subscription topic is missing that space.

If that is not the case then let explore some other scenarios:

1. So if the device is on and you are trying to keep it awake it should be getting the same publish message smartthings/raspberrry/winesensor_cellier_1/switch/state "on" from mbs-server

Yes but for some reason not retained when the device wake up after deep sleep

2. Not sure I understand why MQTT explorer being active or inactive would have anything to do the message going from mbs-server to mqtt broker to the esp8266.

Just a way of troubleshooting it. Disconnecting and connecting MQTT explorer is like my esp8266 waking up and going back to sleep

3. if esp8266 is on and message is being received then it should not go into deep sleep depending on how you have coded it (or ESPHome - i have not worked with that)

Yes but only if published while the device is on (60s every 15min)

4. How is the fact that the message is retained or not preventing deep sleep - i am assuming as long as you send the message to esp8266 while it is not in deep sleep it will stay on

It is hard to time correctly (60s every 15min) so the goal is to publish the message, wait maximum 15 minutes and be 100% sure the device will not be sleeping

[jmbinette]

Could I send you privately my config files and device handlers ? Via e-mail ?

[sgupta999]

sure please feel free to send me.
also I made a small change in the mbs-server-retain-test to more explicitly define the retain option.
could you download the same test file again and try to see if it fixes it.

i think this change should fix it.

my email should be there

[jmbinette]

Same behavior

Could it be because "switch" is a reserved system name in ST??

Cannot find your email address sorry

[sgupta999]

that is odd.
could you just confirm you are renaming this to mbs-server.js and that line 328 of the server file reads

client.publish(topic, value, {retain: retain}, function () 

and the log is still showing ST-MQTT Retain:true?

i was so confident this would fix it.

using "switch" should not be an issue

[jmbinette]

Oh yes you got it !

correct line is :
client.publish(topic, value, {retain: retainflag}, function () {

[jmbinette]

Nice job ! :)

Works good now !

Could I still send you my files and you tell me if I am using the devices.yml correctly ?

I modified MBS smartapp and not really sure I had to.

But now the messages are correctly retained 💯

[sgupta999]

sure - send me your configs and i can look at them
yeah dont think u would have had to modify smartapp for this

thx for finding and troubleshooting this - careless mistake on my end

[sgupta999]

dont think that email is forwarding

try this

[email protected]