Treat Activity start as the first heartbeat time

[tsurdilo]

Currently SDK will start throttling heartbeats after the first one is sent (first one called as part of activity code).
In some use cases this is not optimal where heartbeat throttling should be started at activity start.
This is especially critical for cloud customers where heartbeat is a billable action.

Ask is to treat Activity start as the first heartbeat time so first actual heartbeat sent would be at 4/5 of HeartbeatTimeout defined.

[cretz]

This technically would be a bit of a compatibility break, because people may have built an expectation to see their heartbeat details for their first heartbeat on the pending activity well before the 4/5ths of the timeout has elapsed (e.g. if they have heartbeat w/ details as the first line of their activity and the heartbeat timeout is 10m, this issue would cause the first details data to be delayed by 8m). But this may not be a big deal.

It is also worth noting that there is a difference between when activity code starts and the server believes an activity attempt starts. What if your heartbeat timeout is 5s but the time it took from when the activity attempt started and when it actually started your code (because of various worker things happening, e.g. thread pool limitations) is 3s. We can't wait another 4s to send that heartbeat, we need to send it now. And calculating this time from the server's start time is no good due to clock skew, so workers will need to capture the time of poll receipt and use that to diff against first heartbeat time to see if throttling needed.

Not to say we can't do this, but there are some advanced things to consider.

[tsurdilo]

Thanks for the inputs! Another possible solution could be a “force heartbeat” flag on the heartbeat itself which guarantees a synchronous call to the service.
With this idea is to still throttle even first heartbeat at 4/5 of HeartbeatTimeout, but for users that want to emit heartbeat on activity start (to record payload as you mentioned), they can use this flag

cc @mfateev to add more details please if needed

[mfateev]

I believe adding a "force heartbeat" option and not emitting the first heartbeat right away is a pretty intuitive behavior. I agree that the clock skew might be an issue that needs to be accounted for.