add support to handle ha notifications #3659

zjswhhh · 2025-05-27T02:53:02Z

What I did
Handle ha_set and ha_scope sai notifications.

HLD:
https://github.com/sonic-net/SONiC/blob/master/doc/smart-switch/high-availability/smart-switch-ha-detailed-design.md
https://github.com/sonic-net/SONiC/blob/master/doc/smart-switch/high-availability/smart-switch-ha-dpu-scope-dpu-driven-setup.md#72-ha-role-activation

Per offline discussion with @r12f, haorch will directly write DPU_STATE_DB instead of using zmq for

straightforward to implement
it's not a bottle neck of performance (one table per dpu, only 8 tables for now)

sign-off: Jing Zhang [email protected]

Why I did it

How I verified it
UTs.

Details if related

mssonicbld · 2025-05-27T02:53:09Z

/azp run

azure-pipelines · 2025-05-27T02:53:19Z

Azure Pipelines successfully started running 1 pipeline(s).

vivekrnv · 2025-05-28T03:01:24Z

orchagent/notifications.cpp

@@ -52,6 +52,18 @@ void on_twamp_session_event(uint32_t count, sai_twamp_session_event_notification
    // which causes concurrency access to the DB
 }

+void on_ha_set_event(uint32_t count, sai_ha_set_event_data_t *data)
+{
+    // don't use this event handler, because it runs by libsairedis in a separate thread


This won't work since DPU uses ZMQ. Please check this PR #3547 on what has to be done

Hi @vivekrnv , thanks for bringing it up. Can you share the doc? Per HA detailed design, there is no change mentioned for communication mode between syncd to orchagent.

There was a change done by @prabhataravind to move the communication channel to ZMQ. PFA: sonic-net/sonic-buildimage#21940

… state_activated_rq

mssonicbld · 2025-05-30T06:42:20Z

/azp run

azure-pipelines · 2025-05-30T06:42:30Z

Azure Pipelines successfully started running 1 pipeline(s).

yue-fred-gao · 2025-05-30T13:35:45Z

orchagent/dash/dashhaorch.cpp

+                        SWSS_LOG_NOTICE("DPU is pending on role activation for %s", key.c_str());
+                    }
+
+                    fvs.push_back({"ha_state", sai_ha_state_name.at(ha_scope_event[i].ha_state)});


I don't see ha_state in hld. Is it a new addition?

You are right. Will remove it.

mssonicbld · 2025-05-30T18:50:06Z

/azp run

azure-pipelines · 2025-05-30T18:50:15Z

Azure Pipelines successfully started running 1 pipeline(s).

… state_activated_rq

mssonicbld · 2025-05-30T21:42:54Z

/azp run

azure-pipelines · 2025-05-30T21:43:04Z

Azure Pipelines successfully started running 1 pipeline(s).

prabhataravind

Notification changes lgtm

zjswhhh · 2025-05-31T17:28:22Z

Hi @vivekrnv - appreciate your comments! Can you help review again?

… state_activated_rq

mssonicbld · 2025-06-01T17:37:59Z

/azp run

azure-pipelines · 2025-06-01T17:38:09Z

Azure Pipelines successfully started running 1 pipeline(s).

prsunny · 2025-06-03T16:42:58Z

@zjswhhh , most PR checker issues are fixed as of today. Do you know if the test issues are caused by the PR? Please investigate

… state_activated_rq

zjswhhh · 2025-06-03T22:32:03Z

/azpw

zjswhhh · 2025-06-05T16:49:43Z

/azp run Azure.sonic-swss

azure-pipelines · 2025-06-05T16:49:49Z

Commenter does not have sufficient privileges for PR 3659 in repo sonic-net/sonic-swss

mssonicbld · 2025-06-05T16:50:16Z

/azp run

azure-pipelines · 2025-06-05T16:50:26Z

Azure Pipelines successfully started running 1 pipeline(s).

vivekrnv · 2025-06-05T20:39:55Z

orchagent/dash/dashhaorch.cpp

+    std::string data;
+    std::vector<swss::FieldValueTuple> values;
+
+    consumer.pop(op, data, values);


You can use pops api and process multiple notifications at once.

Also, Make sure doTask(NotificationConsumer &consumer) is called during the empty doTask() Eg:

sonic-swss/orchagent/orchdaemon.cpp

Line 882 in 90fcead

o->doTask();

DashHaOrch::doTask() { # Finish pending tasks doTask(*m_haSetNotificationConsumer); doTask(*m_haScopeNotificationConsumer); }

if not there is a possibility we might miss some notifications

You can use pops api and process multiple notifications at once.

Updated.

Also, Make sure doTask(NotificationConsumer &consumer) is called during the empty doTask()

That's a great tip! I added the executors to m_consumerMap in DashHaOrch constructor

sonic-swss/orchagent/dash/dashhaorch.cpp

Lines 72 to 73 in eec055a

Orch::addExecutor(haSetNotificatier);

Orch::addExecutor(haScopeNotificatier);

So it should have been taken care of:

sonic-swss/orchagent/orch.cpp

Lines 664 to 670 in db7d939

void Orch::doTask()

{

for (auto &it : m_consumerMap)

{

it.second->drain();

}

}

Don't think I need to override the method in DashHaOrch?

Ah, i see. We should be good then.

… state_activated_rq

mssonicbld · 2025-06-08T17:59:28Z

/azp run

azure-pipelines · 2025-06-08T17:59:37Z

Azure Pipelines successfully started running 1 pipeline(s).

theasianpianist · 2025-06-09T21:18:49Z

orchagent/dash/dashhaorch.cpp

@@ -25,6 +57,125 @@ DashHaOrch::DashHaOrch(DBConnector *db, const vector<string> &tables, DashOrch *

    dash_ha_set_result_table_ = make_unique<Table>(app_state_db, APP_DASH_HA_SET_TABLE_NAME);
    dash_ha_scope_result_table_ = make_unique<Table>(app_state_db, APP_DASH_HA_SCOPE_TABLE_NAME);
+
+    m_dpuStateDbConnector = make_unique<DBConnector>("DPU_STATE_DB", 0);


VS tests are failing, looks like we will need swsscommon changes: https://github.com/sonic-net/sonic-swss-common/blob/master/common/database_config.json

… state_activated_rq

mssonicbld · 2025-06-12T04:45:45Z

/azp run

azure-pipelines · 2025-06-12T04:45:55Z

Azure Pipelines successfully started running 1 pipeline(s).

zjswhhh added 3 commits May 27, 2025 02:50

handle ha events

c0547c5

tests

08bfaea

more tests

bfa2b1d

zjswhhh requested a review from prsunny as a code owner May 27, 2025 02:53

zjswhhh changed the title ~~add support to handle ha events~~ add support to handle ha notifications May 27, 2025

zjswhhh requested review from prabhataravind, theasianpianist, r12f, yue-fred-gao and jimmyzhai and removed request for prsunny May 27, 2025 02:53

vivekrnv reviewed May 28, 2025

View reviewed changes

zjswhhh added 2 commits May 29, 2025 23:27

Merge branch 'master' of https://github.com/sonic-net/sonic-swss into…

e888141

… state_activated_rq

zmq is better for performance

92effdf

yue-fred-gao reviewed May 30, 2025

View reviewed changes

remove ha_state from ha_scope_state table

f691bf2

zjswhhh requested review from vivekrnv and yue-fred-gao May 30, 2025 18:50

yue-fred-gao approved these changes May 30, 2025

View reviewed changes

Merge branch 'master' of https://github.com/sonic-net/sonic-swss into…

2f0f842

… state_activated_rq

prabhataravind approved these changes May 31, 2025

View reviewed changes

Merge branch 'master' of https://github.com/sonic-net/sonic-swss into…

2f1749a

… state_activated_rq

Merge branch 'master' of https://github.com/sonic-net/sonic-swss into…

f08baaa

… state_activated_rq

vivekrnv reviewed Jun 5, 2025

View reviewed changes

zjswhhh added 2 commits June 8, 2025 05:09

Merge branch 'master' of https://github.com/sonic-net/sonic-swss into…

cfe98db

… state_activated_rq

use pops

eec055a

zjswhhh force-pushed the state_activated_rq branch from 6ff9ebe to eec055a Compare June 8, 2025 17:59

theasianpianist reviewed Jun 9, 2025

View reviewed changes

Merge branch 'master' of https://github.com/sonic-net/sonic-swss into…

612d3c0

… state_activated_rq

	Orch::addExecutor(haSetNotificatier);
	Orch::addExecutor(haScopeNotificatier);

	void Orch::doTask()
	{
	for (auto &it : m_consumerMap)
	{
	it.second->drain();
	}
	}

add support to handle ha notifications #3659

Are you sure you want to change the base?

add support to handle ha notifications #3659

Uh oh!

Conversation

zjswhhh commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mssonicbld commented May 27, 2025

Uh oh!

azure-pipelines bot commented May 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zjswhhh May 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented May 30, 2025

Uh oh!

azure-pipelines bot commented May 30, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented May 30, 2025

Uh oh!

azure-pipelines bot commented May 30, 2025

Uh oh!

mssonicbld commented May 30, 2025

Uh oh!

azure-pipelines bot commented May 30, 2025

Uh oh!

prabhataravind left a comment

Choose a reason for hiding this comment

Uh oh!

zjswhhh commented May 31, 2025

Uh oh!

mssonicbld commented Jun 1, 2025

Uh oh!

azure-pipelines bot commented Jun 1, 2025

Uh oh!

prsunny commented Jun 3, 2025

Uh oh!

zjswhhh commented Jun 3, 2025

Uh oh!

zjswhhh commented Jun 5, 2025

Uh oh!

azure-pipelines bot commented Jun 5, 2025

Uh oh!

mssonicbld commented Jun 5, 2025

Uh oh!

azure-pipelines bot commented Jun 5, 2025

Uh oh!

vivekrnv Jun 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Jun 8, 2025

Uh oh!

azure-pipelines bot commented Jun 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mssonicbld commented Jun 12, 2025

Uh oh!

azure-pipelines bot commented Jun 12, 2025

Uh oh!

Uh oh!

zjswhhh commented May 27, 2025 •

edited

Loading

zjswhhh May 28, 2025 •

edited

Loading

vivekrnv Jun 5, 2025 •

edited

Loading