[reboot-cause] Use UTC to ensure consistent sorting by jianyuewu · Pull Request #293 · sonic-net/sonic-host-services

jianyuewu · 2025-07-29T09:36:00Z

Previously reboot cause uses local timezone. This could result in the timestamp being in UTC, IDT, or any other timezone depending on the system configuration. When reboot cause history files are sorted, mixing different timezones can lead to incorrect chronological order and cause reboot cause test failure.
Now change the code to use datetime.datetime.utcnow() for reboot_cause_gen_time, ensuring that all timestamps are consistently in UTC.

Details:
Previously reboot-cause history will sort by name, name is using system time, could be UTC or IDT.
2025_07_28_10_48_14 reboot Mon Jul 28 01:45:45 PM IDT 2025 admin N/A
Here left side 10_48_14 is UTC time, while right side 01:45:45 is IDT time.
2025_07_28_13_41_44 reboot Mon Jul 28 01:39:13 PM IDT 2025 admin N/A
Left and right are both IDT time.
We need to make left side time aligned, so sort index will be always correct.

show reboot-cause history cmd:

admin@r-bison-06:~$ show reboot-cause history 
Name                 Cause       Time                             User    Comment                                                                                                                                                                                                                                
-------------------  ----------  -------------------------------  ------  ---------                                                                                                                                                                                                                                      
2025_07_28_13_41_44  reboot      Mon Jul 28 01:39:13 PM IDT 2025  admin   N/A                                                                                                                                                                                                                                           
2025_07_28_13_34_54  reboot      Mon Jul 28 01:32:25 PM IDT 2025  admin   N/A                                                                                                                                                                                                                                            
2025_07_28_13_28_13  Watchdog    N/A                              N/A     Unknown                                                                                                                                                                                                              
2025_07_28_13_22_28  reboot      Mon Jul 28 01:19:58 PM IDT 2025  admin   N/A                                                                                                                                                                                                                                  
2025_07_28_13_17_05  reboot      Mon Jul 28 10:15:33 AM UTC 2025  admin   N/A    // Time should be 01:15:33 AM
2025_07_28_10_48_14  reboot      Mon Jul 28 01:45:45 PM IDT 2025  admin   N/A    // This entry should be after 01:39:13 PM IDT                                                                                                                                                                                                                     
2025_07_28_09_57_42  Power Loss  N/A                              N/A     Unknown                                                                                                           
2025_07_28_09_52_04  Power Loss  N/A                              N/A     Unknown                                                                                                                                                                                                                                        
2025_07_28_09_46_15  Power Loss  N/A                              N/A     Unknown                                                                                                          
2025_07_28_09_40_22  Power Loss  N/A                              N/A     Unknown

Reboot cause history in test:

13:53:45 reboot.check_reboot_cause_history        L0532 INFO   | index:  6, reboot cause: Watchdog, reboot cause from DUT: reboot                                                                                                                                                                                        
13:53:45 reboot.check_reboot_cause_history        L0537 ERROR  | The 6 reboot-cause not match. expected_reboot type=Watchdog, actual_reboot_cause=reboot                                                                                                                                                                 
13:53:45 reboot.check_reboot_cause_history        L0541 INFO   | Current reboot_type_history_queue content:                                                                                                                                                                                                              
13:53:45 reboot.check_reboot_cause_history        L0543 INFO   |   index=0, reboot_type=power off
13:53:45 reboot.check_reboot_cause_history        L0543 INFO   |   index=1, reboot_type=power off
13:53:45 reboot.check_reboot_cause_history        L0543 INFO   |   index=2, reboot_type=power off
13:53:45 reboot.check_reboot_cause_history        L0543 INFO   |   index=3, reboot_type=power off
13:53:45 reboot.check_reboot_cause_history        L0543 INFO   |   index=4, reboot_type=cold
13:53:45 reboot.check_reboot_cause_history        L0543 INFO   |   index=5, reboot_type=cold
13:53:45 reboot.check_reboot_cause_history        L0543 INFO   |   index=6, reboot_type=watchdog
13:53:45 reboot.check_reboot_cause_history        L0543 INFO   |   index=7, reboot_type=cold
13:53:45 reboot.check_reboot_cause_history        L0543 INFO   |   index=8, reboot_type=cold
13:53:45 reboot.check_reboot_cause_history        L0543 INFO   |   index=9, reboot_type=cold

After fix, now both timestamps use UTC.

$ show reboot-cause history 
Name                 Cause        Time                             User    Comment
-------------------  -----------  -------------------------------  ------  ---------
2025_08_06_05_45_15  reboot       Wed Aug  6 05:44:00 AM UTC 2025  admin   N/A
2025_08_06_05_43_13  reboot       Wed Aug  6 05:41:59 AM UTC 2025  admin   N/A
2025_08_06_05_40_56  reboot       Wed Aug  6 05:39:41 AM UTC 2025  admin   N/A
2025_08_06_05_38_42  reboot       Wed Aug  6 05:35:43 AM UTC 2025  admin   N/A
2025_08_06_03_20_24  reboot       Wed Aug  6 03:17:30 AM UTC 2025  admin   N/A
2025_08_01_14_18_34  fast-reboot  Fri Aug  1 02:17:17 PM UTC 2025  admin   N/A
2025_08_01_13_54_35  fast-reboot  Fri Aug  1 01:53:17 PM UTC 2025  admin   N/A
2025_08_01_13_28_36  fast-reboot  Fri Aug  1 01:27:18 PM UTC 2025  admin   N/A
2025_08_01_13_23_31  fast-reboot  Fri Aug  1 01:22:14 PM UTC 2025  admin   N/A
2025_08_01_10_29_28  fast-reboot  Fri Aug  1 10:28:12 AM UTC 2025  admin   N/A

Previously reboot cause uses local timezone. This could result in the timestamp being in UTC, IDT, or any other timezone depending on the system configuration. When reboot cause history files are sorted, mixing different timezones can lead to incorrect chronological order and cause reboot cause test failure. Now change the code to use datetime.datetime.utcnow() for reboot_cause_gen_time, ensuring that all timestamps are consistently in UTC. Signed-off-by: Jianyue Wu <[email protected]>

Signed-off-by: Jianyue Wu <[email protected]>

mssonicbld · 2025-07-29T09:36:08Z

/azp run

azure-pipelines · 2025-07-29T09:36:17Z

Azure Pipelines successfully started running 1 pipeline(s).

r12f · 2025-07-31T15:32:00Z

hi @vvolam , do you mind to help reviewing this PR?

vvolam

LGTM, same change might be needed in scripts/procdockerstatsd as we are still using systemtime.

sonic-host-services/scripts/procdockerstatsd

Line 248 in 0028383

datetimeobj = datetime.now()

Could you also update the PR description with the reboot history command after the fix?

scripts/determine-reboot-cause

jianyuewu · 2025-08-06T01:29:06Z

LGTM, same change might be needed in scripts/procdockerstatsd as we are still using systemtime.

sonic-host-services/scripts/procdockerstatsd

Line 248 in 0028383

datetimeobj = datetime.now()

Could you also update the PR description with the reboot history command after the fix?

OK, will also update this timestamp, and also put new reboot history.

Update 2nd timestamp in reboot cause, also use UTC. Mon Jul 28 10:15:33 AM UTC 2025 Signed-off-by: Jianyue Wu <[email protected]>

mssonicbld · 2025-08-06T02:11:50Z

/azp run

azure-pipelines · 2025-08-06T02:11:59Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Jianyue Wu <[email protected]>

mssonicbld · 2025-08-06T08:17:14Z

/azp run

azure-pipelines · 2025-08-06T08:17:24Z

Azure Pipelines successfully started running 1 pipeline(s).

jianyuewu · 2025-08-06T08:26:42Z

LGTM, same change might be needed in scripts/procdockerstatsd as we are still using systemtime.

sonic-host-services/scripts/procdockerstatsd

Line 248 in 0028383

datetimeobj = datetime.now()

Could you also update the PR description with the reboot history command after the fix?

OK, will also update this timestamp, and also put new reboot history.

Code updated, new reboot history put in description.

jianyuewu · 2025-08-06T08:27:43Z

/azpw run

mssonicbld · 2025-08-06T08:27:46Z

/AzurePipelines run

azure-pipelines · 2025-08-06T08:27:54Z

Azure Pipelines successfully started running 1 pipeline(s).

Signed-off-by: Jianyue Wu <[email protected]>

mssonicbld · 2025-08-06T11:58:30Z

/azp run

azure-pipelines · 2025-08-06T11:58:40Z

Azure Pipelines successfully started running 1 pipeline(s).

jianyuewu · 2025-08-06T11:59:32Z

I tried to run in local via pdb, seems that updated line is covered. Not sure why that line is still not in coverage report.

PDB set_trace (IO-capturing turned off)
/sonic/src/sonic-host-services/scripts/procdockerstatsd(250)run()
-> datetimeobj = datetime.utcnow()

jianyuewu · 2025-08-19T01:52:35Z

@judyjoseph Could you help review this change? Thanks😊

jianyuewu · 2025-08-21T02:44:18Z

@vvolam @judyjoseph , Could you help also review PR for those branches? Thank you~
202505 PR:
#303
For 202412, seems didn't find related branch in this repo, could you help indicate if this repo is correct? Thank you~
https://github.com/[Azure/sonic-host-services.msft](https://github.com/Azure/sonic-host-services.msft#)#

judyjoseph

LGTM

kperumalbfn · 2025-08-25T18:24:54Z

@jianyuewu could you provide 202411 PR as this has CP conflict

yejianquan · 2025-08-26T06:53:13Z

Hi @judyjoseph , could you help to suggest whether this PR should go to 202505 branch, thanks in advance

r12f · 2025-08-26T16:43:57Z

hi @jianyuewu , do you mind helping pick this change to 202411?

jianyuewu · 2025-08-27T06:45:16Z

hi @jianyuewu , do you mind helping pick this change to 202411?

@r12f Sure, backport PR created: #306

yejianquan · 2025-08-28T01:13:12Z

offline synced with Judy, good to backport to 202505

r12f · 2025-08-28T02:28:16Z

hi @jianyuewu , do you mind helping pick this change to 202411?

@r12f Sure, backport PR created: #306

hi @jianyuewu , looks like the PR is created to master, might need some update.

jianyuewu · 2025-08-28T02:30:41Z

hi @jianyuewu , do you mind helping pick this change to 202411?

@r12f Sure, backport PR created: #306

hi @jianyuewu , looks like the PR is created to master, might need some update.
Hi @r12f,
Ah, so sorry, rebased to 202411. #306

r12f · 2025-09-22T17:49:54Z

Removing 202412 tag since the PR is already merged in 202411.

jianyuewu added 2 commits July 29, 2025 04:22

Add UT

eac82ee

Signed-off-by: Jianyue Wu <[email protected]>

jianyuewu marked this pull request as draft July 29, 2025 10:07

dgsudharsan added the Request for msft-202412 Branch label Jul 30, 2025

keboliu added the Request for 202505 Branch label Jul 30, 2025

jianyuewu marked this pull request as ready for review July 30, 2025 05:44

keboliu approved these changes Jul 30, 2025

View reviewed changes

keboliu requested a review from judyjoseph July 30, 2025 06:32

r12f requested a review from vvolam July 31, 2025 15:31

vvolam approved these changes Jul 31, 2025

View reviewed changes

judyjoseph reviewed Jul 31, 2025

View reviewed changes

scripts/determine-reboot-cause Show resolved Hide resolved

Update timestamp for reboot cause

5e569f7

Update 2nd timestamp in reboot cause, also use UTC. Mon Jul 28 10:15:33 AM UTC 2025 Signed-off-by: Jianyue Wu <[email protected]>

Add UT for UTC timestamp

dfefbf7

Signed-off-by: Jianyue Wu <[email protected]>

mssonicbld added the Cherry Pick Conflict_202505 label Aug 6, 2025

Update UT coverage

2ea559e

Signed-off-by: Jianyue Wu <[email protected]>

dgsudharsan requested a review from judyjoseph August 11, 2025 23:27

judyjoseph approved these changes Aug 25, 2025

View reviewed changes

judyjoseph merged commit 805ff5b into sonic-net:master Aug 25, 2025
5 checks passed

r12f added the Request for 202411 Branch label Aug 25, 2025

mssonicbld added the Cherry Pick Conflict_202411 label Aug 25, 2025

yejianquan added the Approved for 202505 Branch label Aug 28, 2025

yejianquan added Included in 202505 Branch and removed Cherry Pick Conflict_202505 labels Sep 4, 2025

r12f added Approved for 202411 Branch Included in 202411 Branch and removed Request for msft-202412 Branch Cherry Pick Conflict_202411 labels Sep 22, 2025

mssonicbld added the Cherry Pick Conflict_202411 label Sep 22, 2025

Conversation

jianyuewu commented Jul 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mssonicbld commented Jul 29, 2025

Uh oh!

azure-pipelines bot commented Jul 29, 2025

Uh oh!

r12f commented Jul 31, 2025

Uh oh!

vvolam left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jianyuewu commented Aug 6, 2025

Uh oh!

mssonicbld commented Aug 6, 2025

Uh oh!

azure-pipelines bot commented Aug 6, 2025

Uh oh!

mssonicbld commented Aug 6, 2025

Uh oh!

azure-pipelines bot commented Aug 6, 2025

Uh oh!

jianyuewu commented Aug 6, 2025

Uh oh!

jianyuewu commented Aug 6, 2025

Uh oh!

mssonicbld commented Aug 6, 2025

Uh oh!

azure-pipelines bot commented Aug 6, 2025

Uh oh!

mssonicbld commented Aug 6, 2025

Uh oh!

azure-pipelines bot commented Aug 6, 2025

Uh oh!

jianyuewu commented Aug 6, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jianyuewu commented Aug 19, 2025

Uh oh!

jianyuewu commented Aug 21, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

judyjoseph left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

kperumalbfn commented Aug 25, 2025

Uh oh!

yejianquan commented Aug 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

r12f commented Aug 26, 2025

Uh oh!

jianyuewu commented Aug 27, 2025

Uh oh!

yejianquan commented Aug 28, 2025

Uh oh!

r12f commented Aug 28, 2025

Uh oh!

jianyuewu commented Aug 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

r12f commented Sep 22, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

jianyuewu commented Jul 29, 2025 •

edited

Loading

jianyuewu commented Aug 6, 2025 •

edited

Loading

jianyuewu commented Aug 21, 2025 •

edited

Loading

yejianquan commented Aug 26, 2025 •

edited

Loading

jianyuewu commented Aug 28, 2025 •

edited

Loading