ros2-testing repository inconsistent references #3

tfoote · 2021-06-09T17:59:38Z

I have checked ROS Answers for similar reports and solutions.
I have checked the ROS Status page and my issue is not listed there.
I have checked the open issues for similar reports.
This is not a security issue If you have a security issue to report please do so via email to [email protected].

Problem description

Report from a user:

Seems like the Packages file for ros2-testing is not up to date:

I.e. ros-foxy-ros-workspace should have version
1.0.1-1focal.20210422.234610 but only 1.0.1-1focal.20210422.235134 is
there:

http://packages.ros.org/ros2-testing/ubuntu/pool/main/r/ros-foxy-ros-workspace/

or in one line ;):

$ wget "http://packages.ros.org/ros2-testing/ubuntu/$(curl "http://packages.ros.org/ros2-testing/ubuntu/dists/focal/main/binary-amd64/Packages" | grep-dctrl -F Package -s Filename ros-foxy-ros-workspace | cut -d" " -f2)"
2021-06-09 16:28:28 ERROR 404: Not Found.

This is currently failing industrial_ci:

https://github.com/vatanaksoytezer/warehouse_ros_mongo/runs/2783974132?check_suite_focus=true

tfoote · 2021-06-09T18:00:24Z

I can't reproduce the error. I suspect that our mirror replication has failed or you got the unfortunate timing of hitting it during a sync. There were three syncs onging near that time.

https://build.ros2.org/job/Grel_sync-packages-to-testing_focal_amd64/lastSuccessfulBuild/
https://build.ros2.org/job/Grel_sync-packages-to-testing_focal_arm64/lastSuccessfulBuild/
https://build.ros2.org/job/Frel_sync-packages-to-testing_focal_amd64/lastSuccessfulBuild/

I can't reproduce the error now. And I checked all three mirrors directly.

tfoote@snowman5:~ Last: [0] (0s Seconds)
$ wget "http://packages.ros.org/ros2-testing/ubuntu/$(curl "http://packages.ros.org/ros2-testing/ubuntu/dists/focal/main/binary-amd64/Packages" | grep-dctrl -F Package -s Filename ros-foxy-ros-workspace | cut -d" " -f2)"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2634k  100 2634k    0     0  3541k      0 --:--:-- --:--:-- --:--:-- 3546k
--2021-06-09 10:54:32--  http://packages.ros.org/ros2-testing/ubuntu/pool/main/r/ros-foxy-ros-workspace/ros-foxy-ros-workspace_1.0.1-1focal.20210422.234610_amd64.deb
Resolving packages.ros.org (packages.ros.org)... 64.50.236.52, 64.50.233.100, 140.211.166.134, ...
Connecting to packages.ros.org (packages.ros.org)|64.50.236.52|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 15968 (16K)
Saving to: ‘ros-foxy-ros-workspace_1.0.1-1focal.20210422.234610_amd64.deb’

ros-foxy-ros-worksp 100%[===================>]  15.59K  --.-KB/s    in 0.07s   

2021-06-09 10:54:32 (236 KB/s) - ‘ros-foxy-ros-workspace_1.0.1-1focal.20210422.234610_amd64.deb’ saved [15968/15968]

tfoote@snowman5:~ Last: [0] (1s Seconds)
$ wget "https://ftp-osl.osuosl.org/pub/ros2-testing/$(curl "http://packages.ros.org/ros2-testing/ubuntu/dists/focal/main/binary-amd64/Packages" | grep-dctrl -F Package -s Filename ros-foxy-ros-workspace | cut -d" " -f2)"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2634k  100 2634k    0     0  3314k      0 --:--:-- --:--:-- --:--:-- 3310k
--2021-06-09 10:55:37--  https://ftp-osl.osuosl.org/pub/ros2-testing/pool/main/r/ros-foxy-ros-workspace/ros-foxy-ros-workspace_1.0.1-1focal.20210422.234610_amd64.deb
Resolving ftp-osl.osuosl.org (ftp-osl.osuosl.org)... 140.211.166.134, 2605:bc80:3010::134
Connecting to ftp-osl.osuosl.org (ftp-osl.osuosl.org)|140.211.166.134|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 15968 (16K)
Saving to: ‘ros-foxy-ros-workspace_1.0.1-1focal.20210422.234610_amd64.deb.1’

ros-foxy-ros-workspace_1.0.1-1foca 100%[================================================================>]  15.59K  --.-KB/s    in 0.03s   

2021-06-09 10:55:37 (529 KB/s) - ‘ros-foxy-ros-workspace_1.0.1-1focal.20210422.234610_amd64.deb.1’ saved [15968/15968]

tfoote@snowman5:~ Last: [0] (1s Seconds)
$ wget "https://ftp-chi.osuosl.org/pub/ros2-testing/$(curl "http://packages.ros.org/ros2-testing/ubuntu/dists/focal/main/binary-amd64/Packages" | grep-dctrl -F Package -s Filename ros-foxy-ros-workspace | cut -d" " -f2)"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2634k  100 2634k    0     0  3293k      0 --:--:-- --:--:-- --:--:-- 3289k
--2021-06-09 10:55:48--  https://ftp-chi.osuosl.org/pub/ros2-testing/pool/main/r/ros-foxy-ros-workspace/ros-foxy-ros-workspace_1.0.1-1focal.20210422.234610_amd64.deb
Resolving ftp-chi.osuosl.org (ftp-chi.osuosl.org)... 64.50.236.52, 2600:3402:200:227::2
Connecting to ftp-chi.osuosl.org (ftp-chi.osuosl.org)|64.50.236.52|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 15968 (16K)
Saving to: ‘ros-foxy-ros-workspace_1.0.1-1focal.20210422.234610_amd64.deb.2’

ros-foxy-ros-workspace_1.0.1-1foca 100%[================================================================>]  15.59K  --.-KB/s    in 0.07s   

2021-06-09 10:55:49 (239 KB/s) - ‘ros-foxy-ros-workspace_1.0.1-1focal.20210422.234610_amd64.deb.2’ saved [15968/15968]

tfoote@snowman5:~ Last: [0] (1s Seconds)
$ wget "https://ftp-nyc.osuosl.org/pub/ros2-testing/$(curl "http://packages.ros.org/ros2-testing/ubuntu/dists/focal/main/binary-amd64/Packages" | grep-dctrl -F Package -s Filename ros-foxy-ros-workspace | cut -d" " -f2)"
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2634k  100 2634k    0     0  3619k      0 --:--:-- --:--:-- --:--:-- 3614k
--2021-06-09 10:55:56--  https://ftp-nyc.osuosl.org/pub/ros2-testing/pool/main/r/ros-foxy-ros-workspace/ros-foxy-ros-workspace_1.0.1-1focal.20210422.234610_amd64.deb
Resolving ftp-nyc.osuosl.org (ftp-nyc.osuosl.org)... 64.50.233.100, 2600:3404:200:237::2
Connecting to ftp-nyc.osuosl.org (ftp-nyc.osuosl.org)|64.50.233.100|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 15968 (16K)
Saving to: ‘ros-foxy-ros-workspace_1.0.1-1focal.20210422.234610_amd64.deb.3’

ros-foxy-ros-workspace_1.0.1-1foca 100%[================================================================>]  15.59K  --.-KB/s    in 0.08s   

2021-06-09 10:55:57 (188 KB/s) - ‘ros-foxy-ros-workspace_1.0.1-1focal.20210422.234610_amd64.deb.3’ saved [15968/15968]

vatanaksoytezer · 2021-06-09T18:19:13Z

@tfoote Thanks a lot for looking into this. I cannot reproduce this right now either, and all the runs are turning back to succeeding, though this was persistent for at least 7-8 hours in recurring runs on multiple repositories in ros-planning and ros-control organizations. We were probably in an unfortunate timing during multiple syncs.

tfoote · 2021-06-09T18:36:56Z

If there was an issue for 7-8 hours prior that sounds like the last syncs resolved the inconsistency in the database. As opposed to causing them. The sync outage is basically as fast as we can copy the files into the repository.

clalancette · 2021-06-09T20:08:47Z

For what it is worth, we did have a problem where the sync-to-testing jobs didn't run properly yesterday (most likely because of the Fastly worldwide outage). Today I manually kicked https://build.ros2.org/job/Frel_sync-packages-to-testing_focal_amd64/ , so it is possible that that resolved it.

jspricke · 2021-06-09T20:42:59Z

Thanks a lot for resolving this, will try to remember to come here instead of writing Tully next time (old habits... :) ).
Btw. Debian has some tools to make sure such errors do not happen when syncing: https://www.debian.org/mirror/ftpmirror
May make sense to use them.

clalancette · 2021-06-14T20:04:11Z

A somewhat similar thing happened to Melodic over the weekend; see https://discourse.ros.org/t/testing-repository-empty-amd64/20908 for some more of the details.

I'm wondering if it would make sense to have the sync-to-testing jobs run nightly, as a backup in case something failed. That way the time frame of the breakage (if it does occur) would at least be time-limited.

tfoote · 2021-06-17T07:19:24Z

In the failed melodic log noticed that there's actually a real error that we should not be having

13:57:15 gpgme gave error GPGME:54:  Unusable secret key

We should make sure that doesn't happen.

tfoote · 2021-08-19T16:28:41Z

I've refiled a tracking ticket on the upstream repository to track it there.

This was referenced Jun 9, 2021

Add Galactic CI moveit/warehouse_ros#75

Merged

Add Galactic CI, cleanup rolling moveit/geometric_shapes#190

Merged

tfoote mentioned this issue Aug 19, 2021

Sync gpgme error: Unusable secret key ros-infrastructure/ros_buildfarm#902

Open

tfoote closed this as completed Aug 19, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ros2-testing repository inconsistent references #3

ros2-testing repository inconsistent references #3

tfoote commented Jun 9, 2021

tfoote commented Jun 9, 2021 •

edited

Loading

vatanaksoytezer commented Jun 9, 2021

tfoote commented Jun 9, 2021

clalancette commented Jun 9, 2021

jspricke commented Jun 9, 2021

clalancette commented Jun 14, 2021

tfoote commented Jun 17, 2021

tfoote commented Aug 19, 2021

ros2-testing repository inconsistent references #3

ros2-testing repository inconsistent references #3

Comments

tfoote commented Jun 9, 2021

Problem description

tfoote commented Jun 9, 2021 • edited Loading

vatanaksoytezer commented Jun 9, 2021

tfoote commented Jun 9, 2021

clalancette commented Jun 9, 2021

jspricke commented Jun 9, 2021

clalancette commented Jun 14, 2021

tfoote commented Jun 17, 2021

tfoote commented Aug 19, 2021

tfoote commented Jun 9, 2021 •

edited

Loading