v3.01.00rc2

See https://github.com/MetPX/sarracenia/releases/tag/v3.01.00rc1 for release notes/changes.

v3.01.00rc2 only fixes the Debian packing to allow the package to be built on Ubuntu 18.04.

v3.01.00rc1

Summary

This release is the first of sr3 v3.01 because of a significant new feature: support for subscribing and/or publishing to multiple brokers in a single config. (Implement #339)

There are some API changes that are noted here: https://metpx.github.io/sarracenia/How2Guides/UPGRADING.html#id1

• NOTICE: can now subscribe to multiple brokers, and publish to multiple post_brokers with a single configuration.

• CHANGE: python API bindings being replaced by subscriptions. API is quite different, see examples. Subscriptions enables multi-queue and multi-broker support for subscribing.

• CHANGE: python API post_ settings being replaced by publishers. need to build sarracenia.config.publisher.Publisher structure now to publish messages. It enables publishing to multiple destinations.

• CHANGE: python API sarracenia.moth.default_options is replaced by sarracenia.moth.default_options() which produces a modifiable copy of the default settings, rather than the original options.

• Implement #1381: Add option to request a particular AMQP/RabbitMQ
  queue type
• MQTT Fixes
• systemd unit improvements (related issues: #1461, #1439, #1319, #1318)
• Fix #1439: co-ordinate sanity and start/stop, to avoid conflicts
• Fix #1436: reduce maximum exponential back-off time
• Fix #1447: more poll refactoring. This fixes a problem where post failures
  from scheduled polls would not be retried until the next scheduled
  appointment. Likely also fixes #1132, unacked messages in polls.
• Fix #1366, #1377 and #1395: change fileOp processing to retry some fileOps
  when they fail. Fixes race conditions with link renames and directory
  removals when the subscriber has multiple instances or runs in a cluster.
• Fix #1453: crash when trying to check the content type of a file that
  doesn't exist.
• Fix #1413: Azure transfer class rename problems
• Fix #1420: nodupe improvement: check nodupe expiry time when checking if
  file is a duplicate. Previously, expiry time was only checked during
  housekeeping which caused problems when nodupe_ttl was < housekeeping
  interval.

What's Changed

• doc update after going through pre-release process by @reidsunderland in #1404
• fix #1390 amend documentation to better describe mirror by @petersilva in #1405
• Add AddSMHeader for client adaptability by @andreleblanc11 in #1406
• AM - SM header treatment improvements based on client recommendations/comments by @andreleblanc11 in #1409
• 3.00.58 upgrade notes by @reidsunderland in #1410
• Fix Azure's renaming not working when directory isn't / by @andreleblanc11 in #1414
• enable persistence for mosquitto by @reidsunderland in #1415
• Email sender bug when file type is None + logging improvements by @andreleblanc11 in #1416
• fix #1420 precise time comparison in nodupe/disk engine by @petersilva in #1423
• fix post_exchange parsing, it had a weirdness. by @petersilva in #1426
• Misc. AM/bulletin renamer fixes - April 2025 by @andreleblanc11 in #1425
• Issue339 try6 implement multi-broker subscriptions by @petersilva in #1418
• implement configs_extant in sr3 status (argument for --dangerWillRobi… by @petersilva in #1435
• Lots of wiski improvements by @andreleblanc11 in #1429
• Issue339 part8 multiple publishers by @petersilva in #1428
• Bulletin renaming miscellaneous fixes - 2025/05 by @andreleblanc11 in #1433
• Issue1437 deep config copies by @petersilva in #1438
• Improve fileEvents documentation by @andreleblanc11 in #1441
• fix geometry for 0.0.8 by @petersilva in #1443
• Fix #1444 by @andreleblanc11 in #1446
• message baseUrl from poll_baseUrl if avail... dir also by @petersilva in #1448
• Issue1436 try2 random ebo maximum by @petersilva in #1445
• fix #1453 - crash in sender when local file not found by @reidsunderland in #1454
• retry failed fileOps by @reidsunderland in #1452
• More multi config unit tests by @petersilva in #1451
• Add JSON option + better convention for filenames for WISKI by @andreleblanc11 in #1455
• Fix issues with poll/scheduled flow and flakey broker by @reidsunderland in #1459
• co-ordinate start/stop with sanity, add new and inter by @petersilva in #1460
• Systemd user unit on debian by @petersilva in #1462
• Fix sr3 start/restart not ignoring disabled configs by @andreleblanc11 in #1464
• Dockerfile base fix by @reidsunderland in #1467
• systemd exitType=cgroup means it does not look dead by @petersilva in #1466
• Mqtt fixes mostly by @petersilva in #1465
• Add queueType logic for Quorum queues/streams by @andreleblanc11 in #1458

Full Changelog: v3.00.58...v3.01.00rc1

Packages Available

(coming soon) Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

PyPI: https://pypi.org/project/metpx-sr3/3.1.0rc1/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• Experimental self-extracting windows installer also attached. WSL is preferred if possible.

v3.00.58 - nodupe improvements and small fixes

Summary

Mostly bug fixes and functionality improvements (e.g. email sender supports attachments, S3 and Azure transfer class improvements).

What's New

• nodupe has been tweaked. The documentation has been improved thanks to @petersilva and there are now two new nodupe bases: name and path_only.
• Possible breakage: the old nodupe_basis name has been renamed to nodupe_basis name_only. Config files will need to be updated to match.
  • See https://metpx.github.io/sarracenia/Reference/sr3_options.7.html#nodupe-basis-data-name-path-default-path and https://metpx.github.io/sarracenia/Explanation/DuplicateSuppression.html
     

What's Changed

• Issue1369 - Add option to send a file as an attachment. by @andreleblanc11 in #1370
• Set the product name and version for AMQP connections issue #1371 by @reidsunderland in #1372
• fix #1373: cp is in /bin by @petersilva in #1375
• #1299 don't use relPath to rename file by @andreleblanc11 in #1378
• S3 Tweaks - anonymous access, create buckets. docs by @reidsunderland in #1380
• Fix #1385 by @andreleblanc11 in #1386
• Fix #1393 by @andreleblanc11 in #1394
• Fix race condition (#1395) by detecting remove events by @reidsunderland in #1397
• #884 Add new option to make safe https character quotes configurable by @andreleblanc11 in #1396
• Issue1382 2 add path_only, name_only, data_only. Improved documentation. by @petersilva in #1399
• add plugin to filter based on contentType by @reidsunderland in #1401
• Fix #1398. Try a close when EOFError is encountered. by @andreleblanc11 in #1403
• Azure transfer tweaks by @reidsunderland in #1400

Full Changelog: v3.00.57...v3.00.58

v3.00.58rc1

Summary

Mostly bug fixes and functionality improvements (e.g. email sender supports attachments, S3 and Azure transfer class improvements).

What's New

• nodupe has been tweaked. The documentation has been improved thanks to @petersilva and there are now two new nodupe bases: name and path_only.
• Possible breakage: the old nodupe_basis name has been renamed to nodupe_basis name_only. Config files will need to be updated to match.
  • See https://metpx.github.io/sarracenia/Reference/sr3_options.7.html#nodupe-basis-data-name-path-default-path and https://metpx.github.io/sarracenia/Explanation/DuplicateSuppression.html
     

What's Changed

• Issue1369 - Add option to send a file as an attachment. by @andreleblanc11 in #1370
• Set the product name and version for AMQP connections issue #1371 by @reidsunderland in #1372
• fix #1373: cp is in /bin by @petersilva in #1375
• #1299 don't use relPath to rename file by @andreleblanc11 in #1378
• S3 Tweaks - anonymous access, create buckets. docs by @reidsunderland in #1380
• Fix #1385 by @andreleblanc11 in #1386
• Fix #1393 by @andreleblanc11 in #1394
• Fix race condition (#1395) by detecting remove events by @reidsunderland in #1397
• #884 Add new option to make safe https character quotes configurable by @andreleblanc11 in #1396
• Issue1382 2 add path_only, name_only, data_only. Improved documentation. by @petersilva in #1399
• add plugin to filter based on contentType by @reidsunderland in #1401
• Fix #1398. Try a close when EOFError is encountered. by @andreleblanc11 in #1403
• Azure transfer tweaks by @reidsunderland in #1400

Full Changelog: v3.00.57...v3.00.58rc1

v3.00.57 - closer to fine

Summary

"The less I seek my source for some definitive, the closer I am to fine." -- Indigo Girls.

The last version (3.0.56) was the first one where we thought sr3 could replace any
v2 configuration. As we did so, a few new regressions were found in some senders.
This release addresses all of those. We are close to there being none left.

There are also improvements in this release:

• #1350 Add “down” and “disconnected” run states to sr3 status
• #1323 DESTFN=NONE::TIME now works.
• #1299 after_gather entry point added.
• #1289 accept statements now provide arguments to plugins
• change in post_exchangeSplit… now based on relPath, not checksum. (work on HPC.)
• fix #1241 sr3 status more compact.
• fix #1206 exclude retries from lag calculation.
• #1301 sr3 status display fractional message rates when (< 1 / second.)
• user mode systemd unit file now tested and working.
• The https://dd.weather.gc.ca web site was revised. Example subscriptions revised to match.
• #1346 warn that queue parameters should be earlier in configs now.
• #1358 date/time duration breakage by @petersilva in #1359

Regressions compared to v2 that are fixed:

• #1337 mirror per umask, not global (sender)
• #1340 rename not working. (sender)
• #1327 inflight NONE in sender causes messages to be retried. (sender)
• #1322 sr3 sender fails to post message when no identity field in message.
• #1351 fixing problem getting local checksums of data that has arbitrary identity method by @petersilva in #1362

HPC mirroring related fixes:

• fix #1363 hardlink support broken. by @petersilva in #1364
• fix #1366 ln -sf leaving tmp files lying around by @petersilva in #1367
• review and improve error handling when invoking makedirs by @petersilva in #1365
• #1297 part2 ordering mdelaylatest by @petersilva in #1368

What's Changed

• 3.00.54 maint by @petersilva in #1179
• got github notice about artifact@v3 being obsoleted by @petersilva in #1287
• Fix issue #1241 - make sr3 status display more compact by @mshak2 in #1290
• Add fileSizeMax to sender #1285 by @andreleblanc11 in #1286
• Check if there is anything to retry before figuring out if we by @petersilva in #1292
• systemd integration hints were v2 oriented by @petersilva in #1293
• Add method to recycle unused instance numbers #1288 by @andreleblanc11 in #1294
• Issue339 try5 fail connections in main loop by @petersilva in #1295
• issue #624 adding INSTANCE to variable substitution by @petersilva in #1296
• Issue634 design documentation review by @petersilva in #1303
• fix #1206 change send signature to support permanent failures by @petersilva in #1305
• Implement #1289 by @reidsunderland in #1306
• Fix #1307 - Decrease info-message length for sr3 cleanup by @mshak2 in #1313
• Fix #1301 by @mshak2 in #1312
• Fix #1309 by @andreleblanc11 in #1310
• exclude retries from lag accumulation by @petersilva in #1304
• Issue1299 - New after_gather entry point + better error handling in sundew_dirPattern by @andreleblanc11 in #1308
• user mode systemctl start should not setuid/setgid by @petersilva in #1317
• get DESTFN=NONE::TIME to work... I think. by @petersilva in #1323
• Fix #1321, allow any component's default_options overrides to work by @reidsunderland in #1326
• Issue1322 post exchange split revamp by @petersilva in #1329
• sender options inflight test fails if None by @petersilva in #1330
• Fix py3.6 crash by re-implementing #1289 by @reidsunderland in #1332
• fix partitions, broken because naturalsize output has no spaces anymore by @petersilva in #1333
• Fix #1325 by handling the exception by @reidsunderland in #1335
• Refactor email plugin config syntax by @reidsunderland in #1338
• Four in 1: Issue #1300 watch file perm, #1316 crash in download decision, #1254 poll fileAgeMin, #1337 mirror is not global by @petersilva in #1336
• fix #1283 disable new config by @petersilva in #1343
• user service cannot depend on multi-user (system) target by @petersilva in #1339
• Many small issues by @petersilva in #1342
• Fixing unit tests 202412 by @petersilva in #1345
• Issue1346 deal with change in semantics for queue declares from previous versions. by @petersilva in #1349
• changing topics for new citypage scheme by @petersilva in #1341
• Fix DiskQueue bugs by @reidsunderland in #1356
• sr3 status fixes #1350 and #1352. by @petersilva in #1355
• Issue1358 date/time duration breakage by @petersilva in #1359
• Add logic from gitlab version of plugins by @andreleblanc11 in #1361
• Issue1351 fixing problem getting local checksums of data that has arbitrary identity method by @petersilva in #1362
• fix #1363 hardlink support broken. by @petersilva in #1364
• fix #1366 ln -sf leaving tmp files lying around by @petersilva in #1367
• review and improve error handling when invoking makedirs by @petersilva in #1365
• Issue1297 part2 ordering mdelaylatest by @petersilva in #1368

Full Changelog: v3.00.56...v3.00.57

Packages Available

(coming soon) Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

PyPI: https://pypi.org/project/metpx-sr3/3.0.57/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• Experimental self-extracting windows installer also attached. WSL is preferred if possible.

v3.0.57rc2

What's Changed

• #1358 date/time duration breakage by @petersilva in #1359
• Add logic from gitlab version of plugins by @andreleblanc11 in #1361
• #1351 fixing problem getting local checksums of data that has arbitrary identity method by @petersilva in #1362
• fix #1363 hardlink support broken. by @petersilva in #1364
• fix #1366 ln -sf leaving tmp files lying around by @petersilva in #1367
• review and improve error handling when invoking makedirs by @petersilva in #1365
• #1297 part2 ordering mdelaylatest by @petersilva in #1368

Full Changelog: v3.00.57rc1...v3.0.57rc2

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.57rc2/

• (coming soon) RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (coming soon) self-extracting windows installer also attached.

v3.00.57rc1 - closer to fine

Summary

"The less I seek my source for some definitive, the closer I am to fine." -- Indigo Girls.

The last version (3.0.56) was the first one where we thought sr3 could replace any
v2 configuration. As we did so, a few new regressions were found in some senders.
This release addresses all of those. We are close to there being none left.

There are also improvements in this release:

• #1350 Add “down” and “disconnected” run states to sr3 status
• #1323 DESTFN=NONE::TIME now works.
• #1299 after_gather entry point added.
• #1289 accept statements now provide arguments to plugins
• change in post_exchangeSplit… now based on relPath, not checksum. (work on HPC.)
• fix #1241 sr3 status more compact.
• fix #1206 exclude retries from lag calculation.
• #1301 sr3 status display fractional message rates when (< 1 / second.)
• user mode systemd unit file now tested and working.
• The https://dd.weather.gc.ca web site was revised. Example subscriptions revised to match.
• #1346 warn that queue parameters should be earlier in configs now.

Sender regressions compared to v2 that are fixed:

• #1337 mirror per umask, not global
• #1340 rename not working.
• #1327 inflight NONE in sender causes messages to be retried.
• #1322 sr3 sender fails to post message when no identity field in message.

As well as many other bug fixes.

Full Changelog: v3.00.56...v3.00.57rc1

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.57rc1/

• (coming soon) RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (coming soon) self-extracting windows installer also attached.

v3.00.56 Recursion is Back Baby, Subtopics After Queues

What's new

• CHANGE: queueName and/or queueShare must be above subtopic in configuration files. (order did not matter in earlier versions)
• CHANGE: in API breakage: replace sarracenia.credentials with sarracenia.config.credentials
• New: standby mode, triggered by attempts 0 in the configuration file, allows clean local queueing of transfers during long outages.
• New: recursion option for polls and watches (actually it was removed before, but added back.)
• New: logDuplicates option to show only files rejected by the duplicate suppression mechanisms.
• rest is fixes of many things. It just behaves better.
  • http download performance regression fixed.
  • ack failure recovery regression fixed.
  • HPC mirroring rename/symlink fix (requires metpx-sr3c 3.24.11 )

What's Changed

• Issue1199 - directory / not interpreted properly. by @petersilva in #1231
• just reducing bogus messages by @petersilva in #1230
• Unquote secret_access_key in s3CloudSender plugin by @reidsunderland in #1228
• sr3 sanity behaves badly with instances > 100. by @petersilva in #1226
• ANOTHER AM U18.04 adaptation + another AM sender problem found in satnet testings by @andreleblanc11 in #1221
• fix crash on sr3c pidfiles (i01.pid) by @reidsunderland in #1234
• Issue35 try 2 stateful binding part1 by @petersilva in #1232
• Switch from global QoS to per-consumer QoS by @reidsunderland in #1235
• Doc sec zoning multihop by @petersilva in #1236
• Fix bug from moving credentials into config subdir by @reidsunderland in #1239
• fix #1242 sftp making subdirs when it shouldn't by @reidsunderland in #1243
• Fix for #1156 by @mshak2 in #1193
• QoS should apply to all instances by @reidsunderland in #1245
• add ncftp client to basic vm config... might fix some flow tests. by @petersilva in #1249
• Json logging (EXPERIMENTAL) by @petersilva in #1238
• Add recursion back by @petersilva in #1237
• fix #1246 delete source when no options given by @petersilva in #1250
• Fixing unit tests by @petersilva in #1251
• testing: add pyftpdlib to apt packages to try avoiding version incompatibility by @reidsunderland in #1252
• suppress error message about missing subscriptions.json by @petersilva in #1256
• Update paho-mqtt package from pip by @reidsunderland in #1260
• Improve Redis Flow Tests on GitHub by @reidsunderland in #1259
• Refactor the interruptible_sleep test by @reidsunderland in #1257
• more information about doing post releases by @petersilva in #1253
• tweaks to flowcb/gather/file.py after experiments with noticing large files. by @petersilva in #1258
• Add cpuS and wVip to status list - CommandLineGuide.rst by @andreleblanc11 in #1263
• Fix HTTP performance regression by @reidsunderland in #1265
• fix date check in messages. by @petersilva in #1264
• fix #1255 remove error message with undeclared value in it by @petersilva in #1268
• Issue1271 - Change raw2bulletin renamer to an after_accept entry point by @andreleblanc11 in #1273
• Issue1270 - fixing relative symlinks with (inappropriate) ../../ in them... by @petersilva in #1272
• Issue1267 queueName not working as expected because of upcoming feature by @petersilva in #1276
• close #1278 we learned that acks should never be retried by @petersilva in #1281
• Fix #1266 - Add a check for any disabled configs when using sr3 start by @mshak2 in #1282
• Issue1271 - Fix accomodation to sarra component by @andreleblanc11 in #1279
• for #1275 standby mode can be implemented with attempts=0 by @petersilva in #1280

Full Changelog: v3.00.55...v3.00.56

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx

PyPI: https://pypi.org/project/metpx-sr3/3.0.56/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (coming soon) self-extracting windows installer also attached.

v3.00.56rc3 back to recursion, subtopics after queues

What's Changed

• close #1278 we learned that acks should never be retried by @petersilva in #1281
• Fix #1266 - Add a check for any disabled configs when using sr3 start by @mshak2 in #1282
• Issue1271 - Fix accomodation to sarra component by @andreleblanc11 in #1279
• for #1275 standby mode can be implemented with attempts=0 by @petersilva in #1280

Full Changelog: v3.00.56rc2...v3.00.56rc3

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.56rc3/

• RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• self-extracting windows installer also attached.

v3.00.56rc2 back to recursion, subtopics after queues

Regression

• known stability regression, when ack fails subscriber must be restarted #1278 (work-around: use v3.00.54.post1 )
  (stable version will not be released without a fix.)

What's New

• need to put queueName settings before the subtopic settings in configuration files, for this and all future versions.
• fixed #1261 performance regression for https download
• HPC rename/symlink mirroring fix: needs to be combined with an upcoming C (metpx-sr3c) package (relative symbolic links and renames when done from a directory that is symlinked to the tree being mirrored.)

What's Changed

• suppress error message about missing subscriptions.json by @petersilva in #1256
• Update paho-mqtt package from pip by @reidsunderland in #1260
• Improve Redis Flow Tests on GitHub by @reidsunderland in #1259
• Refactor the interruptible_sleep test by @reidsunderland in #1257
• more information about doing post releases by @petersilva in #1253
• tweaks to flowcb/gather/file.py after experiments with noticing large files. by @petersilva in #1258
• Add cpuS and wVip to status list - CommandLineGuide.rst by @andreleblanc11 in #1263
• Fix HTTP performance regression by @reidsunderland in #1265
• fix date check in messages. by @petersilva in #1264
• fix #1255 remove error message with undeclared value in it by @petersilva in #1268
• Issue1271 - Change raw2bulletin renamer to an after_accept entry point by @andreleblanc11 in #1273
• Issue1270 - fixing relative symlinks with (inappropriate) ../../ in them... by @petersilva in #1272
• Issue1267 queueName not working as expected because of upcoming feature by @petersilva in #1276

Full Changelog: v3.00.56rc1...v3.00.56rc2

Packages Available

Ubuntu PPA: https://launchpad.net/~ssc-hpc-chp-spc/+archive/ubuntu/metpx-pre-release

PyPI: https://pypi.org/project/metpx-sr3/3.0.56rc2/

• (coming soon) RPM package for Redhat Version 8 (rh8) and 9 (rh9) attached here, install with rpm -ivh ... note:

  • Some dependencies are not readily available. May have to obtain packages
    from pip to get a fully functional installation. Verify with sr3 features after installation, to determine if everything needed is available.

• (coming soon) self-extracting windows installer also attached.