After upgrade some aptly operations became extremely slow (hours vs minutes), suspect ext4 discard on QEMU

[volo-05]

Hi! After upgrading aptly we see a huge slowdown on a clone machine with (as far as we can tell) the same dataset and workload.

What changed

• Old aptly: 1.4.0+ds1-4+b4
• New aptly: 1.6.2

Symptoms (CI timings)

Before:

• Deploy pool and staging: ~5 min
• Deploy releases: ~5 min

After upgrade:

• Deploy pool and staging: ~1h 25m
• Deploy releases: >5h

How we run aptly

We run aptly api serve and our CI connects via a UNIX socket.
rootDir is on ext4 inside a QEMU/KVM VM.

Filesystem / storage

We noticed the filesystem is mounted with discard:
/dev/sda1 ... ext4 (rw,relatime,discard,...)

The virtual disk supports TRIM/UNMAP:

• lsblk -D shows DISC-GRAN=4K, DISC-MAX=1G, ROTA=0 (QEMU HARDDISK)
• /sys/block/sda/queue/discard_max_bytes = 1073741824
• /sys/block/sda/queue/discard_granularity = 4096

TRIM / discard test

With nodiscard, fstrim -av trims a lot of data:

• /: 26.2 GiB trimmed, real time ~12s

With discard, fstrim -av reports:

• /: 0 B trimmed, real time ~0.02s

This suggests continuous discard is active.

Question / request for guidance

1. Is discard known to hurt performance for LevelDB-heavy workloads used by aptly (compactions, table file churn, fsync)?
2. Are there recommended filesystem mount options for aptly databases?
   For example, should we avoid discard and use periodic fstrim instead?
3. Any aptly-side debug flags / logging / profiling you recommend to pinpoint why these steps became so slow after 1.6.2?

If you tell us what logs/metrics you need (aptly debug logs, strace snippet, LevelDB stats, etc.), we can provide them.

Thanks!

aptly config (redacted)

{
  "FileSystemPublishEndpoints": {},
  "S3PublishEndpoints": {
    "deb.wirenboard.com": {
      "acl": "public-read",
      "awsAccessKeyID": "<REDACTED>",
      "awsSecretAccessKey": "<REDACTED>",
      "bucket": "deb.wirenboard.com",
      "plusWorkaround": true,
      "region": "eu-west-1"
    }
  },
  "SwiftPublishEndpoints": {},
  "architectures": [],
  "dependencyFollowAllVariants": false,
  "dependencyFollowRecommends": false,
  "dependencyFollowSource": false,
  "dependencyFollowSuggests": false,
  "downloadConcurrency": 4,
  "downloadSourcePackages": false,
  "downloadSpeedLimit": 0,
  "gpgDisableSign": false,
  "gpgDisableVerify": false,
  "gpgProvider": "internal",
  "ppaCodename": "",
  "ppaDistributorID": "ubuntu",
  "rootDir": "/home/jenkins/aptly-ng"
}

{
"rootDir": "/home/jenkins/aptly-ng",
"downloadConcurrency": 4,
"downloadSpeedLimit": 0,
"architectures": [],
"dependencyFollowSuggests": false,
"dependencyFollowRecommends": false,
"dependencyFollowAllVariants": false,
"dependencyFollowSource": false,
"dependencyVerboseResolve": false,
"gpgDisableSign": false,
"gpgDisableVerify": false,
"gpgProvider": "gpg",
"downloadSourcePackages": false,
"skipLegacyPool": true,
"ppaDistributorID": "ubuntu",
"ppaCodename": "",
"skipContentsPublishing": false,
"FileSystemPublishEndpoints": {},
"S3PublishEndpoints": {},
"SwiftPublishEndpoints": {}
}

[volo-05]

Additional evidence from CI logs (same dataset, clone VM):

Even when snapshots are already published, publish checks became extremely slow after upgrade.

aptly 1.6.2:

• Publishing all/pool.latest: 09:59:56 -> “already published”: 10:05:37 (~5m41s)
• Publishing all/staging.latest: 10:05:42 -> 10:11:36 (~5m54s)
• Each following experimental publish also takes ~5m30s–5m40s before reporting “already published”.

aptly 1.4.0:

• Publishing all/pool.latest: 19:55:45 -> 19:55:48 (~2.6s)
• Publishing all/staging.latest: 19:55:48 -> 19:55:50 (~1.9s)
• Others are also ~1–2 seconds.

Also, our client debug logs show that repeated GET /api/snapshots/... requests that used to complete in <1s total now take ~20–30s total (many requests ~0.7–1.6s each), suggesting a general slowdown of API read/metadata operations (likely DB/FS I/O related).