`opam update`: load only changed opam files #6614

arozovyk · 2025-07-22T12:09:29Z

This PR leverages the fact that we already compute a diff/patch during update to determine exactly which files have changed and it what way. Instead of reloading the entire repository we:

Propagate Patch.operation info from OpamFilename.patch
Add incremental loading via OpamRepositoryState.load_opams_incremental
Maintain existing opam files that haven't changed, avoiding unnecessary I/O

On Windows the OpamRepositoryState.load_opams_from_dir in OpamUpdate.repository currently takes ~10s. This PR brings it down to ~0.01s (the time it takes to load the opam files of a usual repository change).
On unix, it is a bit less dramatic, but still going from ~3.5s to 1ms

closes #5824

src/repository/opamRepositoryBackend.ml

src/state/opamRepositoryState.ml

src/repository/opamRepositoryBackend.ml

src/state/opamRepositoryState.ml

arozovyk · 2025-07-30T13:15:56Z

Updated simplfying a bunch of things.
We can't avoid parsing for diffs in case of Http and local since we have to first preprocess it, so I ended up reusing the result of Patch.parse from OpamSystem.patch in general for all backends.

kit-ty-kate · 2025-07-30T16:24:41Z

We can't avoid parsing for diffs in case of Http and local since we have to first preprocess it

I'm not sure to follow, pre-processing isn't done for http/local (per apply_repo_update)

arozovyk · 2025-07-30T19:58:09Z

pre-processing isn't done for http/local

You're right, my bad. So we can reuse the diffs avoiding Patch.parse for http/local see f0a731c. Not sure it's worth it though, we propagte the full diffs (we need them up until applying in OpamSystem.internal_patch) that, as you said, can be quite large. So memory overhead for to-be-determined speed gain?

kit-ty-kate

First pass of review. I haven't looked at everything yet but i think that's enough for now

src/repository/opamRepositoryBackend.ml

src/client/opamAction.ml

src/repository/opamRepositoryBackend.ml

src/core/opamSystem.ml

src/repository/opamRepository.ml

src/core/opamFilename.mli

src/state/opamRepositoryState.ml

kit-ty-kate · 2025-08-25T09:01:01Z

src/state/opamRepositoryState.ml

+    | Some opams -> opams
+    | None -> OpamPackage.Map.empty
+  in
+  let process_file acc file ~is_removal =


I'm not sure if it's done elsewhere or something, but this function seems to assume every files that can change are opam files. What about changes from their files directory or other non-opam file related changes?

The function purpose is to load opam files to update their definitions in rt, all other files are ignored as it was already the case with original load_opams_from_dir function. Any other extra files are read by OpamFileTools.read_repo_opam and stored through metadata

kit-ty-kate · 2025-08-25T09:03:00Z

tests/reftests/repository-update.test

@@ -0,0 +1,181 @@
+N0REP0


These tests should be added all at once in the first commit of the PR, ran to unsure the initial state is correct, then ran again for each new commits to show new behaviours

we already have a reftest called update.test i think it should probably go there instead

tests/lib/patchDiff.ml

src/repository/opamRepositoryBackend.ml

src/repository/opamRepositoryBackend.mli

kit-ty-kate · 2025-09-02T14:56:59Z

src/repository/opamRepository.ml

    OpamConsole.msg "[%s] synchronised from %s\n"
      (OpamConsole.colorise `green
         (OpamRepositoryName.to_string repo.repo_name))
      (OpamUrl.to_string repo.repo_url);
    log "%a: applying patch update at %a"
      (slog OpamRepositoryName.to_string) repo.repo_name
      (slog OpamFilename.to_string) f;
-    let preprocess =
-      match repo.repo_url.OpamUrl.backend with
-      | `http | `rsync -> false


i don't think we should start running the the preprocessing for local and http repositories. Could you bring the ?preprocess argument back?

Maybe this PR depends on the completion of #6599 first

We are not, this is naturally done by the fact that OpamSystem.parse_patch_file is only called in OpamVCS (in OpamAction.prepare_package_build as well, but that's a different story), for local and http repositories we reuse the diffs directly.

src/core/opamFilename.mli

kit-ty-kate · 2025-09-02T15:10:09Z

tests/reftests/repository-update.test

@@ -0,0 +1,181 @@
+N0REP0


we already have a reftest called update.test i think it should probably go there instead

tests/reftests/repository-update.test

kit-ty-kate · 2025-09-02T15:27:23Z

src/state/opamUpdate.ml

@@ -67,7 +67,7 @@ let repository rt repo =
    in
    OpamProcess.Job.with_text text @@
    OpamRepository.update r repo_root @@+ fun has_changes ->
-    let has_changes = if redirect then `Changes else has_changes in
+    let has_changes = if redirect then `Changes [] else has_changes in


That seems unnecessary. @rjbou do you know why this line was added in #5146 ?

kit-ty-kate · 2025-09-02T15:36:30Z

tests/reftests/repository-update.test

+### <REPO/packages/foo/foo.1/opam>
+opam-version: "2.0"
+synopsis: "Updated package"
+### opam update default


it would be nice if the new incremental behaviour was shown in the test. We could select the right OPAMDEBUGSECTION to show that only a limited number of files are being read

arozovyk · 2025-09-05T15:50:08Z

src/core/opamSystem.ml

  let internal_patch_error fmt =
    Printf.ksprintf (fun str -> raise (Internal_patch_error str)) fmt
  in
+  let patch_info_path =
+    OpamStd.Option.default ("in directory"^dir) patch_filename


Suggested change

OpamStd.Option.default ("in directory"^dir) patch_filename

OpamStd.Option.default ("in directory "^dir) patch_filename

arozovyk · 2025-09-05T16:02:37Z

master_changes.md

To be updated on rebase

kit-ty-kate

lgtm overall, however it's a hard PR to review without its final clean commit structure so i'd rather wait for whenever this is finished.

I also think this PR should be queued on a PR adding a benchmark test for opam update as we currently don't have any. It would ideal to have a test for three kinds of cases:

no diff
small diff (just a few files)
large diff (e.g. from ocaml/opam-repository#2025-01-before-archiving-phase1 to now)

with both local path and local git repositories

kit-ty-kate · 2025-09-05T17:10:10Z

tests/lib/patchDiff.ml

@@ -382,15 +382,14 @@ let diff_patch dir setup =
    let apply ~git diff =
      let git = if git then "GIT " else "" in
      let result =
-        OpamFilename.patch ~allow_unclean:false diff
-          (dir / first)
+        OpamFilename.patch ~allow_unclean:false(`Patch_file diff) (dir / first)


> 80 characters + minor missing space

Suggested change

OpamFilename.patch ~allow_unclean:false(`Patch_file diff) (dir / first)

OpamFilename.patch ~allow_unclean:false (`Patch_file diff)

(dir / first)

kit-ty-kate · 2025-09-05T17:18:36Z

src/repository/opamVCS.ml

+      | Some patch_file ->
+        let p = OpamFilename.to_string patch_file in
+        try
+          let diffs = OpamSystem.parse_patch_file
+              ~dir:(OpamFilename.Dir.to_string repo_root) p in
+          OpamRepositoryBackend.Update_patch (patch_file, diffs)
+        with exn -> Update_err exn


If we had OpamFilename.parse_patch_file for a more OpamFilename friendly API we could simplify to:

Suggested change

| Some patch_file ->

let p = OpamFilename.to_string patch_file in

try

let diffs = OpamSystem.parse_patch_file

~dir:(OpamFilename.Dir.to_string repo_root) p in

OpamRepositoryBackend.Update_patch (patch_file, diffs)

with exn -> Update_err exn

| Some patch_file ->

match OpamFilename.parse_patch_file ~dir:repo_root patch_file with

| diffs -> OpamRepositoryBackend.Update_patch (patch_file, diffs)

| exception exn -> Update_err exn

kit-ty-kate · 2025-09-05T17:19:25Z

src/repository/opamRepositoryBackend.ml

+    | Patch.Create f -> Patch.Create (rm_prefix f)
+    | Patch.Delete f -> Patch.Delete (rm_prefix f)
+    | Patch.Edit (f1, f2) -> Patch.Edit (rm_prefix f1, rm_prefix f2)
+    | Patch.Git_ext (f1, f2, ext) -> Patch.Git_ext (rm_prefix f1, rm_prefix f2, ext)


> 80 columns

Suggested change

| Patch.Git_ext (f1, f2, ext) -> Patch.Git_ext (rm_prefix f1, rm_prefix f2, ext)

| Patch.Git_ext (f1, f2, ext) ->

Patch.Git_ext (rm_prefix f1, rm_prefix f2, ext)

kit-ty-kate · 2025-09-05T22:54:43Z

tests/reftests/update.test

 Now run 'opam upgrade' to apply any package updates.
+### <REPO/repo>


Suggested change

### <REPO/repo>

### : Test incrementality

### rm -rf REPO

### <REPO/repo>

kit-ty-kate added this to the 2.5.0~alpha1 milestone Jul 22, 2025

kit-ty-kate reviewed Jul 22, 2025

View reviewed changes

src/repository/opamRepositoryBackend.ml Outdated Show resolved Hide resolved

kit-ty-kate reviewed Jul 22, 2025

View reviewed changes

src/state/opamRepositoryState.ml Outdated Show resolved Hide resolved

kit-ty-kate reviewed Jul 22, 2025

View reviewed changes

src/state/opamRepositoryState.ml Outdated Show resolved Hide resolved

arozovyk force-pushed the update_read_opams_incremental branch 3 times, most recently from c0d8002 to ba9b12a Compare July 23, 2025 15:04

kit-ty-kate reviewed Jul 24, 2025

View reviewed changes

src/repository/opamRepositoryBackend.ml Show resolved Hide resolved

kit-ty-kate reviewed Jul 24, 2025

View reviewed changes

src/state/opamRepositoryState.ml Outdated Show resolved Hide resolved

src/state/opamRepositoryState.ml Outdated Show resolved Hide resolved

src/state/opamRepositoryState.ml Outdated Show resolved Hide resolved

src/state/opamRepositoryState.ml Outdated Show resolved Hide resolved

arozovyk force-pushed the update_read_opams_incremental branch 2 times, most recently from 2cb1720 to 3136926 Compare July 30, 2025 13:12

kit-ty-kate mentioned this pull request Jul 31, 2025

[WIP] Improve performance of opam update/init by changing the structure of the internal http opam repositories (use the tar.gz as-is) #6625

Draft

rjbou requested review from kit-ty-kate and rjbou August 19, 2025 12:44

kit-ty-kate reviewed Aug 25, 2025

View reviewed changes

kit-ty-kate added PRIORITY AREA: PERFORMANCE labels Aug 27, 2025

arozovyk force-pushed the update_read_opams_incremental branch from f0a731c to 0cfb4a3 Compare September 2, 2025 13:20

arozovyk added 5 commits September 2, 2025 15:22

Propagate the diffs from Patch.parse in OpamSystem.patch

ee51123

Load opam files incrementally using Patch.t diffs

4ed3ddd

Opam update incremental git repo reftest

5f327d9

Reuse patch diffs from repository backend

6f7925a

Use result instead of a new variant

0e20947

arozovyk force-pushed the update_read_opams_incremental branch from 0cfb4a3 to 6b19d06 Compare September 2, 2025 13:23

arozovyk added 3 commits September 2, 2025 15:25

Parse patch file in OpamVCS

35c43fb

Use %a instead of %s

5b8a46b

Avoid open

42b530d

arozovyk added 4 commits September 2, 2025 15:25

Assert diffs = []

8ef78ed

Fold left

fa99d37

Format

9b4d3f9

Patch files are not directories

74dbeb6

arozovyk force-pushed the update_read_opams_incremental branch from 6b19d06 to 74dbeb6 Compare September 2, 2025 13:25

arozovyk requested a review from kit-ty-kate September 2, 2025 13:26

kit-ty-kate reviewed Sep 2, 2025

View reviewed changes

kit-ty-kate mentioned this pull request Sep 3, 2025

enrich lock file to contain info about original opam files #6668

Open

arozovyk added 7 commits September 4, 2025 14:50

Refactor patch in OpamSystem and OpamFilename

3567e8a

Fix patchDiff test

a0c5ce6

Log

964dca2

OpamRepositoryBackend doc

e29e17f

Update repository-update reftest with incremental behaviour

c656c89

Update reftests

c02418c

Move repository-update.test to update.test

2eefceb

arozovyk commented Sep 5, 2025

View reviewed changes

master_changes.md

Copy link

Collaborator Author

arozovyk Sep 5, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be updated on rebase

arozovyk requested a review from kit-ty-kate September 5, 2025 16:23

kit-ty-kate reviewed Sep 5, 2025

View reviewed changes

	OpamStd.Option.default ("in directory"^dir) patch_filename
	OpamStd.Option.default ("in directory "^dir) patch_filename

	OpamFilename.patch ~allow_unclean:false(`Patch_file diff) (dir / first)
	OpamFilename.patch ~allow_unclean:false (`Patch_file diff)
	(dir / first)

	\| Patch.Git_ext (f1, f2, ext) -> Patch.Git_ext (rm_prefix f1, rm_prefix f2, ext)
	\| Patch.Git_ext (f1, f2, ext) ->
	Patch.Git_ext (rm_prefix f1, rm_prefix f2, ext)

		Now run 'opam upgrade' to apply any package updates.
		### <REPO/repo>

opam update: load only changed opam files #6614

Are you sure you want to change the base?

opam update: load only changed opam files #6614

Uh oh!

Conversation

arozovyk commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

arozovyk commented Jul 30, 2025

Uh oh!

kit-ty-kate commented Jul 30, 2025

Uh oh!

arozovyk commented Jul 30, 2025

Uh oh!

kit-ty-kate left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

kit-ty-kate left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

`opam update`: load only changed opam files #6614

`opam update`: load only changed opam files #6614

arozovyk commented Jul 22, 2025 •

edited

Loading

kit-ty-kate left a comment •

edited

Loading