[vcpkg scripts] Use vcpkg fetch for available tools #43309

Neumann-A · 2025-01-16T11:07:48Z

No description provided.

scripts/cmake/vcpkg_find_acquire_program(NINJA).cmake

Neumann-A · 2025-02-03T12:39:11Z

@JonLiu1993: Please mark this as reviewed so that we can get a review from the ms team.

…fetch

BillyONeal · 2025-02-06T23:47:41Z

@JavierMatosD @AugP @vicroms @ras0219-msft and I discussed today.

We are happy with the direction of the 4 tools in Billy's original list being changed to use fetch.

We observe that some tools in Billy's original 'unsafe' list might be OK to go through fetch, in particular assemblers like nasm or yasm. However, fetch doesn't already know about those tools, so the value in attempting to make a change like this in support of them is probably not worth it, so we aren't interested in going there right now.

Neumann-A · 2025-02-07T10:44:59Z

list might be OK to go through fetch, in particular assemblers like nasm or yasm

Wouldn't this break the abi hash if the codegen is different for whatever reason between versions

Neumann-A · 2025-02-07T18:14:35Z

@BillyONeal:

Any Idea:

CMake Error at scripts/test_ports/vcpkg-find-acquire-program/portfile.cmake:66 (message):
  The following programs do not exist:

     NUGET:  trying authoritative source https://dist.nuget.org/win-x86-commandline/v6.10.0/nuget.exe

  Successfully downloaded 71d7307b-nuget.exe, storing to
  https://vcpkgassetcachewus.blob.core.windows.net/cache/71d7307bb89de2df3811419c561efa00618a4c68e6ce481b0bdfc94c7c6c6d126a54eb26a0015686fabf99f109744ca41fead99e97139cdc86dde16a5ec3e7cf?***
  SECRET ***

  Store success

  D:\downloads\tools\nuget-6.10.0-windows\nuget.exe

     NUGET (interpreter): A suitable version of nuget was not found (required v6.10.0).

  Trying to download 71d7307b-nuget.exe using asset cache
  https://vcpkgassetcachewus.blob.core.windows.net/cache/71d7307bb89de2df3811419c561efa00618a4c68e6ce481b0bdfc94c7c6c6d126a54eb26a0015686fabf99f109744ca41fead99e97139cdc86dde16a5ec3e7cf?***
  SECRET ***

  Asset cache miss
Call Stack (most recent call first):
  scripts/ports.cmake:196 (include)

The rest looks like baseline regression

BillyONeal · 2025-02-10T18:31:33Z

Wouldn't this break the abi hash if the codegen is different for whatever reason between versions

The assumption is that an assembler would not do that. To be clear, I'm not saying we should, I just wanted what was said to be recorded should this topic ever come up again.

BillyONeal · 2025-02-10T18:35:24Z

@BillyONeal:

Any Idea:

This looks like something tried to set the entire vcpkg console output as the path to nuget.exe, including status explanations of what we are doing and why.

I observe that many of our existing uses of fetch run it more than once to try to avoid this happening, there might be tool bugs that need to be tackled here.

BillyONeal · 2025-02-10T18:52:37Z

scripts/cmake/vcpkg_find_acquire_program.cmake

+    vcpkg_execute_in_download_mode(
+        COMMAND "$ENV{VCPKG_COMMAND}" fetch "${arg_FETCH_NAME}"
+        RESULT_VARIABLE error_code
+        OUTPUT_VARIABLE ${program}


Ah yeah this explains the funny NuGet behavior. fetch needs to be audited to send status messages to stderr.

https://github.com/microsoft/vcpkg-tool/blob/5d180566dc8204de49a53e167d990e79c03ee7f3/src/vcpkg/commands.fetch.cpp#L14

Or at least pass --x-stderr-status

Seems like it is not enough:

still get:

CMake Error at scripts/test_ports/vcpkg-find-acquire-program/portfile.cmake:66 (message): The following programs do not exist: NUGET: Trying to download 71d7307b-nuget.exe using asset cache https://vcpkgassetcachewus.blob.core.windows.net/cache/71d7307bb89de2df3811419c561efa00618a4c68e6ce481b0bdfc94c7c6c6d126a54eb26a0015686fabf99f109744ca41fead99e97139cdc86dde16a5ec3e7cf?*** SECRET *** Download successful! Asset cache hit, did not try authoritative source https://dist.nuget.org/win-x86-commandline/v6.10.0/nuget.exe D:\downloads\tools\nuget-6.10.0-windows\nuget.exe

Still an issue.

@Neumann-A @dg0yt Finally have what I hope is a fix for this out for review microsoft/vcpkg-tool#1842

scripts/cmake/vcpkg_find_acquire_program.cmake

BillyONeal · 2025-02-11T02:50:47Z

Pushed merge to pick up #43509 as the macs in the old pool are being shut down.

Neumann-A · 2025-02-14T14:50:40Z

convert to draft until fetch only outputs the result to stdout (requires vcpkg-tool change)

* Fixes the incorrect tracking bug introduced here: https://github.com/microsoft/vcpkg-tool/pull/1715/files#r2414987620 * Gets clean_prepare_dir and the last_write_time fixup into the parallelized region * Deletes decompress_in_parallel / cmd_execute_and_capture_output_parallel related to microsoft/vcpkg#43309

* Fixes the incorrect tracking bug introduced here: https://github.com/microsoft/vcpkg-tool/pull/1715/files#r2414987620 * Gets clean_prepare_dir and the last_write_time fixup into the parallelized region Related to microsoft/vcpkg#43309 * Plumbs error handling through to one place * Deletes decompress_in_parallel / cmd_execute_and_capture_output_parallel

…1805) * Fixes the incorrect tracking bug introduced here: https://github.com/microsoft/vcpkg-tool/pull/1715/files#r2414987620 * Gets clean_prepare_dir and the last_write_time fixup into the parallelized region Related to microsoft/vcpkg#43309 * Plumbs error handling through to one place * Deletes decompress_in_parallel / cmd_execute_and_capture_output_parallel

…cture. Unblocks microsoft/vcpkg#43309 by making --x-stderr-status actually work to `vcpkg fetch`

Use ninja from vcpkg fetch

c3b1e19

Neumann-A commented Jan 16, 2025

View reviewed changes

scripts/cmake/vcpkg_find_acquire_program(NINJA).cmake Outdated Show resolved Hide resolved

dg0yt reviewed Jan 16, 2025

View reviewed changes

scripts/cmake/vcpkg_find_acquire_program(NINJA).cmake Outdated Show resolved Hide resolved

LilyWangLL assigned JonLiu1993 Jan 17, 2025

JonLiu1993 changed the title ~~Use ninja from vcpkg fetch~~ [vcpkg scripts] Use ninja from vcpkg fetch Jan 17, 2025

JonLiu1993 added the category:tool-update The issue is with build tool or build script, which requires update or should be executed correctly label Jan 17, 2025

JonLiu1993 added the info:needs-maintainer-attention Lets the current 'on rotation' vcpkg maintainer know they need to look at this. label Feb 5, 2025

Neumann-A added 2 commits February 6, 2025 08:49

Merge remote-tracking branch 'upstream/master' into ninja_from_vcpkg_…

d576cd0

…fetch

Introduce function z_use_vcpkg_fetch

77abc3b

Neumann-A changed the title ~~[vcpkg scripts] Use ninja from vcpkg fetch~~ [vcpkg scripts] Use vcpkg fetch for available tools Feb 6, 2025

JonLiu1993 removed the info:needs-maintainer-attention Lets the current 'on rotation' vcpkg maintainer know they need to look at this. label Feb 7, 2025

Merge branch 'microsoft:master' into ninja_from_vcpkg_fetch

f634f2d

BillyONeal reviewed Feb 10, 2025

View reviewed changes

Neumann-A commented Feb 10, 2025

View reviewed changes

scripts/cmake/vcpkg_find_acquire_program.cmake Show resolved Hide resolved

Neumann-A and others added 2 commits February 10, 2025 19:56

use --x-stderr-status

cbf8e93

Merge remote-tracking branch 'origin/master' into HEAD

b955502

Neumann-A marked this pull request as draft February 14, 2025 14:49

LilyWangLL assigned Mengna-Li Apr 25, 2025

This was referenced Oct 7, 2025

Remove the 'upgrade' message. microsoft/vcpkg-tool#1802

Merged

Include NuGet console output on failures. microsoft/vcpkg-tool#1800

Merged

BillyONeal mentioned this pull request Oct 8, 2025

Fix ZipBinaryProvider unpack reporting + parallelize more microsoft/vcpkg-tool#1805

Merged

BillyONeal added a commit to BillyONeal/vcpkg-tool that referenced this pull request Nov 11, 2025

Plumb DiagnosticContext into ToolCache / download / caching infrastru…

486fbf9

…cture. Unblocks microsoft/vcpkg#43309 by making --x-stderr-status actually work to `vcpkg fetch`

BillyONeal mentioned this pull request Nov 11, 2025

Plumb DiagnosticContext into ToolCache / download / caching. microsoft/vcpkg-tool#1842

Open

[vcpkg scripts] Use vcpkg fetch for available tools #43309

Are you sure you want to change the base?

[vcpkg scripts] Use vcpkg fetch for available tools #43309

Conversation

Neumann-A commented Jan 16, 2025

Uh oh!

Uh oh!

Uh oh!

Neumann-A commented Feb 3, 2025

Uh oh!

BillyONeal commented Feb 6, 2025

Uh oh!

Neumann-A commented Feb 7, 2025

Uh oh!

Neumann-A commented Feb 7, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

BillyONeal commented Feb 10, 2025

Uh oh!

BillyONeal commented Feb 10, 2025

Uh oh!

BillyONeal Feb 10, 2025

Choose a reason for hiding this comment

Uh oh!

BillyONeal Feb 10, 2025

Choose a reason for hiding this comment

Uh oh!

Neumann-A Feb 12, 2025

Choose a reason for hiding this comment

Uh oh!

dg0yt May 13, 2025

Choose a reason for hiding this comment

Uh oh!

dg0yt Jul 3, 2025

Choose a reason for hiding this comment

Uh oh!

BillyONeal Nov 11, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

BillyONeal commented Feb 11, 2025

Uh oh!

Neumann-A commented Feb 14, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Neumann-A commented Feb 7, 2025 •

edited

Loading