Skip to content

Commit b0f23d0

Browse files
ericphansonDilumAluthgejakobnissenhyrodiumtimholy
authored
[release-1.5] Backport bugfixes to release-1.5 branch (#175)
* Before building and testing the package, make sure that the UUID has not been edited (#128) (cherry picked from commit 21843d0) * CI: Standardize the workflow for testing and changing the UUID (#129) (cherry picked from commit cd002c3) * fix #131 and add test (#132) (cherry picked from commit adbb974) * Improve inferability of download() (#133) (cherry picked from commit 848d374) * fix ci badge (#137) (cherry picked from commit 3870614) * Fix a handful of invalidations in expression-checking (#138) ChainRulesCore defines `==(a, b::AbstractThunk)` and its converse, and this invalidates a couple of poorly-typed Symbol checks. This more "SSA-like" way of writing the code is easier to infer. (cherry picked from commit 25f7af3) * tests: skip wrong host test for SSL_NO_VERIFY (fix #139) (#140) Since #114, we only turn off peer verification, not host verification when the `SSL_NO_VERIFY` variables are set. This means that the last set of tests in the "SSL no verify override" testset *should* fail for `wrong.host.badssl.com`. That is not what I was seeing, however — the test was still passing — which I found puzzling but just moved on with my life at the time. It turns out that the test *does* fail if libcurl is build with OpenSSL. Since whether the test passes or not for that host depends on how things are built, this change simply skips the test (by popping the URL from the set of tested URLS for that testset). The tests above that which use the easy hook mechanism are fixed in a different way: for those I made the hook disable both host and peer verification, which should fix the tests for any bad host including when the server sends the wrong host name. (cherry picked from commit e22219f) * Fix input body size detection for IOBuffer(codeunits(str)) (#143) Somewhat surprisingly, the type of this is not IOBuffer, but a related type (Base.GenericIOBuffer{Base.CodeUnits{UInt8, String}}). (cherry picked from commit 470b7f0) * Typo fix: indiation -> indication (#144) (cherry picked from commit 5f1509d) * use Timer instead of libuv timer API (cherry picked from commit 11493ff) * use FDWatcher instead of libuv poll API (cherry picked from commit 4c1d2af) * fix wrong definition of curl_socket_t on Windows (cherry picked from commit 2eb0491) * Revert "stop using raw libuv API" (#156) (cherry picked from commit c91876a) * Revert "Revert "stop using raw libuv API" (#156)" This reverts commit c91876a. (cherry picked from commit 69acc13) * add missing locks during Timer callbacks (cherry picked from commit 43a3484) * fix Timer usage (#158) (cherry picked from commit 62b497e) * Workaround for missing isopen check in FDWatcher (#161) (possible multithread race with this still needs to be fixed) (cherry picked from commit 7f91b8a) * Check for timer isopen correctly (#162) (cherry picked from commit 4250b35) * remove trailing whitespace (cherry picked from commit d8c626b) * Avoid infinite recursion in `timer_callback` (#164) Fixes #163 (cherry picked from commit a55825b) * should also look into headers for input_size (#167) If no content length is set while uploading some contents, Curl defaults to use chunked transfer encoding. In some cases we want to prevent that because the server may not support chunked transfers. With this change, the request method will also look at the headers while determining the input size and if found call `set_upload_size` as usual. So to switch off chunked transfers, one must also know and set the content length header while invoking `download` or `request` methods. (cherry picked from commit ab628ab) * rename: singularize add_{upload,seek}_callback These only add one callback so having them be plural is weird. (cherry picked from commit 5bd0826) * add support for setting a debug callback (cherry picked from commit 55a0c39) * end-to-end tests for #167 This adds end-to-end tests for the changes introduced in #167. Verbose mode is switched off for these tests, but switching it on would show that not setting content-length headers results in chunked transfer encoding while setting it prevents that. Both tests should pass. (cherry picked from commit 911368d) * tests: use debug option to test for non/chunked uploads This combines the functionality from the previous two commits to not only trigger both chunked and non-chunked uploads, but also test for that difference by capturing and inspecting the debug events. (cherry picked from commit 4e0408a) * bump patch Co-authored-by: Dilum Aluthge <[email protected]> Co-authored-by: Jakob Nybo Nissen <[email protected]> Co-authored-by: Yuto Horikawa <[email protected]> Co-authored-by: Tim Holy <[email protected]> Co-authored-by: Stefan Karpinski <[email protected]> Co-authored-by: Chris Foster <[email protected]> Co-authored-by: Benoît Legat <[email protected]> Co-authored-by: Jameson Nash <[email protected]> Co-authored-by: Tanmay Mohapatra <[email protected]>
1 parent 26d79af commit b0f23d0

12 files changed

+338
-199
lines changed

.ci/change-uuid.jl

-5
This file was deleted.

.ci/test_and_change_uuid.jl

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
@static if Base.VERSION >= v"1.6"
2+
using TOML
3+
using Test
4+
else
5+
using Pkg: TOML
6+
using Test
7+
end
8+
9+
# To generate the new UUID, we simply modify the first character of the original UUID
10+
const original_uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"
11+
const new_uuid = "e43a241f-c20a-4ad4-852c-f6b1247861c6"
12+
13+
# `@__DIR__` is the `.ci/` folder.
14+
# Therefore, `dirname(@__DIR__)` is the repository root.
15+
const project_filename = joinpath(dirname(@__DIR__), "Project.toml")
16+
17+
@testset "Test that the UUID is unchanged" begin
18+
project_dict = TOML.parsefile(project_filename)
19+
@test project_dict["uuid"] == original_uuid
20+
end
21+
22+
write(
23+
project_filename,
24+
replace(
25+
read(project_filename, String),
26+
r"uuid = .*?\n" => "uuid = \"$(new_uuid)\"\n",
27+
),
28+
)

.github/workflows/ci.yml

+1-1
Original file line numberDiff line numberDiff line change
@@ -47,7 +47,7 @@ jobs:
4747
${{ runner.os }}-test-${{ env.cache-name }}-
4848
${{ runner.os }}-test-${{ matrix.os }}
4949
${{ runner.os }}-
50-
- run: julia --color=yes .ci/change-uuid.jl
50+
- run: julia --color=yes .ci/test_and_change_uuid.jl
5151
- uses: julia-actions/julia-buildpkg@v1
5252
- uses: julia-actions/julia-runtest@v1
5353
- uses: julia-actions/julia-processcoverage@v1

Project.toml

+2-1
Original file line numberDiff line numberDiff line change
@@ -1,10 +1,11 @@
11
name = "Downloads"
22
uuid = "f43a241f-c20a-4ad4-852c-f6b1247861c6"
33
authors = ["Stefan Karpinski <[email protected]> and contributors"]
4-
version = "1.5.2"
4+
version = "1.5.3"
55

66
[deps]
77
ArgTools = "0dad84c5-d112-42e6-8d28-ef12dabb789f"
8+
FileWatching = "7b1f6079-737a-58dc-b8bc-7a2ca5c1b5ee"
89
LibCURL = "b27032c2-a3e7-50c8-80cd-2d36dbcbfd21"
910
NetworkOptions = "ca575930-c2e3-43a9-ace4-1e988b2c1908"
1011

README.md

+17-7
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,6 @@
11
# Downloads
22

3-
[![Build Status](https://travis-ci.com/JuliaLang/Downloads.jl.svg?branch=master)](https://travis-ci.com/JuliaLang/Downloads.jl)
3+
[![Build Status](https://github.com/JuliaLang/Downloads.jl/workflows/CI/badge.svg)](https://github.com/JuliaLang/Downloads.jl/actions)
44
[![Codecov](https://codecov.io/gh/JuliaLang/Downloads.jl/branch/master/graph/badge.svg)](https://codecov.io/gh/JuliaLang/Downloads.jl)
55

66
The `Downloads` package provides a single function, `download`, which provides
@@ -16,10 +16,10 @@ Julia 1.3 through 1.5 as well.
1616

1717
The public API of `Downloads` consists of two functions and three types:
1818

19-
- `download` download a file from a URL, erroring if it can't be downloaded
19+
- `download` download a file from a URL, erroring if it can't be downloaded
2020
- `request` — request a URL, returning a `Response` object indicating success
21-
- `Response` a type capturing the status and other metadata about a request
22-
- `RequestError` an error type thrown by `download` and `request` on error
21+
- `Response` a type capturing the status and other metadata about a request
22+
- `RequestError` an error type thrown by `download` and `request` on error
2323
- `Downloader` — an object encapsulating shared resources for downloading
2424

2525
### download
@@ -31,6 +31,7 @@ download(url, [ output = tempfile() ];
3131
[ timeout = <none>, ]
3232
[ progress = <none>, ]
3333
[ verbose = false, ]
34+
[ debug = <none>, ]
3435
[ downloader = <default>, ]
3536
) -> output
3637
```
@@ -41,6 +42,7 @@ download(url, [ output = tempfile() ];
4142
- `timeout :: Real`
4243
- `progress :: (total::Integer, now::Integer) --> Any`
4344
- `verbose :: Bool`
45+
- `debug :: (type, message) --> Any`
4446
- `downloader :: Downloader`
4547

4648
Download a file from the given url, saving it to `output` or if not specified, a
@@ -67,12 +69,18 @@ which will be called whenever there are updates about the size and status of the
6769
ongoing download. The callback must take two integer arguments: `total` and
6870
`now` which are the total size of the download in bytes, and the number of bytes
6971
which have been downloaded so far. Note that `total` starts out as zero and
70-
remains zero until the server gives an indiation of the total size of the
72+
remains zero until the server gives an indication of the total size of the
7173
download (e.g. with a `Content-Length` header), which may never happen. So a
7274
well-behaved progress callback should handle a total size of zero gracefully.
7375

74-
If the `verbose` optoin is set to true, `libcurl`, which is used to implement
75-
the download functionality will print debugging information to `stderr`.
76+
If the `verbose` option is set to true, `libcurl`, which is used to implement
77+
the download functionality will print debugging information to `stderr`. If the
78+
`debug` option is set to a function accepting two `String` arguments, then the
79+
verbose option is ignored and instead the data that would have been printed to
80+
`stderr` is passed to the `debug` callback with `type` and `message` arguments.
81+
The `type` argument indicates what kind of event has occurred, and is one of:
82+
`TEXT`, `HEADER IN`, `HEADER OUT`, `DATA IN`, `DATA OUT`, `SSL DATA IN` or `SSL
83+
DATA OUT`. The `message` argument is the description of the debug event.
7684

7785
### request
7886

@@ -85,6 +93,7 @@ request(url;
8593
[ timeout = <none>, ]
8694
[ progress = <none>, ]
8795
[ verbose = false, ]
96+
[ debug = <none>, ]
8897
[ throw = true, ]
8998
[ downloader = <default>, ]
9099
) -> Union{Response, RequestError}
@@ -97,6 +106,7 @@ request(url;
97106
- `timeout :: Real`
98107
- `progress :: (dl_total, dl_now, ul_total, ul_now) --> Any`
99108
- `verbose :: Bool`
109+
- `debug :: (type, message) --> Any`
100110
- `throw :: Bool`
101111
- `downloader :: Downloader`
102112

src/Curl/Curl.jl

+26-2
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ export
66
set_url,
77
set_method,
88
set_verbose,
9+
set_debug,
910
set_body,
1011
set_upload_size,
1112
set_seeker,
@@ -26,15 +27,38 @@ export
2627
remove_handle
2728

2829
using LibCURL
29-
using LibCURL: curl_off_t
30+
using LibCURL: curl_off_t, libcurl
3031
# not exported: https://github.com/JuliaWeb/LibCURL.jl/issues/87
3132

3233
# constants that LibCURL should have but doesn't
3334
const CURLE_PEER_FAILED_VERIFICATION = 60
3435
const CURLSSLOPT_REVOKE_BEST_EFFORT = 1 << 3
3536

37+
# these are incorrectly defined on Windows by LibCURL:
38+
if Sys.iswindows()
39+
const curl_socket_t = Base.OS_HANDLE
40+
const CURL_SOCKET_TIMEOUT = Base.INVALID_OS_HANDLE
41+
else
42+
const curl_socket_t = Cint
43+
const CURL_SOCKET_TIMEOUT = -1
44+
end
45+
46+
# definitions affected by incorrect curl_socket_t (copied verbatim):
47+
function curl_multi_socket_action(multi_handle, s, ev_bitmask, running_handles)
48+
ccall((:curl_multi_socket_action, libcurl), CURLMcode, (Ptr{CURLM}, curl_socket_t, Cint, Ptr{Cint}), multi_handle, s, ev_bitmask, running_handles)
49+
end
50+
function curl_multi_assign(multi_handle, sockfd, sockp)
51+
ccall((:curl_multi_assign, libcurl), CURLMcode, (Ptr{CURLM}, curl_socket_t, Ptr{Cvoid}), multi_handle, sockfd, sockp)
52+
end
53+
54+
# additional curl_multi_socket_action method
55+
function curl_multi_socket_action(multi_handle, s, ev_bitmask)
56+
curl_multi_socket_action(multi_handle, s, ev_bitmask, Ref{Cint}())
57+
end
58+
59+
using FileWatching
3660
using NetworkOptions
37-
using Base: preserve_handle, unpreserve_handle
61+
using Base: OS_HANDLE, preserve_handle, unpreserve_handle
3862

3963
include("utils.jl")
4064

src/Curl/Easy.jl

+65-8
Original file line numberDiff line numberDiff line change
@@ -9,6 +9,7 @@ mutable struct Easy
99
res_hdrs :: Vector{String}
1010
code :: CURLcode
1111
errbuf :: Vector{UInt8}
12+
debug :: Union{Function,Nothing}
1213
end
1314

1415
const EMPTY_BYTE_VECTOR = UInt8[]
@@ -25,6 +26,7 @@ function Easy()
2526
String[],
2627
typemax(CURLcode),
2728
zeros(UInt8, CURL_ERROR_SIZE),
29+
nothing,
2830
)
2931
finalizer(done!, easy)
3032
add_callbacks(easy)
@@ -106,6 +108,19 @@ function set_verbose(easy::Easy, verbose::Bool)
106108
setopt(easy, CURLOPT_VERBOSE, verbose)
107109
end
108110

111+
function set_debug(easy::Easy, debug::Function)
112+
hasmethod(debug, Tuple{String,String}) ||
113+
throw(ArgumentError("debug callback must take (::String, ::String)"))
114+
easy.debug = debug
115+
add_debug_callback(easy)
116+
set_verbose(easy, true)
117+
end
118+
119+
function set_debug(easy::Easy, debug::Nothing)
120+
easy.debug = nothing
121+
remove_debug_callback(easy)
122+
end
123+
109124
function set_body(easy::Easy, body::Bool)
110125
setopt(easy, CURLOPT_NOBODY, !body)
111126
end
@@ -116,7 +131,7 @@ function set_upload_size(easy::Easy, size::Integer)
116131
end
117132

118133
function set_seeker(seeker::Function, easy::Easy)
119-
add_seek_callbacks(easy)
134+
add_seek_callback(easy)
120135
easy.seeker = seeker
121136
end
122137

@@ -159,7 +174,7 @@ function enable_progress(easy::Easy, on::Bool=true)
159174
end
160175

161176
function enable_upload(easy::Easy)
162-
add_upload_callbacks(easy::Easy)
177+
add_upload_callback(easy::Easy)
163178
setopt(easy, CURLOPT_UPLOAD, true)
164179
end
165180

@@ -229,6 +244,17 @@ function status_ok(proto::AbstractString, status::Integer)
229244
end
230245
status_ok(proto::Nothing, status::Integer) = false
231246

247+
function info_type(type::curl_infotype)
248+
type == 0 ? "TEXT" :
249+
type == 1 ? "HEADER IN" :
250+
type == 2 ? "HEADER OUT" :
251+
type == 3 ? "DATA IN" :
252+
type == 4 ? "DATA OUT" :
253+
type == 5 ? "SSL DATA IN" :
254+
type == 6 ? "SSL DATA OUT" :
255+
"UNKNOWN"
256+
end
257+
232258
function get_effective_url(easy::Easy)
233259
url_ref = Ref{Ptr{Cchar}}()
234260
@check curl_easy_getinfo(easy.handle, CURLINFO_EFFECTIVE_URL, url_ref)
@@ -252,11 +278,13 @@ function get_response_info(easy::Easy)
252278
for hdr in easy.res_hdrs
253279
if contains(hdr, r"^\s*$")
254280
# ignore
255-
elseif (m = match(r"^(HTTP/\d+(?:.\d+)?\s+\d+\b.*?)\s*$", hdr)) !== nothing
256-
message = m.captures[1]
281+
elseif (m = match(r"^(HTTP/\d+(?:.\d+)?\s+\d+\b.*?)\s*$", hdr); m) !== nothing
282+
message = m.captures[1]::SubString{String}
257283
empty!(headers)
258-
elseif (m = match(r"^(\S[^:]*?)\s*:\s*(.*?)\s*$", hdr)) !== nothing
259-
push!(headers, lowercase(m.captures[1]) => m.captures[2])
284+
elseif (m = match(r"^(\S[^:]*?)\s*:\s*(.*?)\s*$", hdr); m) !== nothing
285+
key = lowercase(m.captures[1]::SubString{String})
286+
val = m.captures[2]::SubString{String}
287+
push!(headers, key => val)
260288
else
261289
@warn "malformed HTTP header" url status header=hdr
262290
end
@@ -374,6 +402,19 @@ function progress_callback(
374402
return 0
375403
end
376404

405+
function debug_callback(
406+
handle :: Ptr{Cvoid},
407+
type :: curl_infotype,
408+
data :: Ptr{Cchar},
409+
size :: Csize_t,
410+
easy_p :: Ptr{Cvoid},
411+
)::Cint
412+
easy = unsafe_pointer_to_objref(easy_p)::Easy
413+
@assert easy.handle == handle
414+
easy.debug(info_type(type), unsafe_string(data, size))
415+
return 0
416+
end
417+
377418
function add_callbacks(easy::Easy)
378419
# pointer to easy object
379420
easy_p = pointer_from_objref(easy)
@@ -402,7 +443,7 @@ function add_callbacks(easy::Easy)
402443
setopt(easy, CURLOPT_XFERINFODATA, easy_p)
403444
end
404445

405-
function add_upload_callbacks(easy::Easy)
446+
function add_upload_callback(easy::Easy)
406447
# pointer to easy object
407448
easy_p = pointer_from_objref(easy)
408449

@@ -413,7 +454,7 @@ function add_upload_callbacks(easy::Easy)
413454
setopt(easy, CURLOPT_READDATA, easy_p)
414455
end
415456

416-
function add_seek_callbacks(easy::Easy)
457+
function add_seek_callback(easy::Easy)
417458
# pointer to easy object
418459
easy_p = pointer_from_objref(easy)
419460

@@ -423,3 +464,19 @@ function add_seek_callbacks(easy::Easy)
423464
setopt(easy, CURLOPT_SEEKFUNCTION, seek_cb)
424465
setopt(easy, CURLOPT_SEEKDATA, easy_p)
425466
end
467+
468+
function add_debug_callback(easy::Easy)
469+
# pointer to easy object
470+
easy_p = pointer_from_objref(easy)
471+
472+
# set debug callback
473+
debug_cb = @cfunction(debug_callback,
474+
Cint, (Ptr{Cvoid}, curl_infotype, Ptr{Cchar}, Csize_t, Ptr{Cvoid}))
475+
setopt(easy, CURLOPT_DEBUGFUNCTION, debug_cb)
476+
setopt(easy, CURLOPT_DEBUGDATA, easy_p)
477+
end
478+
479+
function remove_debug_callback(easy::Easy)
480+
setopt(easy, CURLOPT_DEBUGFUNCTION, C_NULL)
481+
setopt(easy, CURLOPT_DEBUGDATA, C_NULL)
482+
end

0 commit comments

Comments
 (0)