0.0.16: JSON Error when listing containers

[MCOfficer]

Hello,

I just updated to 0.0.16, and a debian server i test on does not seem to like it, throwing a JSON error when trying to list containers:

    let docker = Docker::connect_with_defaults().unwrap();
    match docker.list_containers(None, None, None, ContainerFilters::new()) {
        Ok(result) => handle_containers(&opt, docker, result),
        Err(e) => println!("Error connecting to docker daemon: {} | cause: {} | trace: {}", e, e.cause().unwrap(), e.backtrace().unwrap()),
    };

Error connecting to docker daemon: json error | cause: missing field `IPv4Address` at line 1 column 54229 | trace: 

The same code works, on the same machine, with 0.0.15. My development PC works fine with both.

The server's docker version:

$ sudo docker version
Client:
 Version:           18.06.3-ce
 API version:       1.38
 Go version:        go1.10.3
 Git commit:        d7080c1
 Built:             Wed Feb 20 02:28:55 2019
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          18.06.3-ce
  API version:      1.38 (minimum version 1.12)
  Go version:       go1.10.3
  Git commit:       d7080c1
  Built:            Wed Feb 20 02:27:21 2019
  OS/Arch:          linux/amd64
  Experimental:     false

and my computer's:

❯ sudo docker version
Client:
 Version:           19.03.8-ce
 API version:       1.40
 Go version:        go1.14
 Git commit:        afacb8b7f0
 Built:             Mon Mar 16 22:23:09 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server:
 Engine:
  Version:          19.03.8-ce
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.14
  Git commit:       afacb8b7f0
  Built:            Mon Mar 16 22:22:53 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.3.3.m
  GitCommit:        d76c121f76a5fc8a462dc64594aea72fe18e1178.m
 runc:
  Version:          1.0.0-rc10
  GitCommit:        dc9208a3303feef5b3839f4323d9beb36df0a9dd
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

Please tell me if there's anything else i can provide.

[eldesh]

Thank you for your report. I'll check the api.

[eldesh]

@MCOfficer
Some container's network configuration seem to be not deserializable.

1. Can you tell me your container's configurations like below?

$ docker inspect <container-id> | jq '.[].NetworkSettings'

or more specific:

$ docker inspect <container-id> | jq '.[].NetworkSettings.Networks'

2. Can you tell me the way to create the container in question if you can?

[MCOfficer]

1. Can you tell me your container's configurations like below?

I will try, but it might take some time (there are >50 containers on the server in question). Is there any way to log which container fails? I will be able to test on monday.

2. Can you tell me the way to create the container in question if you can?

No can do, i'm afraid. If you tell me what to look for, i might be able to create a MRE, though.

[eldesh]

Is there any way to log which container fails?

It is difficult.

I expect to the deserializer of the network infomations (especially, IPAMConfig type) have some bugs.
Can you release your containers network configurations like below?

$ docker ps -a -q | while read id; do docker inspect $id | jq -c '.[].NetworkSettings.Networks.bridge.IPAMConfig'; done

Even an example of something different from null would be helpful.

[emk]

Thank you for maintaining dockworker!

I'm currently trying to replace boondock with dockworker in cage, and I'm hitting this same error.

I fought with quite a few of these errors back in the day when working on boondock, and here's the strategy I used:

1. Assume that basically all fields in all structures returned by docker should have type Option<T>, until proven otherwise. There's no internal documentation in the docker source about what can be null and what can't, and sooner or later, it feels like almost anything can be null.
2. Centralize all JSON decoding. There should only be one call to serde_json::from_slice in the library. And it should be serde_json::from_slice, unfortunately, and not serde_json::from_reader, because...
3. There needs to be some mechanism for automatically logging JSON parse errors with the surrounding context, so that you can see what broke serde_json. In boondock, I included the entire JSON blob in the error, but that could be cut down to just show the 30 characters on either side.

I would actually love to completely deprecate boondock in favor of dockworker, and just submit occasional PRs to dockworker. But to do that, there would need to be some kind of mutually agreed-upon solution for (1-3). Would that be of interest to you, if I did the refactoring?

[MCOfficer]

Is there any way to log which container fails?

It is difficult.

I expect to the deserializer of the network infomations (especially, IPAMConfig type) have some bugs.
Can you release your containers network configurations like below?

$ docker ps -a -q | while read id; do docker inspect $id | jq -c '.[].NetworkSettings.Networks.bridge.IPAMConfig'; done

Even an example of something different from null would be helpful.

Oh damn, i forgot this. Thanks for posting and thus giving me a notification.

I just checked, and all containers have IPAMConfig: null - all except for one, which has IPAMConfig: {} for some reason. Apart from that, it looks like any other container, which are produced en masse by our pipeline. Here are its Network settings (I obfuscated some values so they might be nonsensical):

        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "7c5ebca03e210aa5cdfa81206950a72584930291812fc82502ae0406efca60cf",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "3306/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/7c5ebcaace21",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "0a9c1de4bebcbf778248009fe2b4a747478e2136645563de7ba8d48f287d9388",
            "Gateway": "172.11.0.1",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "171.11.0.70",
            "IPPrefixLen": 16,
            "IPv6Gateway": "",
            "MacAddress": "01:42:0c:11:c0:f9",
            "Networks": {
                "bridge": {
                    "IPAMConfig": {},
                    "Links": null,
                    "Aliases": null,
                    "NetworkID": "c6bcc45303b33fb881911c25e755da483291123b0a8099e42b2226bcd4f2d549",
                    "EndpointID": "0a9c1de4bebcbf778248009fe2b4a74432012136645563de7ba8719e987d9388",
                    "Gateway": "172.11.0.1",
                    "IPAddress": "172.11.0.70",
                    "IPPrefixLen": 16,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "01:42:0c:11:c0:f9",
                    "DriverOpts": null
                }
            }
        }

[eldesh]

@MCOfficer
I opend a PR #85 for fixing this error. Can you check it?
You can use it like:
dockworker = { git = "https://github.com/eldesh/dockworker", branch = "fix/network_settings" } in Cargo.toml.

[eldesh]

@emk Thank you for your suggestions. I have opend a new issue #86 for discussion.