updated sapi to image 40857b84-1709-11e6-babc-e7e08d4dbcbb, now 'sdcadm check-config' reports missing data for manta services

[pccowboy]

Took an update this weekend to sapi, while working out another issue. Seemed like everything was working fine, but trying to login to madtom0 this morning, I got a message saying:

opt/smartdc/bin/sdc-login: fatal error: No VMs found matching criteria

Checked in vmadm list, the VM is indeed there. Went to run sdcadm check-config, and saw this:
https://gist.github.com/pccowboy/33e6033068acac54233ca8435884d57b

I tried to rollback the update, same results. So I re-ran the update, but now stuck with the missing bits in SAPI, and unable to use sdc-login.

'sdcadm services' reports the following for sapi:

vm     4a397376-cb4c-483c-a198-f4bfda5f56fa  sapi             9d5bbe5c-6b20-11e6-97fb-cfb0d0cbc94a

bahamat mentioned that all the missing items were related to manta.

manta services were working, at least from manta0. I could use mls and mget, even though the only data out there were reports, it worked without issue. Outside of manta0, mls timed out.

Since I did not have data in my manta that I needed, I tried manta-factoryreset and reprovisioning manta using

manta-init -s lab -e [email protected]   #forced me to use a new email, said the old one could not be re-used
manta-deploy-lab

Even after that, the output from 'sdcadm check-config' stayed the same as the above gist, none of the missing items were added to sapi by the reprovision of manta.

Is there a way to rebuild the SAPI data for manta that I have missed?

Here is the data from 'svcadm history' that I believe references the update I took.

        {
            "service": {
                "uuid": "c5bf2921-514e-4e0f-aa2c-7bd3f89591f2",
                "name": "dhcpd",
                "application_uuid": "09c1d8bc-e9a1-4eba-b2af-8842b2855ed4",
                "params": {
                    "archive_on_delete": true,
                    "package_name": "sdc_128",
                    "image_uuid": "40857b84-1709-11e6-babc-e7e08d4dbcbb",
                    "maintain_resolvers": true,
                    "networks": [
                        "admin"
                    ],
                    "filesystems": [
                        {
                            "source": "/usbkey/os",
                            "target": "/tftpboot/os",
                            "type": "lofs",
                            "options": [
                                "ro",
                                "nodevices"
                            ]
                        }
                    ],
                    "tags": {
                        "smartdc_role": "dhcpd",
                        "smartdc_type": "core"
                    },
                    "cpu_shares": 128,
                    "cpu_cap": 100,
                    "zfs_io_priority": 10,
                    "max_lwps": 1000,
                    "max_physical_memory": 128,
                    "max_locked_memory": 128,
                    "max_swap": 256,
                    "quota": "25",
                    "package_version": "1.0.0",
                    "billing_id": "73a1ca34-1e30-48c7-8681-70314a9c67d3",
                    "customer_metadata": {}
                },
                "metadata": {
                    "SERVICE_NAME": "dhcpd",
                    "SERVICE_DOMAIN": "dhcpd.home.pccowboy.com",
                    "sapi-url": "http://10.67.41.27",
                    "assets-ip": "10.67.41.3",
                    "user-script": "#!/usr/bin/bash\n#\n# This Source Code Form is subject to the terms of the Mozilla Pu
blic\n# License, v. 2.0. If a copy of the MPL was not distributed with this\n# file, You can obtain one at http://mozilla
.org/MPL/2.0/.\n#\n\n#\n# Copyright (c) 2014, Joyent, Inc.\n#\n\nexport PS4='[\\D{%FT%TZ}] ${BASH_SOURCE}:${LINENO}: ${FU
NCNAME[0]:+${FUNCNAME[0]}(): }'\n\nset -o xtrace\nset -o errexit\nset -o pipefail\n\n#\n# The presence of the /var/svc/.r
an-user-script file indicates that the\n# instance has already been setup (i.e. the instance has booted previously).\n#\n
# Upon first boot, run the setup.sh script if present. On all boots including\n# the first one, run the configure.sh scri
pt if present.\n#\n\nSENTINEL=/var/svc/.ran-user-script\n\nDIR=/opt/smartdc/boot\n\nif [[ ! -e ${SENTINEL} ]]; then\n
if [[ -f ${DIR}/setup.sh ]]; then\n        ${DIR}/setup.sh 2>&1 | tee /var/svc/setup.log\n    fi\n\n    touch ${SENTINEL}
\nfi\n\nif [[ ! -f ${DIR}/configure.sh ]]; then\n    echo \"Missing ${DIR}/configure.sh cannot configure.\"\n    exit 1\n
fi\n\nexec ${DIR}/configure.sh\n"
                },
                "type": "vm"
            },
            "type": "update-service",
            "image": {
                "v": 2,
                "uuid": "549795a6-4217-11e6-a3e8-dfba28abbe5d",
                "owner": "00000000-0000-0000-0000-000000000000",
                "name": "dhcpd",
                "version": "master-20160704T183440Z-gf985c3d",
                "state": "active",
                "disabled": false,
                "public": false,
                "published_at": "2016-07-04T18:44:12.612Z",
                "type": "zone-dataset",
                "os": "smartos",
                "files": [
                    {
                        "sha1": "4a8b1973f4744262dbe827f43c0fb27503e37a7a",
                        "size": 47090264,
                        "compression": "gzip"
                    }
                ],
               "description": "SDC",
                "requirements": {
                    "min_platform": {
                        "7.0": "20130506T233003Z"
                    },
                    "networks": [
                        {
                            "name": "net0",
                            "description": "admin"
                        }
                    ]
                },
                "origin": "fd2cc906-8938-11e3-beab-4359c665ac99",
                "tags": {
                    "smartdc_service": true
                },
                "channels": [
                    "dev"
                ]
            },
            "inst": {
                "type": "vm",
                "alias": "dhcpd0",
                "version": "master-20160510T234059Z-gf985c3d",
                "instance": "926d7ffe-eac0-4bf3-a37c-e544435b195e",
                "zonename": "926d7ffe-eac0-4bf3-a37c-e544435b195e",
                "service": "dhcpd",
                "image": "40857b84-1709-11e6-babc-e7e08d4dbcbb",
                "state": "running",
                "server": "12399ec8-ef21-230e-8ee9-0025901107e2",
                "hostname": "headnode",
                "server_ip": "10.67.41.2",
                "ip": "10.67.41.4",
                "healthy": true
            }
        },

Thanks in advance...
David