How to reboot during provisioning? Request for docs or feature

[robotrapta]

Description

Howdy y'all! I need to restart my machine during provisioning. I'm new to packer coming from systems like ansible and chef. I've read a bunch of docs on this, and am still confused. So I think this is at least a doc bug, and perhaps a full feature request.

I know this issue has been discussed in #1983 and that there was a proposal to build a native provisioner in #4555 - a native feature makes a ton of sense to me.

I know the recommended way to do this is with a shell provisioner. This seems to rely on the retry mechanism, treating the reboot as a kinda-expected error and then getting the system to recover from it appropriately. Maybe this is useful for other reasons, but it feels like an ugly hack. A hack would be okay if it was clearly documented and worked reliably. But it's not clearly documented - there is no example that I can find showing how to do this. My first few attempts to get it to work after reading the docs were unreliable - sometimes it failed, and sometimes it re-ran things unnecessarily. So it would be great to just tell people the standard way to do this if there isn't a built-in way to do it.

I think a good way to do it is this:

  provisioner "shell" {
    expect_disconnect = true
    inline = [
      "sudo reboot now",
    ]
    pause_after  = "10s"
  }

The pause_after I believe is important to minimize the risk of an expected race condition in issuing the next provisioning command? Which seems to me like a pretty strong argument for making this a native feature.

If that's in fact correct, putting that example code in the docs would be awesome. Thanks!

Use Case(s)

I'm trying to install nvidia drivers to use CUDA, which generally requires a reboot.

[azr]

Hello @robotrapta ! Thanks for opening, yeah, I agree with you here about the fact that this feel like a hack. The thing is that Packer sorts of expects to log in to an instance once — in the beginning — and there is no internal/native way to 'reconnect' that feel great. So we would like to introduce a new feature soon to be able to 'connect' in the middle of a build. This would allow changing SSH settings or reboot after an installation. But that one will not come straight away, as we have quite a large and growing to-do list. Making a docs page about that would be a good idea, I think. I'll bring that one up to the team.

One thing that comes to mind here is that you could install the drivers at the end of your provisioning steps, and just shutdown/save the machine. Upon next boot, things should be configured. If you have more things to install/configure, then you could for example start another build ?

With that said, and if that does not work out, do you mind sharing your build file ? And your logs ? Maybe we can help you better/differently from there.

[keviiin38]

Hello ! I'm trying to achieve something similar:

• Provision using Ansible to install Nvidia CUDA drivers and other stuff
• Need a reboot of the instance to reload kernel, etc...
• Finally running a last shell with Serverspec and checking that everything is ok

{
    "provisioners": [
        {
            "type": "ansible",
            "playbook_file": "playbook.yml"
        },
        {
            "type": "shell",
            "inline": [ "reboot now" ],
            "expect_disconnect": true
        },
        {
            "type": "file",
            "source": "serverspec/",
            "destination": "/tmp",
            "pause_before": "30s"
        },
        {
            "type": "shell",
            "script": "serverspec.sh"
        }
    ]
}

I'm using pause_before in the next provisioner instead of the pause_after, but don't know which one is better.

[robotrapta]

Hi @azr thanks for the suggestion of installing the drivers at the end - unfortunately that doesn't work for me. There's a bunch of software I need to install which depends on having CUDA installed, and some of those installations will fail if it can't confirm nvidia hardware/drivers present.

[ccrvlh]

Hi, similar use case here, and found the results to be sort of inconsistent.
My current template is something like:

  provisioner "shell" {
    pause_before = "10s"
    script = "./scripts/updates.sh"
  }

  provisioner "ansible-local" {
    role_paths      = ["./roles"]
    playbook_file   = "./roles/ubuntu/ubuntu.yml"
    command         = "sudo ansible-playbook -i localhost -e 'ansible_python_interpreter=/usr/bin/python3'"
  }

  provisioner "shell" {
    script = "./scripts/reboot.sh"
    expect_disconnect = true
  }

  provisioner "shell" {
    pause_before = "120s"
    script = "./scripts/finish.sh"
    max_retries = 3
  }

What I've found is that I always can see the message that the reboot.sh prints (Rebooting to apply updates), and sometimes I see the next message (Pausing for 2 minutes before next text or something similar, can't remeber). But sometimes I the build just fails after the reboot message. This seems strange, since I've been following the VMs and reboot usually take around 30-60 seconds, and I do have the retry mechanism for the finish provisioner. Couldn't quite find a reliable way to do that. I ran about 10-20 builds today and is completely hit or miss.

I understand the solution proposed by @azr but just as @robotrapta I also wanted to perform actions after the reboot. There may be a work around, sure, but the reboot is just the natural way to go for us.

The impression I get is that Packer "crashes" (probably not the right wording, pardon me) after the reboot, even with expect_disconnect and doesn't understand the next task to perform (wait to reconnect).

I could try a few more tests with debug logs on maybe.