certificate verify failed (unable to get issuer certificate)

[muscat]

hi!

I'm trying to use your image on a Raspberry Pi 4.
aarch64 (ARM), 8Gb RAM, 22Gb free disk space.

I'm launching it with the recommended parameters:
docker run --name puppet --hostname puppet.vpn.rv.ua -p 8140:8140 -v ./code:/etc/puppetlabs/code -v ./ca:/etc/puppetlabs/puppetserver/ca ghcr.io/voxpupuli/container-puppetserver

The server starts up. Here are the logs.

System configuration values:
* HOSTNAME: 'puppet.vpn.rv.ua'
* hostname -f: 'puppet.vpn.rv.ua'
* PUPPETSERVER_HOSTNAME: 'puppet'
* PUPPETSERVER_PORT: '8140'
* Certname: 'puppet.pem'
* DNS_ALT_NAMES: ''
* SSLDIR: '/etc/puppetlabs/puppet/ssl'
CA Certificate:
subject=CN = "Puppet CA generated on puppet.vpn.rv.ua at 2024-01-16 14:31:17 +0000"
issuer=CN = Puppet Root CA: 1b57b4220adc31
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Subject Key Identifier: 
                4D:A0:F5:23:FC:B6:4A:4E:7F:EF:39:6C:E0:7D:73:1A:AB:60:9F:E8
            Netscape Comment: 
                Puppet Server Internal Certificate
            X509v3 Authority Key Identifier: 
                16:6C:70:7B:5D:56:FE:35:13:4F:64:1D:34:1C:C5:74:83:8C:26:B6
Certificate puppet.pem:
subject=CN = puppet
issuer=CN = "Puppet CA generated on puppet.vpn.rv.ua at 2024-01-16 14:31:17 +0000"
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:FALSE
            Netscape Comment: 
                Puppet Server Internal Certificate
            X509v3 Authority Key Identifier: 
                4D:A0:F5:23:FC:B6:4A:4E:7F:EF:39:6C:E0:7D:73:1A:AB:60:9F:E8
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Subject Key Identifier: 
                EE:43:BC:F1:6F:2C:A9:C5:62:26:42:BC:9D:21:B6:D1:D5:DA:9D:58
            1.3.6.1.4.1.34380.1.3.39: 
                ..true
            X509v3 Subject Alternative Name: 
                DNS:puppet, DNS:puppet

...


2024-01-16 14:33:39,574 INFO  [p.t.s.w.jetty9-core] Starting web server.
2024-01-16 14:33:39,589 INFO  [o.e.j.s.Server] jetty-9.4.53.v20231009; built: 2023-10-09T12:29:09.265Z; git: 27bde00a0b95a1d5bbee0eae7984f891d2d0f8c9; jvm 17.0.9+9-Ubuntu-122.04
2024-01-16 14:33:39,786 INFO  [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@7cedcbc9{/puppet-ca,null,AVAILABLE}
2024-01-16 14:33:39,790 INFO  [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@40ffcd92{/puppet-admin-api,null,AVAILABLE}
2024-01-16 14:33:39,855 INFO  [o.e.j.s.session] DefaultSessionIdManager workerName=node0
2024-01-16 14:33:39,856 INFO  [o.e.j.s.session] No SessionScavenger set, using defaults
2024-01-16 14:33:39,864 INFO  [o.e.j.s.session] node0 Scavenging every 600000ms
2024-01-16 14:33:39,994 INFO  [p.t.s.m.jolokia] Using policy access restrictor classpath:/jolokia-access.xml
2024-01-16 14:33:40,164 INFO  [o.e.j.s.h.ContextHandler] Started o.e.j.s.ServletContextHandler@79246c19{/metrics/v2,null,AVAILABLE}
2024-01-16 14:33:40,165 INFO  [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@284daed8{/puppet,null,AVAILABLE}
2024-01-16 14:33:40,259 INFO  [o.e.j.u.s.SslContextFactory] x509=X509@ef88faa(private key,h=[puppet],a=[],w=[]) for InternalSslContextFactory@470d0fef[provider=null,keyStore=null,trustStore=null]
2024-01-16 14:33:40,528 INFO  [o.e.j.s.AbstractConnector] Started ServerConnector@49c9371c{SSL, (ssl, http/1.1)}{0.0.0.0:8140}
2024-01-16 14:33:40,529 INFO  [o.e.j.s.Server] Started @91746ms
2024-01-16 14:33:40,548 INFO  [p.t.s.s.status-core] Starting background monitoring of cpu usage metrics
2024-01-16 14:33:40,564 INFO  [p.t.s.s.status-service] Registering status callback function for service 'status-service', version 1.1.2
2024-01-16 14:33:40,566 INFO  [p.t.s.s.status-service] Registering status service HTTP API at /status
2024-01-16 14:33:40,601 INFO  [o.e.j.s.h.ContextHandler] Started o.e.j.s.h.ContextHandler@2454e868{/status,null,AVAILABLE}
2024-01-16 14:33:40,662 INFO  [p.s.a.analytics-service] Puppet Server Update Service has successfully started and will run in the background
2024-01-16 14:33:40,674 INFO  [p.s.m.master-service] Puppet Server has successfully started and is now ready to handle requests
2024-01-16 14:55:03,325 INFO  [p.s.a.dropsonde] Successfully submitted module metrics via Dropsonde.

`

puppet agent config:

[main]
server = puppet.vpn.rv.ua
ca_server = puppet.vpn.rv.ua
environment = production
report = false
runinterval = 300
rundir = /var/run/puppet
factpath=$vardir/lib/facter
ssldir = /var/lib/puppet/ssl

[agent]
server = puppet.vpn.rv.ua

When a client tries to connect, I get an "unable to get issuer certificate" error.
logs on the client:

root@ip-172-31-1-176:~# puppet agent --verbose --onetime --no-daemonize --logdest console
Info: Creating a new SSL key for ip-172-31-1-176.us-east-2.compute.internal
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for ip-172-31-1-176.us-east-2.compute.internal
Info: Certificate Request fingerprint (SHA256): 85:DF:EA:7C:CB:28:80:36:E5:59:9A:9A:82:89:38:B1:44:0B:9C:C7:16:CF:1A:9F:22:2D:CC:50:3A:AE:D3:F0
Info: Caching certificate for ip-172-31-1-176.us-east-2.compute.internal
Error: request https://puppet.vpn.rv.ua:8140//puppet-ca/v1/certificate_revocation_list/ca failed: SSL_connect returned=1 errno=0 peeraddr=91.223.227.227:8140 state=error: certificate verify failed (unable to get issuer certificate)
Error: Could not request certificate: SSL_connect returned=1 errno=0 peeraddr=91.223.227.227:8140 state=error: certificate verify failed (unable to get issuer certificate): [unable to get issuer certificate for /CN=Puppet CA generated on puppet.vpn.rv.ua at 2024-01-16 14:31:17 \+0000]
Exiting; failed to retrieve certificate and waitforcert is disabled

logs on the server at the same moment:

3.141.192.43 - - - 16/Jan/2024:15:25:22 +0000 "GET /puppet-ca/v1/certificate/ca?environment=production&fail_on_404=true HTTP/1.1" 200 2791 3.141.192.43 3.141.192.43 8140 91
3.141.192.43 - - - 16/Jan/2024:15:25:23 +0000 "GET /puppet-ca/v1/certificate/ip-172-31-1-176.us-east-2.compute.internal?environment=production& HTTP/1.1" 404 69 3.141.192.43 3.141.192.43 8140 12
3.141.192.43 - - - 16/Jan/2024:15:25:23 +0000 "GET /puppet-ca/v1/certificate_request/ip-172-31-1-176.us-east-2.compute.internal?environment=production& HTTP/1.1" 404 77 3.141.192.43 3.141.192.43 8140 17
2024-01-16 15:25:25,475 INFO  [p.p.certificate-authority] Entity CA signed 1 certificate: ip-172-31-1-176.us-east-2.compute.internal.
3.141.192.43 - - - 16/Jan/2024:15:25:25 +0000 "PUT /puppet-ca/v1/certificate_request/ip-172-31-1-176.us-east-2.compute.internal?environment=production& HTTP/1.1" 200 0 3.141.192.43 3.141.192.43 8140 1412
3.141.192.43 - - - 16/Jan/2024:15:25:25 +0000 "GET /puppet-ca/v1/certificate/ip-172-31-1-176.us-east-2.compute.internal?environment=production& HTTP/1.1" 200 1668 3.141.192.43 3.141.192.43 8140 20

The client is a fresh EC2 instance, x86_64, Ubuntu 22 LTS.
only puppet-agent is installed, and the server configuration is specified.

please, help, how to solve it ?
thanks in advance

[rwaffen]

hmm strange, this should work, will try to reproduce

[rwaffen]

I normally run it from a compose.yaml. maybe this helps

services:
  puppet:
    image: ghcr.io/container-puppetserver:7.14.0-latest
    hostname: puppet
    environment:
      - PUPPETSERVER_HOSTNAME=puppet
      - PUPPETSERVER_PORT=8140
      - PUPPETDB_HOSTNAME=puppetdb
      - PUPPETDB_SSL_PORT=8081
      - USE_PUPPETDB=true
      - AUTOSIGN=true
      # For private repos, use git@github.com:user/repo.git and provide SSH keys
      # - R10K_REMOTE=https://github.com/betadots/demo-control-repo.git
    volumes:
      - puppetserver:/opt/puppetlabs/server/data/puppetserver
      - puppetserver-ssl:/etc/puppetlabs/puppet/ssl
      - puppetserver-ca:/etc/puppetlabs/puppetserver/ca
    restart: always
    ports:
      - 8140:8140

https://github.com/voxpupuli/crafty/blob/887019ac1ceff40a3659e526a4fb02e527e2ec65/puppet/oss/compose.yaml#L4-L22

[muscat]

Thank you for your response and advice.

I tried to reproduce...

I took a clean Ubuntu22 on ARM architecture at AWS.

Ubuntu 22.04 LTS, arm64 jammy image build on 2023-12-07
t4g.smalll, 2 vCPU, 2 GiB Memory, 20Gb EBS volume.

The security group is clear: allowing any traffic to any.
An external IP was added, domain 'puppet.vpn.rv.ua' was pointed to this external IP.

On this server, I performed the following tasks:

• set hostname (puppet.vpn.rv.ua)
• install docker (by official script curl -fsSL https://get.docker.com -o get-docker.sh)
• install docker-compose
• create docker-file.yml
  and tried to run a container.

Using EXACTLY your file, I got these errors:"
ERROR: Named volume "puppetserver:/opt/puppetlabs/server/data/puppetserver:rw" is used in service "puppet" but no declaration was found in the volumes section.
and
ERROR: Head "https://ghcr.io/v2/container-puppetserver/manifests/7.14.0-latest": name invalid

i manually get an image:

root@puppet:~# docker pull voxpupuli/container-puppetserver:7.14.0-latest
...
Status: Downloaded newer image for voxpupuli/container-puppetserver:7.14.0-latest
docker.io/voxpupuli/container-puppetserver:7.14.0-latest

adjusted file to this state (just add volumes and use already downloaded image):

services:
  puppet:
    image: voxpupuli/container-puppetserver:7.14.0-latest
    hostname: puppet
    environment:
      - PUPPETSERVER_HOSTNAME=puppet
      - PUPPETSERVER_PORT=8140
      - PUPPETDB_HOSTNAME=puppetdb
      - PUPPETDB_SSL_PORT=8081
      - USE_PUPPETDB=true
      - AUTOSIGN=true
      # For private repos, use git@github.com:user/repo.git and provide SSH keys
      # - R10K_REMOTE=https://github.com/betadots/demo-control-repo.git
    volumes:
      - puppetserver:/opt/puppetlabs/server/data/puppetserver
      - puppetserver-ssl:/etc/puppetlabs/puppet/ssl
      - puppetserver-ca:/etc/puppetlabs/puppetserver/ca
    restart: always
    ports:
      - 8140:8140

volumes:
  puppetserver:
  puppetserver-ssl:
  puppetserver-ca:

run:

root@puppet:~# docker-compose up -d --build
Creating network "root_default" with the default driver
Creating volume "root_puppetserver" with default driver
Creating volume "root_puppetserver-ssl" with default driver
Creating volume "root_puppetserver-ca" with default driver
Creating root_puppet_1 ... done

container is successfully running:

root@puppet:~# docker ps
CONTAINER ID   IMAGE                                            COMMAND                  CREATED              STATUS                        PORTS                                       NAMES
2251fc060859   voxpupuli/container-puppetserver:7.14.0-latest   "dumb-init /docker-e…"   About a minute ago   Up About a minute (healthy)   0.0.0.0:8140->8140/tcp, :::8140->8140/tcp   root_puppet_1

logs are healthy:

...
2024-01-23 08:31:53,970 INFO  [p.s.a.analytics-service] Puppet Server Update Service has successfully started and will run in the background
2024-01-23 08:31:53,988 INFO  [p.s.m.master-service] Puppet Server has successfully started and is now ready to handle requests
2024-01-23 08:31:55,629 INFO  [p.d.version-check] Newer version 8.4.0 is available! Visit https://puppet.com/docs/puppetserver/latest/release_notes.html for details.
127.0.0.1 - - - 23/Jan/2024:08:32:04 +0000 "GET /status/v1/simple HTTP/1.1" 200 7 127.0.0.1 127.0.0.1 8140 109

now, going on the client instance
it's almost exactly the same instance, just on x86_64.

install puppet agent:

root@ip-172-31-0-177:~# apt update
root@ip-172-31-0-177:~# apt install puppet
...
Setting up puppet (5.5.22-4ubuntu0.2) ...

create a minimal config for puppet agent:

root@ip-172-31-0-177:~# cat /etc/puppet/puppet.conf
[main]
server = puppet.vpn.rv.ua
certname = test1
runinterval = 30m

and run agent:

root@ip-172-31-0-177:~# puppet agent -t
Info: Creating a new SSL key for test1
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for test1
Info: Certificate Request fingerprint (SHA256): F4:13:68:AD:A2:91:64:12:91:A0:76:CA:10:40:C7:3A:2F:CD:87:89:49:DC:3B:15:72:54:42:5D:85:F5:F0:3F
Info: Caching certificate for test1
Error: request https://puppet.vpn.rv.ua:8140//puppet-ca/v1/certificate_revocation_list/ca failed: SSL_connect returned=1 errno=0 peeraddr=18.157.83.126:8140 state=error: certificate verify failed (unable to get issuer certificate)
Error: Could not request certificate: SSL_connect returned=1 errno=0 peeraddr=18.157.83.126:8140 state=error: certificate verify failed (unable to get issuer certificate): [unable to get issuer certificate for /CN=Puppet CA generated on puppet at 2024-01-23 08:31:14 \+0000]
Exiting; failed to retrieve certificate and waitforcert is disabled

server logs shown:

18.185.90.160 - - - 23/Jan/2024:08:39:30 +0000 "GET /puppet-ca/v1/certificate/ca?environment=production&fail_on_404=true HTTP/1.1" 200 2774 18.185.90.160 18.185.90.160 8140 34
18.185.90.160 - - - 23/Jan/2024:08:39:30 +0000 "GET /puppet-ca/v1/certificate/test1?environment=production& HTTP/1.1" 404 32 18.185.90.160 18.185.90.160 8140 4
18.185.90.160 - - - 23/Jan/2024:08:39:30 +0000 "GET /puppet-ca/v1/certificate_request/test1?environment=production& HTTP/1.1" 404 40 18.185.90.160 18.185.90.160 8140 5
2024-01-23 08:39:31,124 INFO  [p.p.certificate-authority] Signed certificate request for test1
18.185.90.160 - - - 23/Jan/2024:08:39:31 +0000 "PUT /puppet-ca/v1/certificate_request/test1?environment=production& HTTP/1.1" 200 0 18.185.90.160 18.185.90.160 8140 595
18.185.90.160 - - - 23/Jan/2024:08:39:31 +0000 "GET /puppet-ca/v1/certificate/test1?environment=production& HTTP/1.1" 200 1575 18.185.90.160 18.185.90.160 8140 6

the problem still persist :(
please, help me solve a problem.

I can provide access to both the server and the client.

[muscat]

[rwaffen]

so you have two vms? on one docker with the puppetserver running? and the second vm is an agent?
when you start the server with docker internal name puppet and vm external name puppet.vpn.rv.ua this will be a cert miss match. try to set DNS_ALT_NAMES.

services:
  puppet:
    image: voxpupuli/container-puppetserver:7.14.0-latest
    hostname: puppet
    environment:
      - PUPPETSERVER_HOSTNAME=puppet
      - DNS_ALT_NAMES=puppet.vpn.rv.ua
      ...

[rwaffen]

there for you might need to throw your ca away. you can do that by purging the puppetserver-ssl, puppetserver-ca volumes or just use local bind mounts if you arent used to volumes.

you might use full pathes or relativ pathes

    volumes:
      - ./puppetserver:/opt/puppetlabs/server/data/puppetserver
      - ./puppetserver-ssl:/etc/puppetlabs/puppet/ssl
      - /my/path/on/the/host/puppetserver-ca:/etc/puppetlabs/puppetserver/ca

[muscat]

so you have two vms? on one docker with the puppetserver running? and the second vm is an agent?

no. It is a two separate machines (EC2).
on first is running puppet-server as a docker container.
the second one - is a 'destination', which should be provisioned by the puppet server

when you start the server with docker internal name puppet and vm external name puppet.vpn.rv.ua this will be a cert miss match. try to set DNS_ALT_NAMES.
- PUPPETSERVER_HOSTNAME=puppet
- DNS_ALT_NAMES=puppet.vpn.rv.ua

ok, will check soon

[muscat]

bad news, everyone

I used the next docker-compose.yml file on the server (a virgin clean new EC2):

services:
  puppet:
    image: voxpupuli/container-puppetserver:7.14.0-latest
    hostname: puppet
    environment:
      - PUPPETSERVER_HOSTNAME=puppet
      - DNS_ALT_NAMES=puppet.vpn.rv.ua
      - PUPPETSERVER_PORT=8140
      - PUPPETDB_HOSTNAME=puppetdb
      - PUPPETDB_SSL_PORT=8081
      - USE_PUPPETDB=true
      - AUTOSIGN=true
      # For private repos, use git@github.com:user/repo.git and provide SSH keys
      # - R10K_REMOTE=https://github.com/betadots/demo-control-repo.git
    volumes:
      - ./data:/opt/puppetlabs/server/data/puppetserver
      - ./ssl:/etc/puppetlabs/puppet/ssl
      - ./ca:/etc/puppetlabs/puppetserver/ca
    restart: always
    ports:
      - 8140:8140

changes are in:
envs - add DNS_ALT_NAMES
volumes - point directs to local folders

maintenance tasks:
set hostname, install docker & docker-composer, create docker-compose.yml, ...

run:

root@ip-172-31-0-184:~# docker-compose up -d --build
Creating network "root_default" with the default driver
Pulling puppet (voxpupuli/container-puppetserver:7.14.0-latest)...
7.14.0-latest: Pulling from voxpupuli/container-puppetserver
ce9ebea987c2: Pull complete
...
a7120fa89f31: Pull complete
Digest: sha256:65a37305a1e5a54703224af59157c275c8b8e2f8c87a47361c17e7f7d00a83f1
Status: Downloaded newer image for voxpupuli/container-puppetserver:7.14.0-latest
Creating root_puppet_1 ... done

server starts as it should:

root@ip-172-31-0-184:~# docker logs root_puppet_1 --follow
...
System configuration values:
* HOSTNAME: 'puppet'
* hostname -f: 'puppet'
* PUPPETSERVER_HOSTNAME: 'puppet'
* PUPPETSERVER_PORT: '8140'
* Certname: 'puppet.pem'
* DNS_ALT_NAMES: 'puppet.vpn.rv.ua'
* SSLDIR: '/etc/puppetlabs/puppet/ssl'
CA Certificate:
subject=CN = "Puppet CA generated on puppet at 2024-01-23 17:26:03 +0000"
issuer=CN = Puppet Root CA: 0e0f3db1b15c83
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:TRUE
            X509v3 Key Usage: critical
                Certificate Sign, CRL Sign
            X509v3 Subject Key Identifier:
                7B:77:5F:3F:23:DD:75:FF:86:9E:F2:6D:D5:9E:30:77:7D:F1:1B:AB
            Netscape Comment:
                Puppet Server Internal Certificate
            X509v3 Authority Key Identifier:
                6C:62:FB:CE:C0:D3:A8:3E:BC:3C:F0:26:80:01:6F:D9:EC:F1:FA:9A
Certificate puppet.pem:
subject=CN = puppet
issuer=CN = "Puppet CA generated on puppet at 2024-01-23 17:26:03 +0000"
        X509v3 extensions:
            X509v3 Basic Constraints: critical
                CA:FALSE
            Netscape Comment:
                Puppet Server Internal Certificate
            X509v3 Authority Key Identifier:
                7B:77:5F:3F:23:DD:75:FF:86:9E:F2:6D:D5:9E:30:77:7D:F1:1B:AB
            X509v3 Extended Key Usage: critical
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Subject Key Identifier:
                EE:AA:7A:38:DE:CC:68:E1:80:42:4D:5B:D0:EC:B7:26:05:2D:61:C8
            1.3.6.1.4.1.34380.1.3.39:
                ..true
            X509v3 Subject Alternative Name:
                DNS:puppet, DNS:puppet.vpn.rv.ua

...

2024-01-23 17:26:33,734 INFO  [p.s.a.analytics-service] Puppet Server Update Service has successfully started and will run in the background
2024-01-23 17:26:33,759 INFO  [p.s.m.master-service] Puppet Server has successfully started and is now ready to handle requests
2024-01-23 17:26:33,866 WARN  [c.p.p.ShellUtils] Executed an external process which logged to STDERR: /opt/puppetlabs/puppet/bin/ruby: No such file or directory -- /opt/puppetlabs/server/data/puppetserver/dropsonde/bin/dropsonde (LoadError)
2024-01-23 17:26:33,883 WARN  [p.s.a.dropsonde] Failed to submit module metrics via Dropsonde. Error: /opt/puppetlabs/puppet/bin/ruby: No such file or directory -- /opt/puppetlabs/server/data/puppetserver/dropsonde/bin/dropsonde (LoadError)
2024-01-23 17:26:34,406 INFO  [p.d.version-check] Newer version 8.4.0 is available! Visit https://puppet.com/docs/puppetserver/latest/release_notes.html for details.

.
.
.
.
.

go to the client's side

start a fresh new ubuntu22 @ x86_64 EC2:

install puppet
create a config for puppet agent:

[main]
 server = puppet.vpn.rv.ua
 certname = test1
 runinterval = 30m

try to run:

ubuntu@ip-172-31-0-187:~$ sudo puppet agent -t
Info: Creating a new SSL key for test1
Info: Caching certificate for ca
Info: csr_attributes file loading from /etc/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for test1
Info: Certificate Request fingerprint (SHA256): 93:70:EE:57:7C:45:66:EE:71:D1:2B:AF:0B:9B:AA:D9:94:86:D1:2A:0F:96:CD:8D:FA:CF:0B:CA:15:75:EF:3F
Info: Caching certificate for test1
Error: request https://puppet.vpn.rv.ua:8140//puppet-ca/v1/certificate_revocation_list/ca failed: SSL_connect returned=1 errno=0 peeraddr=18.157.83.126:8140 state=error: certificate verify failed (unable to get issuer certificate)
Error: Could not request certificate: SSL_connect returned=1 errno=0 peeraddr=18.157.83.126:8140 state=error: certificate verify failed (unable to get issuer certificate): [unableto get issuer certificate for /CN=Puppet CA generated on puppet at 2024-01-23 17:26:03 \+0000]
Exiting; failed to retrieve certificate and waitforcert is disabled

server says:

3.67.19.230 - - - 23/Jan/2024:18:19:16 +0000 "GET /puppet-ca/v1/certificate/ca?environment=production&fail_on_404=true HTTP/1.1" 200 2773 3.67.19.230 3.67.19.230 8140 22
3.67.19.230 - - - 23/Jan/2024:18:19:16 +0000 "GET /puppet-ca/v1/certificate/test1?environment=production& HTTP/1.1" 404 32 3.67.19.230 3.67.19.230 8140 3
3.67.19.230 - - - 23/Jan/2024:18:19:16 +0000 "GET /puppet-ca/v1/certificate_request/test1?environment=production& HTTP/1.1" 404 40 3.67.19.230 3.67.19.230 8140 4
2024-01-23 18:19:17,076 INFO  [p.p.certificate-authority] Signed certificate request for test1
3.67.19.230 - - - 23/Jan/2024:18:19:17 +0000 "PUT /puppet-ca/v1/certificate_request/test1?environment=production& HTTP/1.1" 200 0 3.67.19.230 3.67.19.230 8140 523
3.67.19.230 - - - 23/Jan/2024:18:19:17 +0000 "GET /puppet-ca/v1/certificate/test1?environment=production& HTTP/1.1" 200 1574 3.67.19.230 3.67.19.230 8140 4
127.0.0.1 - - - 23/Jan/2024:18:19:17 +0000 "GET /status/v1/simple HTTP/1.1" 200 7 127.0.0.1 127.0.0.1 8140 8
127.0.0.1 - - - 23/Jan/2024:18:19:38 +0000 "GET /status/v1/simple HTTP/1.1" 200 7 127.0.0.1 127.0.0.1 8140 8

client successfully resolve the DNS:

ubuntu@ip-172-31-0-187:~$ host puppet.vpn.rv.ua
puppet.vpn.rv.ua has address 18.157.83.126

server is located on the desired IP:

root@puppet:~# curl 2ip.ua
 ip             : 18.157.83.126
 hostname       : ec2-18-157-83-126.eu-central-1.compute.amazonaws.com
 provider       : Amazon.com Inc.
 location       : Germany (DE), Frankfurt Am Main

.
.
.

folders 'data', 'ssl', 'ca' on the server contain fresh data.

root@puppet:~# ls -l data ssl ca
ca:
total 48
-rw-r----- 1 lxd docker 1978 Jan 23 17:26 ca_crl.pem
-rw-r----- 1 lxd docker 3899 Jan 23 17:26 ca_crt.pem
-rw-r----- 1 lxd docker 3243 Jan 23 17:26 ca_key.pem
-rw-r----- 1 lxd docker  800 Jan 23 17:26 ca_pub.pem
-rw-r----- 1 lxd docker 1978 Jan 23 17:26 infra_crl.pem
-rw-r----- 1 lxd docker    1 Jan 23 17:26 infra_inventory.txt
-rw-r----- 1 lxd docker    1 Jan 23 17:26 infra_serials
-rw-r----- 1 lxd docker  127 Jan 23 18:19 inventory.txt
drwxr-x--- 2 lxd docker 4096 Jan 23 18:19 requests
-rw-r----- 1 lxd docker 3243 Jan 23 17:26 root_key.pem
-rw-r--r-- 1 lxd docker    4 Jan 23 18:19 serial
drwxr-x--- 2 lxd docker 4096 Jan 23 18:19 signed

data:
total 44
drwxr-x---  2 lxd docker 4096 Jan 23 17:26 bucket
drwxr-x---  2 lxd docker 4096 Jan 23 17:26 facts.d
drwxr-x---  2 lxd docker 4096 Jan 23 17:26 lib
drwxr-x---  2 lxd docker 4096 Jan 23 17:26 locales
drwxr-x---  2 lxd docker 4096 Jan 23 17:26 preview
drwxr-x---  2 lxd docker 4096 Jan 23 17:26 reports
-rw-r-----  1 lxd docker    1 Jan 23 17:26 restartcounter
drwxr-x---  2 lxd docker 4096 Jan 23 17:26 server_data
drwxr-xr-t  2 lxd docker 4096 Jan 23 17:26 state
drwxr-xr-x 10 lxd docker 4096 Jan 23 17:25 vendored-jruby-gems
drwxr-x---  2 lxd docker 4096 Jan 23 17:26 yaml

ssl:
total 24
lrwxrwxrwx 1 lxd docker   31 Jan 23 17:26 ca -> /etc/puppetlabs/puppetserver/ca
drwxr-xr-x 2 lxd docker 4096 Jan 23 17:26 certificate_requests
drwxr-xr-x 2 lxd docker 4096 Jan 23 17:26 certs
-rw-r--r-- 1 lxd docker 1978 Jan 23 17:26 crl.pem
drwxr-x--- 2 lxd docker 4096 Jan 23 17:26 private
drwxr-x--- 2 lxd docker 4096 Jan 23 17:26 private_keys
drwxr-xr-x 2 lxd docker 4096 Jan 23 17:26 public_keys

i
really
need
help

[muscat]

weird situation: I set up a similar puppet server (as a docker container too) on the x86_64 architecture several months ago
and I didn't imagine that I could get stuck on that kind of problem

i used this config:
docker run -t
--log-opt max-size=10000m --log-opt max-file=1
--name puppet
--hostname puppet.domain.com
-p 8140:8140
-v /root/puppet/environments:/etc/puppetlabs/code/environments
-d
puppet/puppetserver:5.3.7

and everything working fine

[rwaffen]

i will try to reproduce this again. in all my tests while building the images i didn't had such problems

[muscat]

thanks a lot.
I can provide access to my EC2 for tests.

[rwaffen]

hmm maybe my comment was missleading. when you dont have/want a puppetdb you shouldn't configure it.

I tested your setup.

one vm (at gcp) as puppetserver: puppet.priv.rw.betadots.training
one vm (at gcp) as agent: worker-0.priv.rw.betadots.training
both Ubuntu 22.04.3 LTS. Internal network open for all traffic.

on puppet.priv.rw.betadots.training installed docker. and run this compose.yaml

services:
  puppet:
    image: voxpupuli/container-puppetserver:7.14.0-latest
    hostname: puppet
    environment:
      - PUPPETSERVER_HOSTNAME=puppet
      - DNS_ALT_NAMES=puppet.priv.rw.betadots.training
      - PUPPETSERVER_PORT=8140
      - USE_PUPPETDB=false
      - AUTOSIGN=true
    volumes:
      - ./data:/opt/puppetlabs/server/data/puppetserver
      - ./ssl:/etc/puppetlabs/puppet/ssl
      - ./ca:/etc/puppetlabs/puppetserver/ca
    restart: always
    ports:
      - 8140:8140

root@puppet:~# docker compose up
...
puppet-1  | 2024-01-24 16:48:52,058 INFO  [p.s.m.master-service] Puppet Server has successfully started and is now ready to handle requests

on worker-0.priv.rw.betadots.training I installed puppet agent. version 7.28.0.

root@worker-0:~# cat /etc/puppetlabs/puppet/puppet.conf
[agent]
server   = puppet.priv.rw.betadots.training
certname = test1

Agent Run

root@worker-0:~# puppet agent -t
Info: Creating a new RSA SSL key for test1
Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for test1
Info: Certificate Request fingerprint (SHA256): E2:A3:F0:0C:51:99:2A:AE:6D:6C:DE:A8:B8:F2:A6:04:2B:F1:D5:5B:F3:32:4F:D8:3E:35:16:DB:38:3B:E4:00
Info: Downloaded certificate for test1 from https://puppet.priv.rw.betadots.training:8140/puppet-ca/v1
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Notice: Requesting catalog from puppet.priv.rw.betadots.training:8140 (10.0.1.1)
Info: Caching catalog for test1
Info: Applying configuration version '1706115646'
Notice: Applied catalog in 0.01 seconds

PuppetServer Log

puppet-1  | 10.0.1.20 - - - 24/Jan/2024:17:00:10 +0000 "GET /puppet-ca/v1/certificate/ca HTTP/1.1" 200 2776 10.0.1.20 10.0.1.20 8140 23
puppet-1  | 10.0.1.20 - - - 24/Jan/2024:17:00:10 +0000 "GET /puppet-ca/v1/certificate_revocation_list/ca HTTP/1.1" 200 1464 10.0.1.20 10.0.1.20 8140 7
puppet-1  | 2024-01-24 17:00:14,628 INFO  [p.p.certificate-authority] Signed certificate request for test1
puppet-1  | 10.0.1.20 - - - 24/Jan/2024:17:00:14 +0000 "PUT /puppet-ca/v1/certificate_request/test1 HTTP/1.1" 200 0 10.0.1.20 10.0.1.20 8140 475
puppet-1  | 127.0.0.1 - - - 24/Jan/2024:17:00:29 +0000 "GET /status/v1/simple HTTP/1.1" 200 7 127.0.0.1 127.0.0.1 8140 10
puppet-1  | 10.0.1.20 - - - 24/Jan/2024:17:00:44 +0000 "GET /puppet-ca/v1/certificate/test1 HTTP/1.1" 200 1573 10.0.1.20 10.0.1.20 8140 6
puppet-1  | 10.0.1.20 - - - 24/Jan/2024:17:00:45 +0000 "GET /puppet/v3/file_metadatas/plugins?recurse=false&links=manage&checksum_type=sha256&source_permissions=ignore&environment=production HTTP/1.1" 200 198 10.0.1.20 10.0.1.20 8140 337
puppet-1  | 10.0.1.20 - - - 24/Jan/2024:17:00:45 +0000 "GET /puppet/v3/file_metadatas/pluginfacts?recurse=true&max_files=-1&ignore=.svn&ignore=CVS&ignore=.git&ignore=.hg&links=follow&checksum_type=sha256&source_permissions=use&environment=production HTTP/1.1" 200 197 10.0.1.20 10.0.1.20 8140 36
puppet-1  | 10.0.1.20 - - - 24/Jan/2024:17:00:45 +0000 "GET /puppet/v3/file_metadatas/plugins?recurse=true&max_files=-1&ignore=.svn&ignore=CVS&ignore=.git&ignore=.hg&links=follow&checksum_type=sha256&source_permissions=ignore&environment=production HTTP/1.1" 200 200 10.0.1.20 10.0.1.20 8140 26
puppet-1  | 2024-01-24 17:00:46,451 INFO  [puppetserver] Puppet Compiled catalog for test1 in environment production in 0.27 seconds
puppet-1  | 10.0.1.20 - - - 24/Jan/2024:17:00:46 +0000 "POST /puppet/v3/catalog/test1?environment=production HTTP/1.1" 200 307 10.0.1.20 10.0.1.20 8140 594
puppet-1  | 2024-01-24 17:00:46,636 INFO  [puppetserver] //test1/Puppet Using environment 'production'
puppet-1  | 2024-01-24 17:00:46,637 INFO  [puppetserver] //test1/Puppet Retrieving pluginfacts
puppet-1  | 2024-01-24 17:00:46,637 INFO  [puppetserver] //test1/Puppet Retrieving plugin
puppet-1  | 2024-01-24 17:00:46,638 INFO  [puppetserver] //test1/Puppet Requesting catalog from puppet.priv.rw.betadots.training:8140 (10.0.1.1)
puppet-1  | 2024-01-24 17:00:46,638 INFO  [puppetserver] //test1/Puppet Caching catalog for test1
puppet-1  | 2024-01-24 17:00:46,639 INFO  [puppetserver] //test1/Puppet Applying configuration version '1706115646'
puppet-1  | 2024-01-24 17:00:46,639 INFO  [puppetserver] //test1/Puppet Applied catalog in 0.01 seconds
puppet-1  | 10.0.1.20 - - - 24/Jan/2024:17:00:46 +0000 "PUT /puppet/v3/report/test1?environment=production HTTP/1.1" 200 7 10.0.1.20 10.0.1.20 8140 108

This is working for me. 🤔

your EC2 instances use public ips? there isnt any filter in between? local firewall?

your log line looks a bit odd

Error: request https://puppet.vpn.rv.ua:8140//puppet-ca/v1/certificate_revocation_list/ca failed: SSL_connect returned=1 errno=0 peeraddr=18.157.83.126:8140 state=error: certificate verify failed (unable to get issuer certificate)

why is there a double slash 🤔
puppet.vpn.rv.ua:8140**//**puppet-ca

but this shouldn't cause such problems. 🤔

[tuxmea]

Are you sure, that you can run a Puppet 5 agent against a Puppet 7 server?
Your old example also uses Puppet 5 server.
Please use a Puppet 7 agent (or at least a Puppet 6 agent)
Puppet documentation says puppet 7 (and 8) are compatible with Puppet agent 4 or newer.
But I doubt that this is true.