This is to deploy the recent kolla-ansible openstack. The problem happens since last week. I retested it on a good environment successfully deployed on 7/14/2023 and got bootstrap-servers failed this time.

The environments use Ubuntu 22.04 and external ceph for cinder and glance backend storage. The error could be due to some recent changes on kolla-ansible openstack, ubuntu, or ceph. It affects compute nodes only.

Is there any way for a workaround? I am thinking maybe there is a way to maneuver the repo cache on the ubuntu 22.04 nodes. Maybe based on the ansible output below, I can get some insight on what exactly that ansible task was doing. I read the man page of apt-key(8) but have no idea what to act. I'd appreciate it if someone can teach me some tricks here.

I also need to find the community of kolla-ansible openstack and report the problem there. I'd appreciate it if someone can point to me the right place.

Here is the error (edited):

TASK [openstack.kolla.baremetal : Enable ceph apt repository] **********************************************************************fatal: [kao12.cec.edu]: FAILED! => {"changed": false, "msg": "Failed to update apt cache: E:The repository 'https://download.ceph.com/debian-pacific jammy Release' does not have a Release file., W:Updating from such a repository can't be done securely, and is therefore disabled by default., W:See apt-secure(8) manpage for repository creation and user configuration details., W:https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details., W:http://linux.dell.com/repo/community/openmanage/11000/jammy/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details."}

The previous ansible task was to updating /etc/apt/trusted.gpg (edited):

TASK [openstack.kolla.baremetal : Install ceph apt gpg key] *****************************************************************ok: [kao12.cec.edu]

The verbose error (edited)

<kao12.cec.edu> (1, b'\n{"failed": true, "msg": "Failed to update apt cache: W:https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details., E:The repository 'https://download.ceph.com/debian-pacific jammy Release' does not have a Release file., W:Updating from such a repository can't be done securely, and is therefore disabled by default., W:See apt-secure(8) manpage for repository creation and user configuration details., W:http://linux.dell.com/repo/community/openmanage/11000/jammy/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details.", "invocation": {"module_args": {"repo": "deb https://download.ceph.com/debian-pacific/ jammy main", "filename": "ceph", "state": "present", "update_cache": true, "update_cache_retries": 5, "update_cache_retry_max_delay": 12, "install_python_apt": true, "validate_certs": true, "mode": null, "codename": null}}}\n', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.3, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for \r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/kolla/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/kolla/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 228094\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n")<kao12.cec.edu> Failed to connect to the host via ssh: OpenSSH_8.9p1 Ubuntu-3ubuntu0.3, OpenSSL 3.0.2 15 Mar 2022debug1: Reading configuration data /etc/ssh/ssh_configdebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/.conf matched no filesdebug1: /etc/ssh/ssh_config line 21: Applying options for *debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/kolla/.ssh/known_hosts'debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/kolla/.ssh/known_hosts2'debug1: auto-mux: Trying existing masterdebug2: fd 3 setting O_NONBLOCKdebug2: mux_client_hello_exchange: master version 4debug3: mux_client_forwards: request forwardings: 0 local, 0 remotedebug3: mux_client_request_session: enteringdebug3: mux_client_request_alive: enteringdebug3: mux_client_request_alive: done pid = 228094debug3: mux_client_request_session: session request sentdebug1: mux_client_request_session: master session id: 2debug3: mux_client_read_packet: read header failed: Broken pipedebug2: Received exit status from master 1fatal: [kao12.cec.edu]: FAILED! => {"changed": false,"invocation": {"module_args": {"codename": null,"filename": "ceph","install_python_apt": true,"mode": null,"repo": "deb https://download.ceph.com/debian-pacific/ jammy main","state": "present","update_cache": true,"update_cache_retries": 5,"update_cache_retry_max_delay": 12,"validate_certs": true}},"msg": "Failed to update apt cache: W:https://download.docker.com/linux/ubuntu/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details., E:The repository 'https://download.ceph.com/debian-pacific jammy Release' does not have a Release file., W:Updating from such a repository can't be done securely, and is therefore disabled by default., W:See apt-secure(8) manpage for repository creation and user configuration details., W:http://linux.dell.com/repo/community/openmanage/11000/jammy/dists/jammy/InRelease: Key is stored in legacy trusted.gpg keyring (/etc/apt/trusted.gpg), see the DEPRECATION section in apt-key(8) for details."}