Just in the past few days my GCE startup script has been failing when creating new instances. I believe it is because the apt-get upgrade command is causing the google-startup-scripts service to shutdown or restart. I added the export lines and --force-conf options to try and remedy the issue, but it still fails. If I remove the apt-get upgrade command from the script, everything completes successfully. However, I then have to login to the server for a final "upgrade". Any thoughts on how I can fix this?
It may be worth noting that when I run apt-get upgrade manually, it asks me 2 prompts: 1) do I want to overwrite the sshserver config file (I don't), and 2) restart the following services (list below)
OS Ubuntu 22.04.1 LTS (e2-standard-2 compute engine)
Startup Script Snippet
apt-get -y update
export DEBIAN_FRONTEND=noninteractive
export NEEDRESTART_MODE=l
apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" upgrade
Log Snippet
After unpacking the google-guest-agent, it appears to shutdown the script runner
google_metadata_script_runner startup-script-url: Preparing to unpack .../61-dbus-user-session_1.12.20-2ubuntu4.1_amd64.deb ...
google_metadata_script_runner startup-script-url: Unpacking dbus-user-session (1.12.20-2ubuntu4.1) over (1.12.20-2ubuntu4) ...
google_metadata_script_runner startup-script-url: Preparing to unpack .../62-google-compute-engine-oslogin_20220714.00-0ubuntu1~22.04.1_amd64.deb ...
google_metadata_script_runner startup-script-url: Unpacking google-compute-engine-oslogin (20220714.00-0ubuntu1~22.04.1) over (20210907.00-0ubuntu2) ...
google_metadata_script_runner startup-script-url: Preparing to unpack .../63-google-guest-agent_20220622.00-0ubuntu2~22.04.0_amd64.deb ...
systemd Stopping Google Compute Engine Shutdown Scripts...
systemd google-startup-scripts.service: Main process exited, code=killed, status=15/TERM
systemd google-startup-scripts.service: Failed with result 'signal'.
systemd google-startup-scripts.service: Unit process 1263 (startup-script-) remains running after unit stopped.
systemd google-startup-scripts.service: Unit process 1632 (apt-get) remains running after unit stopped.
systemd google-startup-scripts.service: Unit process 4500 (dpkg) remains running after unit stopped.
systemd google-startup-scripts.service: Unit process 4501 (sh) remains running after unit stopped.
systemd google-startup-scripts.service: Unit process 4502 (sh) remains running after unit stopped.
systemd google-startup-scripts.service: Unit process 4503 (dpkg-status) remains running after unit stopped.
systemd google-startup-scripts.service: Unit process 5377 (preinst) remains running after unit stopped.
systemd google-startup-scripts.service: Unit process 5382 (systemctl) remains running after unit stopped.
systemd Stopped Google Compute Engine Startup Scripts.
systemd google-startup-scripts.service: Consumed 25.932s CPU time.
Services asked to restart after running "apt-get upgrade" manually
[*] apache2.service
[*] chrony.service
[ ] dbus.service
[*] fail2ban.service
[ ] [email protected]
[*] google-cloud-ops-agent-diagnostics.service
[*] google-cloud-ops-agent-fluent-bit.service
[*] google-cloud-ops-agent-opentelemetry-collector.service
[*] google-osconfig-agent.service
[*] packagekit.service
[*] polkit.service
[*] [email protected]
[*] [email protected]
[*] systemd-udevd.service
[*] tomcat8.service
[ ] unattended-upgrades.service
[ ] [email protected]
Apparently, I had an instance template that referenced an older OS image (ARM instead of x86) which was incompatible with the newly created instance. After updating to a new x86 OS image, the issue resolved itself.