[Suggestion] Elevate from Ubuntu 22.04 to 24.04 behavior on failure
Answered
Good day,
I once again tried my luck with the elevation script and once again it failed randomly...
The pre-checks were all green, then the CPanel mirror servers acted up. I had to manually clean up and update "apt" and restart on stage 2. Then it finally entered stage 3 and stayed silent.
At least, this time I thought I was kind of lucky as I was able to ping it. Last time, on another server, it got stuck in the kernel boot and had to recover via rescue console...
Anyhow, the server now is in a zombie state and the hoster for that server does not offer weekend services. So, no remote terminal access... What I actually figured is that the zombie state allows to send emails at least. Which is how I got the information that stage 3 has failed.
The error in question was unnamed, just that
"The do-release-upgrade script failed with the following error message:"
was written in the email.
My suggestion: if your elevate-cpanel script fails for unknown reasons, not covered by the prechecks in the scripts, could you at least make sure that the sshd service is up and running upon terminating the elevate-cpanel script?
I don't know if it is safe to reboot the machine to eventually get the shhd service up or if I shoot myself in the leg by attempting so and killing it fully. Hence, I will wait till tomorrow and see that I get a remote terminal access to fix whatever issue has arisen...
Thank you.
Kind regards,
P.S. I was kind of lucky to have had an unsupported sources list file in the apt configuration directory. I saw in the logs, that the system tried to elevate in the background since February and was blocked by that. Could you please not have elevation done as an automated task? As you can see this is a risky attempt.
-
Ok, let me add some more information as there is more broken in stage 3 than I was expecting.
With terminal access I found no logs for dist-upgrade issues. However, after some digging I found the issue. In my installation the system runs on mysql.. and for whatever reason, after entering Stage 3 the mysql-client package started causing conflicts and the package was set "on hold". That led the Ubuntu provided update script to terminate immediately. The short solution is "apt remove mysql-client" then I could continue the elevate process.
Now, the SSH issue is rather strange. I verified the SSH service running and listening. BUT, for some reason only local connections were allowed, despite the binding clearly showing me it listening on wildcards (0.0.0.0) and not localhost only. "sshd -T" also confirmed that. Disabling the iptables service didn't solve the issue either. Very strange... After elevation was done, SSH returned back to normal state. I would appreciate it if you can clarify what happens with the SSH service in stage 3.
Somehow the user "dovecot" and its group "dovecot" are not getting added as well, causing the next break in the upgrade process. Adding both, group and user, manually enabled the continuation of the elevation process.
Having the installation on a md0 (software raid) causes a failure when grub is about to be updated. Manual interaction required. Just calling "update-grub" did the trick.
0 -
Hey there! I've reached out to the ELevate team and I'll let you know what they have to say!
0 -
I spoke with the team and they confirmed that there are no changes in the script that affect SSH access. If the server is left in a state where it doesn't work properly SSH could be affected, but that isn't something the script specifically shuts off.
Local connections being the only allowed method gets removed in stage 4 of ELevate as that runs, so I wouldn't expect that to be a persistent issue either.
It's also worth noting that cPanel *never* attempts to run ELevate in an automated fashion, so I'm not sure what you were seeing when you mention that background process "since February" as ELevate has to be manually initiated by a user.
It might be best to create a ticket so the system can be examined directly. If we're able to reproduce any of these issues we'd be happy to have our developers take a look!
0 -
Thank you for the quick reply.
Local connections being the only allowed method gets removed in stage 4 of ELevate as that runs, so I wouldn't expect that to be a persistent issue either.
So, that fits my observation as stated above (local only connections). Isn't it possible to allow outside connections in stage 3 when it fails? Sure, not everyone has a hoster that only is reachable 8-6 from Mo till Fr, but it would reduce downtime for such cases and makes things quicker to fix.
It's also worth noting that cPanel *never* attempts to run ELevate in an automated fashion,
Then I might be misinterpreting the log? In the elevate-cpanel.log file, I have several entries since February and I never executed that script. Does the system do background checks for some (undocumented) reason? If that's also not the case, then I can attach it to a ticket if you like. Not sure if that brings some insight at all.
Also, did the ELevate team say something about the missing dovecot user and group? Actually, I do wonder why both were removed in the first place.
0 -
I'm being told it would be best to create a ticket since we've never seen this behavior on our side before, so I don't have any specific advice I can provide.
0 -
Ok, as you have requested, I have created a ticket. The number is 95993851.
0 -
Thanks for that - I'm following along on my end as well.
There may be a slight delay on this issue as we're prioritizing the recent security event, but we'll get on it as soon as we can.
0 -
Thank you. We got a quick resolution already. It is indeed a background check, to see if an elevation option is available, executed by another script which is called by the default cpanel cron jobs.
0 -
I'm glad we were able to clear that up!
0
Please sign in to leave a comment.
Comments
9 comments