'You must reboot the server to apply kernel updates' persists after multiple reboots
My VPS is running v86.0.18 (RELEASE) on CENTOS 7.8 and I have a persistent "You must reboot the server to apply kernel updates" warning that doesn't go away after rebooting (several times). The latest "yum -y update" results in "No packages marked for update' and "whmapi1 system_needs_reboot" produces:
kernel:
boot_version: 3.10.0-1062.18.1.el7.x86_64
running_version: 3.10.0-862.14.4.el7.x86_64
needs_reboot: 1
metadata:
command: system_needs_reboot
reason: OK
result: 1
version: 1
"rpm -qa |grep kernel" results:
kernel-3.10.0-1062.18.1.el7.x86_64
kernel-3.10.0-957.27.2.el7.x86_64
kernel-headers-3.10.0-1127.el7.x86_64
kernel-devel-3.10.0-862.14.4.el7.x86_64
kernel-3.10.0-862.14.4.el7.x86_64
kernel-devel-3.10.0-1062.12.1.el7.x86_64
kernel-3.10.0-1062.12.1.el7.x86_64
kernel-devel-3.10.0-1062.18.1.el7.x86_64
kernel-3.10.0-1127.el7.x86_64
kernel-devel-3.10.0-1127.el7.x86_64
kernel-devel-3.10.0-1062.9.1.el7.x86_64
texlive-l3kernel-svn29409.SVN_4469-45.el7.noarch
and finally "uname -r" produces:
3.10.0-862.14.4.el7.x86_64
I'm not sure if this issue will just go away on its own or if I need to do something to make the warning disappear? Thanks!
-
Awesome, this tells us everything! - where to edit it: # DO NOT EDIT THIS FILE # # It is automatically generated by grub2-mkconfig using templates # from /etc/grub.d and settings from /etc/default/grub #
- It also tells us your default right now is 1:set default="1"
This explains why you're not booting into the new kernel - it should be 0 Give me a sec to find out where you should be editing/etc/grub.d/
- if you could provide me anls -lah
of that directory in the meantime it would be fantastic. I believe I can also tell you how to disable the submenus/advanced config :)0 -
Thanks! Here's the ls -lah of /etc/grub.d/ drwx------ 2 root root 4.0K Apr 28 01:32 . drwxr-xr-x 102 root root 24K May 1 13:30 .. -rwxr-xr-x 1 root root 8.5K Apr 3 10:24 00_header -rwxr-xr-x 1 root root 232 Apr 3 10:24 01_users -rwxr-xr-x 1 root root 11K Apr 3 10:24 10_linux -rwxr-xr-x 1 root root 11K Apr 3 10:24 20_linux_xen -rwxr-xr-x 1 root root 2.5K Apr 3 10:24 20_ppc_terminfo -rwxr-xr-x 1 root root 11K Apr 3 10:24 30_os-prober -rwxr-xr-x 1 root root 214 Apr 3 10:24 40_custom -rwxr-xr-x 1 root root 216 Apr 3 10:24 41_custom -rw-r--r-- 1 root root 483 Apr 3 10:24 README 0 -
So without /etc/default/grub (which the system I just spun up doesn't have either) you should be able to do the following: (some of this overlaps with what you've done already but the steps are important to have all in one place I believe) View the menuentries: awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg
VIew the default/what the system is set to boot into:grub2-editenv list
Compare the above and determine which menuentry starting at 0 -> X you want to boot into for example I have the following (I just installed the new kernel and didn't reboot in order to have this example)-bash-4.2# awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg 0 : CentOS Linux 7 Rescue a1772cbb489b4174a47ccf9deca6e297 (3.10.0-1127.el7.x86_64) 1 : CentOS Linux (3.10.0-1127.el7.x86_64) 7 (Core) 2 : CentOS Linux (3.10.0-1062.9.1.el7.x86_64) 7 (Core) 3 : CentOS Linux (0-rescue-87206729330b4fdda4e2599815d07128) 7 (Core)
Based on this menuentry 0 is a rescue so for my system I'd want to pick 1-bash-4.2# grub2-editenv list saved_entry=CentOS Linux (3.10.0-1127.el7.x86_64) 7 (Core)
Now to edit the default order (example is using my menuentry) :grub2-set-default 1
view the change:-bash-4.2# grub2-editenv list saved_entry=1
This will be different on your system since I believe the saved menuentry you want is 0 For the submenus to be disabled I'm struggling with a way to do this without/etc/default/grub
existing,grub2-mkconfig
does this when you have multiple kernels installed. Without the setting to disable in/etc/default/grub
I'm running into dead ends for how to manipulate it. I believe you need to edit/etc/grub.d/40_custom
but because I'm not 100% on that I don't want to recommend it.0 -
Mine already appears to be set to boot to 0 # awk -F\' '$1=="menuentry " {print i++ " : " $2}' /etc/grub2.cfg 0 : CentOS Linux (3.10.0-1127.el7.x86_64) 7 (Core) 1 : CentOS Linux 7 (Core) # grub2-editenv list saved_entry=0
Unfortunately, that doesn't appear to be happening.0 -
My assumption is in that case, something is happening when you're rebooting. Funny question but what's in /etc/fstab
? the output of the following would be useful:cat /etc/fstab0 -
/dev/sda1 / ext3 usrjquota=quota.user,jqfmt=vfsv0 0 1 (I took out extra spaces) 0 -
Ok, well that rules out that weird anomaly - I saw one case where this was happening due to /boot being commented out in /etc/fstab but since you have everything mounted on / that wouldn't be relevant. What I think is happening is grub2's failsafe booting where if it can't update to that kernel it tries one other. You're running a few versions behind even with 3.10.0-862.14.4.el7.x86_64 so this has been going on for a bit and every kernel installed is newer than the one you're running. You could reinstall the newest kernel: yum -y reinstall kernel-3.10.0-1127.el7.x86_64
But I'm not sure at this point that would help. You could try pointing to the next newer kernel: kernel-3.10.0-957.27.2.el7.x86_64 which is I believe within a submenu These articles are some of the things I've been looking through:0 -
# grubby --default-kernel /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 # grubby --default-index 1 (and 957.27.2 got removed earlier in this thread by a command suggested by GOT) 0 -
Wait a minute here though. It seems grubby is seeing entirely different settings. I forgot about the cleaning old packages my fault, there's a lot of info here. The menuentry/index it's loading is indeed 1 from your /boot/grub2/grub.cfg my test system updates this automatically and it seems that somewhere months ago yours stop synchronizing with the grub2 config? this may be interesting to see the output of: grubby --info=ALL
You can set the default index with:grubby --set-default-index=0
You can specify a kernel version with:grubby --set-default /boot/vmlinuz-3.10.0-1127.el7.x86_64
I'd be really curious to see if setting the kernel or setting the index with grubby resolves this.0 -
Here's the output of the command: # grubby --info=ALL index=0 kernel=/boot/vmlinuz-3.10.0-1127.el7.x86_64 args="ro LANG=en_US.UTF-8" root=/dev/sda1 initrd=/boot/initramfs-3.10.0-1127.el7.x86_64.img title=CentOS Linux (3.10.0-1127.el7.x86_64) 7 (Core) index=1 kernel=/boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 args="ro" root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e initrd=/boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img title=CentOS Linux 7 (Core) index=2 non linux entry index=3 kernel=/boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 args="ro" root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e initrd=/boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img title=CentOS Linux (3.10.0-1062.18.1.el7.x86_64) 7 (Core) index=4 kernel=/boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 args="ro single" root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e initrd=/boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img title=CentOS Linux (3.10.0-1062.18.1.el7.x86_64) 7 (Core) (recovery mode) index=5 kernel=/boot/vmlinuz-3.10.0-1062.12.1.el7.x86_64 args="ro" root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e initrd=/boot/initramfs-3.10.0-1062.12.1.el7.x86_64.img title=CentOS Linux (3.10.0-1062.12.1.el7.x86_64) 7 (Core) index=6 kernel=/boot/vmlinuz-3.10.0-1062.12.1.el7.x86_64 args="ro single" root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e initrd=/boot/initramfs-3.10.0-1062.12.1.el7.x86_64.img title=CentOS Linux (3.10.0-1062.12.1.el7.x86_64) 7 (Core) (recovery mode) index=7 kernel=/boot/vmlinuz-3.10.0-862.14.4.el7.x86_64 args="ro" root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e initrd=/boot/initrd-3.10.0-862.14.4.el7.x86_64.img title=CentOS Linux (3.10.0-862.14.4.el7.x86_64) 7 (Core) index=8 kernel=/boot/vmlinuz-3.10.0-862.14.4.el7.x86_64 args="ro single" root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e initrd=/boot/initrd-3.10.0-862.14.4.el7.x86_64.img title=CentOS Linux (3.10.0-862.14.4.el7.x86_64) 7 (Core) (recovery mode) index=9 non linux entry index=10 non linux entry0 -
scratch that this kernel version was released in September of 2018 - how long have you had this server? Are you aware if the kernel has ever actually updated? 0 -
So the info shows us at least that index 0 is indeed the kernel we want to get to but it's not being shown as the default index or kernel according to grubby 0 -
I've had the server a very long time, but it was migrated from ServInt to Leaseweb in October 2018. Are you suggesting I should run "grubby --set-default-index=0" and then reboot? 0 -
I've had the server a very long time, but it was migrated from ServInt to Leaseweb in October 2018.
That lines up exactly with the last time this kernel was updated just FYIAre you suggesting I should run "grubby --set-default-index=0" and then reboot?
I am suggesting you can try it but the way things are configured right now, I'm nervous to say that with confidence. If I were you, I would call my hosting provider or datacenter and initiate a discussion about this. I don't know what happened here but something is very wrong and your kernel isn't updating nor has it since it was migrated to this provider which has left you two years behind on updates.0 -
If I were you, I would call my hosting provider or datacenter and initiate a discussion about this. I don't know what happened here but something is very wrong and your kernel isn't updating nor has it since it was migrated to this provider which has left you two years behind on updates.
Got it, thanks. Before the acquisition of ServInt by Leaseweb, I never had to update the kernel myself, so it wasn't on my radar. Instead of doing anything particularly risky, I'll reach out to them and have them take a look at the situation. I like using WHM/cPanel, which is one reason I've not left for a host that has their own proprietary dashboard/panel system.0 -
Got it, thanks. Before the acquisition of ServInt by Leaseweb, I never had to update the kernel myself, so it wasn't on my radar. Instead of doing anything particularly risky, I'll reach out to them and have them take a look at the situation. I like using WHM/cPanel, which is one reason I've not left for a host that has their own proprietary dashboard/panel system.
I wish I could tell you with confidence to update that but really it's not something that is in our scope of support and our tech support team would not be able to help you with this or I would have directed you there, because we don't have console access to your server. Luckily you have console access you could potentially boot into safe mode but I'd feel a lot better if your provider took a look since only the one kernel shows up there.I like using WHM/cPanel, which is one reason I've not left for a host that has their own proprietary dashboard/panel system.
There are tons of hosts who offer cPanel & WHM though but we're glad you like using it :) Let me know what they say and if you try running the grubby command, I'm feeling pretty invested in this issue now and I really want to know how it plays out.0 -
Looks like I accidentally executed the "grubby --set-default-index=0" earlier and I've rebooted since then, so here are some updates: # grub2-editenv list saved_entry=0 # grubby --default-kernel /boot/vmlinuz-3.10.0-1127.el7.x86_64
But I still have the "You must reboot the server to apply kernel updates" showing in WHM after reboot, and this is in the "whmapi1 system_needs_reboot" output:boot_version: 3.10.0-1127.el7.x86_64 running_version: 3.10.0-862.14.4.el7.x86_64
I'll follow up with the host and will be sure to come back to update.0 -
Yea boot version is set to the correct kernel and you're still running the old kernel - but i sincerely feel like we made progress :highfive:? 0 -
The command has already been run, but I am looking forward to hearing the response from the datacenter @aztopdavid on how this is to be resolved! 0 -
I finally submitted a ticket to the provider and here's the response: [QUOTE]Your server had an issue with grub2 that wasn't picking up the new update. I have fixed these issues and you can reboot at your convenience and there server will come up on the latest kernel.
I rebooted and now it's running 3.10.0-1127.e17.x86_64 "# grubby --default-kernel" now returns "/boot/vmlinuz-3.10.0-1127.e17.x86_64" "# grubby --default-index" now returns "0" They weren't specific about the "issue with grub2" -- I could reopen the ticket and ask if that would be helpful.0 -
They weren't specific about the "issue with grub2" -- I could reopen the ticket and ask if that would be helpful.
Ahh that's a bummer, but it's quite alright, no need to do that. I am so happy to hear that they were able to get it resolved for you though :D0 -
Thanks to those who interacted with this thread with helpful information and suggestions! 0
Please sign in to leave a comment.
Comments
52 comments