Skip to main content

'You must reboot the server to apply kernel updates' persists after multiple reboots

Comments

54 comments

  • GOT
    My guess is you have an lxc based vm. Known issue. You can turn this notice off in tweak settings but we haven't found a fix. Seems endemic with lxc.
    0
  • aztopdavid
    I just ran "systemd-detect-virt" and the result was "kvm" so it doesn't look like my host is using lxc
    0
  • GOT
    Looking closer at your initial post, your system is not booting into the latest kernel, that is why you are getting the message. Latest is 1062.18.1 Whereas your running is 862.14.4 I would take a look here
    0
  • aztopdavid
    Thanks, I saw that in an older thread, but wasn't sure if that would be an advisable solution for me. Perhaps I should just ask the host if/when they plan on doing system maintenance involving rebooting the node?
    0
  • GOT
    Doesn't really involve rebooting the node. For some reason your system is not flagging new kernels to be utilized, which is normally the default behavior, but this does not require a node reboot to do so.
    0
  • aztopdavid
    OK, I'm working my way through the steps in "How to set default boot kernel on Linux" but it's not going all that smoothly. When I ran: # awk -F\' '$1=="menuentry " {print $2}' /etc/grub2.cfg I got: CentOS Linux (3.10.0-1127.el7.x86_64) 7 (Core) CentOS Linux 7 (Core) However, when I poked around in the grub2.cfg file, looking at the details of the "menuentry" values in the "/etc/grub.d/10_linux" section, I found these, which include eight different "submenu" choices for that generic "CentOS Linux 7" value from the output above: menuentry 'CentOS Linux (3.10.0-1127.el7.x86_64) 7 (Core)' linux16 /boot/vmlinuz-3.10.0-1127.el7.x86_64 root=/dev/sda1 ro LANG=en_US.UTF-8 initrd16 /boot/initramfs-3.10.0-1127.el7.x86_64.img menuentry 'CentOS Linux 7 (Core)' linux16 /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro initrd16 /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img submenu 'Advanced options for CentOS Linux 7 (Core)' menuentry 'CentOS Linux (3.10.0-1062.18.1.el7.x86_64) 7 (Core)' linux16 /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro initrd16 /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img menuentry 'CentOS Linux (3.10.0-1062.18.1.el7.x86_64) 7 (Core) (recovery mode)' linux16 /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro single initrd16 /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img menuentry 'CentOS Linux (3.10.0-1062.12.1.el7.x86_64) 7 (Core)' linux16 /boot/vmlinuz-3.10.0-1062.12.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro initrd16 /boot/initramfs-3.10.0-1062.12.1.el7.x86_64.img menuentry 'CentOS Linux (3.10.0-1062.12.1.el7.x86_64) 7 (Core) (recovery mode)' linux16 /boot/vmlinuz-3.10.0-1062.12.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro single initrd16 /boot/initramfs-3.10.0-1062.12.1.el7.x86_64.img menuentry 'CentOS Linux (3.10.0-957.27.2.el7.x86_64) 7 (Core)' linux16 /boot/vmlinuz-3.10.0-957.27.2.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro initrd16 /boot/initramfs-3.10.0-957.27.2.el7.x86_64.img menuentry 'CentOS Linux (3.10.0-957.27.2.el7.x86_64) 7 (Core) (recovery mode)' linux16 /boot/vmlinuz-3.10.0-957.27.2.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro single initrd16 /boot/initramfs-3.10.0-957.27.2.el7.x86_64.img menuentry 'CentOS Linux (3.10.0-862.14.4.el7.x86_64) 7 (Core)' linux16 /boot/vmlinuz-3.10.0-862.14.4.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro initrd16 /boot/initrd-3.10.0-862.14.4.el7.x86_64.img menuentry 'CentOS Linux (3.10.0-862.14.4.el7.x86_64) 7 (Core) (recovery mode)' linux16 /boot/vmlinuz-3.10.0-862.14.4.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro single initrd16 /boot/initrd-3.10.0-862.14.4.el7.x86_64.img
    So to my untrained eye, it appears that the first entry results in booting 1127, while the second option basically refers to the following three submenu choices: 1062.18.1 957.27.2 862.14.4 and as you mentioned before, there's a mismatch between the boot and the running version that's not currently being resolved by rebooting: boot_version: 3.10.0-1062.18.1.el7.x86_64 running_version: 3.10.0-862.14.4.el7.x86_64 But if I successfully apply the procedures mentioned in "How to set default boot kernel?" might that result in running 1127 instead of 1062.18.1? Also, "/etc/default/grub" (mentioned in the article) doesn't exist. I'm not sure the article covers a setup with all the different "submenu" choices present on my system -- it appears to deal with a configuration with discrete "menuentry" values to choose from.
    0
  • GOT
    1127 is the very newest kernel for Cent7 so no suprise there. You've got a lot of kernels installed. It may be less confusing for you if you run package-cleanup -y --oldkernels --count=3 Which will remove all but the latest kernels.
    0
  • aztopdavid
    That appears only to have removed 957.27.2
    0
  • GOT
    Does your host provide you a kvm console?
    0
  • aztopdavid
    They have what looks to be a proprietary cloud dashboard. I've just logged in there and am rebooting using that interface, instead of from inside WHM.
    0
  • aztopdavid
    So, that didn't do it, after rebooting in my host's dashboard and logging back into WHM, I still had the "You must reboot..." warning, so I also did a Graceful Reboot in WHM and it's still there. I suppose I could sign up for KernelCare.
    0
  • GOT
    Does your host provide a console where you can see the server rebooting as if you were in front of it, if so you could select the kernel it boots into. Output of this please: cat /etc/sysconfig/kernel
    0
  • GOT
    I suspect a few things, but the article should help you. Assuming you have grep GRUB_DEFAULT /etc/default/grub GRUB_DEFAULT=saved Then running grub2-set-default 0 Then run grub2-mkconfig -o /boot/grub2/grub.cfg Then reboot. If the grub_default setting above is not "saved" then change it before running the other commands.
    0
  • aztopdavid
    cat /etc/sysconfig/kernel = "No such file or directory" I also already mentioned that /etc/default/grub doesn't exist, which is partly why I'm stuck. There's a "console" button in the host interface and I'm running it, but it appears to give me the same view as my PuTTY session with su root
    0
  • GOT
    if you have the console open while its booting you might (might) see a centos boot menu that lets you pick your kernel. If you are missing those files something is wacky with your set up and I could not help you further without risking a non-bootable system.
    0
  • aztopdavid
    Yes, a non-bootable system is what I'm trying to avoid. Here's what I saw in the console during rebooting - note that there's only the one choice:
    0
  • GOT
    Yeah that makes more sense, the lernels are not getting installed correctly. I don't really know how to advise you here. You could ask your host it could be something they are doing on deployment for stability reasons. If you don't want to move servers, get kernelcare to keep the kernel patched and turn off the reboot notices in tweak settings.
    0
  • aztopdavid
    Thanks! I'm actually shopping for a new host -- I'd be happy for anyone's suggestions, as long as that doesn't violate the forum policies.
    0
  • GOT
    Others can. I cannot. :)
    0
  • GOT
    Beyond what's in my sig anyway, lol
    0
  • aztopdavid
    :-) (I'll take a look)
    0
  • cPanelLauren
    I usually send these resources to folks with this issue:
    0
  • aztopdavid
    Yes, thanks, you'll see that was provided earlier in the thread and I went through that very thoroughly, but it doesn't appear to be an option given some specifics of my server config. Please take a look at the details I provided earlier, as I got stuck.
    0
  • cPanelLauren
    Right, which is odd but none the less what's the output for the following? cat /etc/grub.conf
    stat /etc/grub.conf
    0
  • aztopdavid
    cat: /etc/grub.conf: No such file or directory stat: cannot stat "/etc/grub.conf": No such file or directory
    0
  • cPanelLauren
    Ok...usually that's a symlink to /boot/grub/grub.conf
    which is what I was really looking for - does that exist? If not what is the output of the following: ls -lah /boot/ |grep grub
    rpm -qa |grep grub
    0
  • aztopdavid
    Yes - I provided some fairly detailed excerpts of the conf file earlier in the thread -- the relevant "menuentry" values. drwxr-xr-x 2 root root 4.0K Oct 29 2018 grub drwx------ 2 root root 4.0K May 1 08:10 grub2 grub2-pc-modules-2.02-0.81.el7.centos.noarch grub2-pc-2.02-0.81.el7.centos.x86_64 grub2-tools-minimal-2.02-0.81.el7.centos.x86_64 grub2-common-2.02-0.81.el7.centos.noarch grub2-tools-extra-2.02-0.81.el7.centos.x86_64 grub2-2.02-0.81.el7.centos.x86_64 grubby-8.28-26.el7.x86_64 grub2-tools-2.02-0.81.el7.centos.x86_64
    0
  • cPanelLauren
    I don't want the menuentry, I want the whole file which is why I asked for it. Can you please provide what I am requesting, I did read the thread - I'm attempting to help here but I can't do it if you're not providing me with exactly what I am requesting. I have no access to your system and no access to one *LIKE* yours so it places anyone attempting to provide assistance in a precarious position.
    0
  • cPanelLauren
    One huge difference is the fact you have grub2-pc added: My test environment runs the following so it makes sense why I don't' have the same items as you and I'm running on KVM as well: [root@server default]# rpm -qa |grep grub grub-0.97-94.el6.x86_64 grubby-8.28-8.el7.x86_64
    So that helps guide the direction. Do you then have /etc/grub.d
    present?
    0
  • aztopdavid
    Yes, /etc/grub.d is present, and here are the contents of /boot/grub2/grub.cfg # # DO NOT EDIT THIS FILE # # It is automatically generated by grub2-mkconfig using templates # from /etc/grub.d and settings from /etc/default/grub # ### BEGIN /etc/grub.d/00_header ### set pager=1 if [ -s $prefix/grubenv ]; then load_env fi if [ "${next_entry}" ] ; then set default="${next_entry}" set next_entry= save_env next_entry set boot_once=true else set default="1" fi if [ x"${feature_menuentry_id}" = xy ]; then menuentry_id_option="--id" else menuentry_id_option="" fi export menuentry_id_option if [ "${prev_saved_entry}" ]; then set saved_entry="${prev_saved_entry}" save_env saved_entry set prev_saved_entry= save_env prev_saved_entry set boot_once=true fi function savedefault { if [ -z "${boot_once}" ]; then saved_entry="${chosen}" save_env saved_entry fi } function load_video { if [ x$feature_all_video_module = xy ]; then insmod all_video else insmod efi_gop insmod efi_uga insmod ieee1275_fb insmod vbe insmod vga insmod video_bochs insmod video_cirrus fi } if [ x$feature_default_font_path = xy ] ; then font=unicode else insmod part_msdos insmod ext2 set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 a44fb57f-8c52-7db3-f787-3f47c406178e else search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e fi font="/usr/share/grub/unicode.pf2" fi if loadfont $font ; then set gfxmode=auto load_video insmod gfxterm set locale_dir=$prefix/locale set lang=en_US insmod gettext fi terminal_output gfxterm if [ x$feature_timeout_style = xy ] ; then set timeout_style=menu set timeout=5 # Fallback normal timeout code in case the timeout_style feature is # unavailable. else set timeout=5 fi ### END /etc/grub.d/00_header ### ### BEGIN /etc/grub.d/01_users ### if [ -f ${prefix}/user.cfg ]; then source ${prefix}/user.cfg if [ -n "${GRUB2_PASSWORD}" ]; then set superusers="root" export superusers password_pbkdf2 root ${GRUB2_PASSWORD} fi fi ### END /etc/grub.d/01_users ### ### BEGIN /etc/grub.d/10_linux ### menuentry 'CentOS Linux (3.10.0-1127.el7.x86_64) 7 (Core)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-simple-a44fb57f-8c52-7db3-f787-3f47c406178e' { load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod ext2 set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 a44fb57f-8c52-7db3-f787-3f47c406178e else search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e fi linux16 /boot/vmlinuz-3.10.0-1127.el7.x86_64 root=/dev/sda1 ro LANG=en_US.UTF-8 initrd16 /boot/initramfs-3.10.0-1127.el7.x86_64.img } menuentry 'CentOS Linux 7 (Core)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-simple-a44fb57f-8c52-7db3-f787-3f47c406178e' { load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod ext2 set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 a44fb57f-8c52-7db3-f787-3f47c406178e else search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e fi linux16 /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro initrd16 /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img } submenu 'Advanced options for CentOS Linux 7 (Core)' $menuentry_id_option 'gnulinux-advanced-a44fb57f-8c52-7db3-f787-3f47c406178e' { menuentry 'CentOS Linux (3.10.0-1062.18.1.el7.x86_64) 7 (Core)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1062.18.1.el7.x86_64-advanced-a44fb57f-8c52-7db3-f787-3f47c406178e' { load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod ext2 set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 a44fb57f-8c52-7db3-f787-3f47c406178e else search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e fi linux16 /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro initrd16 /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img } menuentry 'CentOS Linux (3.10.0-1062.18.1.el7.x86_64) 7 (Core) (recovery mode)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1062.18.1.el7.x86_64-recovery-a44fb57f-8c52-7db3-f787-3f47c406178e' { load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod ext2 set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 a44fb57f-8c52-7db3-f787-3f47c406178e else search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e fi linux16 /boot/vmlinuz-3.10.0-1062.18.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro single initrd16 /boot/initramfs-3.10.0-1062.18.1.el7.x86_64.img } menuentry 'CentOS Linux (3.10.0-1062.12.1.el7.x86_64) 7 (Core)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1062.12.1.el7.x86_64-advanced-a44fb57f-8c52-7db3-f787-3f47c406178e' { load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod ext2 set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 a44fb57f-8c52-7db3-f787-3f47c406178e else search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e fi linux16 /boot/vmlinuz-3.10.0-1062.12.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro initrd16 /boot/initramfs-3.10.0-1062.12.1.el7.x86_64.img } menuentry 'CentOS Linux (3.10.0-1062.12.1.el7.x86_64) 7 (Core) (recovery mode)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-1062.12.1.el7.x86_64-recovery-a44fb57f-8c52-7db3-f787-3f47c406178e' { load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod ext2 set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 a44fb57f-8c52-7db3-f787-3f47c406178e else search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e fi linux16 /boot/vmlinuz-3.10.0-1062.12.1.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro single initrd16 /boot/initramfs-3.10.0-1062.12.1.el7.x86_64.img } menuentry 'CentOS Linux (3.10.0-862.14.4.el7.x86_64) 7 (Core)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-862.14.4.el7.x86_64-advanced-a44fb57f-8c52-7db3-f787-3f47c406178e' { load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod ext2 set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 a44fb57f-8c52-7db3-f787-3f47c406178e else search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e fi linux16 /boot/vmlinuz-3.10.0-862.14.4.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro initrd16 /boot/initrd-3.10.0-862.14.4.el7.x86_64.img } menuentry 'CentOS Linux (3.10.0-862.14.4.el7.x86_64) 7 (Core) (recovery mode)' --class rhel fedora --class gnu-linux --class gnu --class os --unrestricted $menuentry_id_option 'gnulinux-3.10.0-862.14.4.el7.x86_64-recovery-a44fb57f-8c52-7db3-f787-3f47c406178e' { load_video set gfxpayload=keep insmod gzio insmod part_msdos insmod ext2 set root='hd0,msdos1' if [ x$feature_platform_search_hint = xy ]; then search --no-floppy --fs-uuid --set=root --hint-bios=hd0,msdos1 --hint-efi=hd0,msdos1 --hint-baremetal=ahci0,msdos1 a44fb57f-8c52-7db3-f787-3f47c406178e else search --no-floppy --fs-uuid --set=root a44fb57f-8c52-7db3-f787-3f47c406178e fi linux16 /boot/vmlinuz-3.10.0-862.14.4.el7.x86_64 root=UUID=a44fb57f-8c52-7db3-f787-3f47c406178e ro single initrd16 /boot/initrd-3.10.0-862.14.4.el7.x86_64.img } } ### END /etc/grub.d/10_linux ### ### BEGIN /etc/grub.d/20_linux_xen ### ### END /etc/grub.d/20_linux_xen ### ### BEGIN /etc/grub.d/20_ppc_terminfo ### ### END /etc/grub.d/20_ppc_terminfo ### ### BEGIN /etc/grub.d/30_os-prober ### ### END /etc/grub.d/30_os-prober ### ### BEGIN /etc/grub.d/40_custom ### # This file provides an easy way to add custom menu entries. Simply type the # menu entries you want to add after this comment. Be careful not to change # the 'exec tail' line above. ### END /etc/grub.d/40_custom ### ### BEGIN /etc/grub.d/41_custom ### if [ -f ${config_directory}/custom.cfg ]; then source ${config_directory}/custom.cfg elif [ -z "${config_directory}" -a -f $prefix/custom.cfg ]; then source $prefix/custom.cfg; fi ### END /etc/grub.d/41_custom ###
    0

Please sign in to leave a comment.