Reinstall compromised SSHD via ssh

May 15, 2014 20:37

You should be able to safely "yum reinstall openssh-server" while logged in. Then I would log out, restart SSH via WHM, and change the root password via WHM as well. I'm only advising this because you stated you're moving accounts off of the server; obviously if sshd was tampered with you'll want to re-image or retire the system. Always good to check your local machines for any malware as well.

0

Infopro

May 16, 2014 02:40

I think I'd like to see more of the email, TBH.

0

sozotech

May 16, 2014 15:17

Here is what the email said. [QUOTE]Time: Thu May 15 12:00:51 2014 -0400 The following list of files have FAILED the md5sum comparison test. This means that the file has been changed in some way. This could be a result of an OS update or application upgrade. If the change is unexpected it should be investigated: /usr/bin/slogin: FAILED /usr/bin/ssh: FAILED /usr/sbin/sshd: FAILED

0

Infopro

May 16, 2014 16:13

I thought so. This email means your system was updated. ConfigServer/LFD as the title of that email should show, is alerting you to the fact that these files were changed in some way. If your system was freshly updated, manually or automatically, and you have no other reason to suspect compromise other than this email, you should be fine. Restarting CSF/LFD is suggested.

0

Infopro

May 16, 2014 16:15

Here's a similar email for an extreme example:

The following list of files have FAILED the md5sum comparison test. This means that the file has been changed in some way. This could be a result of an OS update or application upgrade. If the change is unexpected it should be investigated:

/usr/bin/addr2line: FAILED
/usr/bin/afs5log: FAILED
/usr/bin/ar: FAILED
/usr/bin/as: FAILED
/usr/bin/aulastlog: FAILED
/usr/bin/ausyscall: FAILED
/usr/bin/c++: FAILED
/usr/bin/cal: FAILED
/usr/bin/cancel: FAILED
/usr/bin/cancel.cups: FAILED
/usr/bin/cc: FAILED
/usr/bin/certutil: FAILED
/usr/bin/c++filt: FAILED
/usr/bin/chage: FAILED
/usr/bin/chfn: FAILED
/usr/bin/chrt: FAILED
/usr/bin/chsh: FAILED
/usr/bin/cmsutil: FAILED
/usr/bin/col: FAILED
/usr/bin/colcrt: FAILED
/usr/bin/colrm: FAILED
/usr/bin/column: FAILED
/usr/bin/cpp: FAILED
/usr/bin/crash: FAILED
/usr/bin/crlutil: FAILED
/usr/bin/crontab: FAILED
/usr/bin/cupstestdsc: FAILED
/usr/bin/cupstestppd: FAILED
/usr/bin/curl: FAILED
/usr/bin/cvs: FAILED
/usr/bin/cytune: FAILED
/usr/bin/ddate: FAILED
/usr/bin/dig: FAILED
/usr/bin/esd: FAILED
/usr/bin/esdcat: FAILED
/usr/bin/esdctl: FAILED
/usr/bin/esdfilt: FAILED
/usr/bin/esdloop: FAILED
/usr/bin/esdmon: FAILED
/usr/bin/esdplay: FAILED
/usr/bin/esdrec: FAILED
/usr/bin/esdsample: FAILED
/usr/bin/faillog: FAILED
/usr/bin/fastjar: FAILED
/usr/bin/fdformat: FAILED
/usr/bin/file: FAILED
/usr/bin/flock: FAILED
/usr/bin/floppy: FAILED
/usr/bin/g++: FAILED
/usr/bin/gcc: FAILED
/usr/bin/gcj-dbtool: FAILED
/usr/bin/gcov: FAILED
/usr/bin/gencat: FAILED
/usr/bin/getconf: FAILED
/usr/bin/getent: FAILED
/usr/bin/gethostip: FAILED
/usr/bin/getopt: FAILED
/usr/bin/ghostscript: FAILED
/usr/bin/gij: FAILED
/usr/bin/gjarsigner: FAILED
/usr/bin/gkeytool: FAILED
/usr/bin/gpasswd: FAILED
/usr/bin/gprof: FAILED
/usr/bin/grepjar: FAILED
/usr/bin/grmic: FAILED
/usr/bin/grmiregistry: FAILED
/usr/bin/gs: FAILED
/usr/bin/hexdump: FAILED
/usr/bin/host: FAILED
/usr/bin/i386-redhat-linux-c++: FAILED
/usr/bin/i386-redhat-linux-g++: FAILED
/usr/bin/i386-redhat-linux-gcc: FAILED
/usr/bin/iconv: FAILED
/usr/bin/ionice: FAILED
/usr/bin/ipcrm: FAILED
/usr/bin/ipcs: FAILED
/usr/bin/isc-config.sh: FAILED
/usr/bin/isosize: FAILED
/usr/bin/jv-convert: FAILED
/usr/bin/kill: FAILED
/usr/bin/ksh: FAILED
/usr/bin/lastlog: FAILED
/usr/bin/ld: FAILED
/usr/bin/lddlibc4: FAILED
/usr/bin/locale: FAILED
/usr/bin/localedef: FAILED
/usr/bin/logger: FAILED
/usr/bin/look: FAILED
/usr/bin/lp: FAILED
/usr/bin/lp.cups: FAILED
/usr/bin/lpoptions: FAILED
/usr/bin/lppasswd: FAILED
/usr/bin/lpq: FAILED
/usr/bin/lpq.cups: FAILED
/usr/bin/lpr: FAILED
/usr/bin/lpr.cups: FAILED
/usr/bin/lprm: FAILED
/usr/bin/lprm.cups: FAILED
/usr/bin/lpstat: FAILED
/usr/bin/lpstat.cups: FAILED
/usr/bin/mcookie: FAILED
/usr/bin/modutil: FAILED
/usr/bin/namei: FAILED
/usr/bin/newgrp: FAILED
/usr/bin/nm: FAILED
/usr/bin/nslookup: FAILED
/usr/bin/nsupdate: FAILED
/usr/bin/objcopy: FAILED
/usr/bin/objdump: FAILED
/usr/bin/openssl: FAILED
/usr/bin/pdffonts: FAILED
/usr/bin/pdfimages: FAILED
/usr/bin/pdfinfo: FAILED
/usr/bin/pdftohtml: FAILED
/usr/bin/pdftops: FAILED
/usr/bin/pdftotext: FAILED
/usr/bin/pk12util: FAILED
/usr/bin/pkcs11_eventmgr: FAILED
/usr/bin/pkcs11_inspect: FAILED
/usr/bin/pkcs11_setup: FAILED
/usr/bin/pklogin_finder: FAILED
/usr/bin/protoize: FAILED
/usr/bin/python: FAILED
/usr/bin/python2: FAILED
/usr/bin/python2.4: FAILED
/usr/bin/ranlib: FAILED
/usr/bin/readelf: FAILED
/usr/bin/rename: FAILED
/usr/bin/renice: FAILED
/usr/bin/rev: FAILED
/usr/bin/rngtest: FAILED
/usr/bin/rpcgen: FAILED
/usr/bin/rpm2cpio: FAILED
/usr/bin/rpmbuild: FAILED
/usr/bin/rpmdb: FAILED
/usr/bin/rpmquery: FAILED
/usr/bin/rpmsign: FAILED
/usr/bin/rpmverify: FAILED
/usr/bin/scp: FAILED
/usr/bin/script: FAILED
/usr/bin/setsid: FAILED
/usr/bin/setterm: FAILED
/usr/bin/sftp: FAILED
/usr/bin/sg: FAILED
/usr/bin/signtool: FAILED
/usr/bin/signver: FAILED
/usr/bin/size: FAILED
/usr/bin/slogin: FAILED
/usr/bin/smtpd.pyc: FAILED
/usr/bin/smtpd.pyo: FAILED
/usr/bin/sprof: FAILED
/usr/bin/ssh: FAILED
/usr/bin/ssh-add: FAILED
/usr/bin/ssh-agent: FAILED
/usr/bin/ssh-copy-id: FAILED
/usr/bin/ssh-keygen: FAILED
/usr/bin/ssh-keyscan: FAILED
/usr/bin/ssltap: FAILED
/usr/bin/strace: FAILED
/usr/bin/strings: FAILED
/usr/bin/strip: FAILED
/usr/bin/sudo: FAILED
/usr/bin/sudoedit: FAILED
/usr/bin/syslinux: FAILED
/usr/bin/tailf: FAILED
/usr/bin/ul: FAILED
/usr/bin/unprotoize: FAILED
/usr/bin/whereis: FAILED
/usr/bin/write: FAILED
/usr/bin/xmlcatalog: FAILED
/usr/bin/xmllint: FAILED
/usr/bin/ypcat: FAILED
/usr/bin/ypchfn: FAILED
/usr/bin/ypchsh: FAILED
/usr/bin/ypmatch: FAILED
/usr/bin/yppasswd: FAILED
/usr/bin/ypwhich: FAILED
/usr/sbin/accept: FAILED
/usr/sbin/adduser: FAILED
/usr/sbin/automount: FAILED
/usr/sbin/bind-chroot-admin: FAILED
/usr/sbin/build-locale-archive: FAILED
/usr/sbin/chpasswd: FAILED
/usr/sbin/crond: FAILED
/usr/sbin/cupsaddsmb: FAILED
/usr/sbin/cupsctl: FAILED
/usr/sbin/cupsd: FAILED
/usr/sbin/cupsdisable: FAILED
/usr/sbin/cupsenable: FAILED
/usr/sbin/cupsfilter: FAILED
/usr/sbin/dns-keygen: FAILED
/usr/sbin/dnssec-keygen: FAILED
/usr/sbin/dnssec-signzone: FAILED
/usr/sbin/exportfs: FAILED
/usr/sbin/firstboot: FAILED
/usr/sbin/fsadm: FAILED
/usr/sbin/glibc_post_upgrade.i686: FAILED
/usr/sbin/groupadd: FAILED
/usr/sbin/groupdel: FAILED
/usr/sbin/groupmod: FAILED
/usr/sbin/grpck: FAILED
/usr/sbin/grpconv: FAILED
/usr/sbin/grpunconv: FAILED
/usr/sbin/gss_clnt_send_err: FAILED
/usr/sbin/hwclock: FAILED
/usr/sbin/iconvconfig: FAILED
/usr/sbin/iconvconfig.i686: FAILED
/usr/sbin/kudzu: FAILED
/usr/sbin/lpadmin: FAILED
/usr/sbin/lpc: FAILED
/usr/sbin/lpc.cups: FAILED
/usr/sbin/lpinfo: FAILED
/usr/sbin/lpmove: FAILED
/usr/sbin/lsof: FAILED
/usr/sbin/lvchange: FAILED
/usr/sbin/lvconvert: FAILED
/usr/sbin/lvcreate: FAILED
/usr/sbin/lvdisplay: FAILED
/usr/sbin/lvextend: FAILED
/usr/sbin/lvm: FAILED
/usr/sbin/lvmchange: FAILED
/usr/sbin/lvmdiskscan: FAILED
/usr/sbin/lvmsadc: FAILED
/usr/sbin/lvmsar: FAILED
/usr/sbin/lvreduce: FAILED
/usr/sbin/lvremove: FAILED
/usr/sbin/lvrename: FAILED
/usr/sbin/lvresize: FAILED
/usr/sbin/lvs: FAILED
/usr/sbin/lvscan: FAILED
/usr/sbin/lwresd: FAILED
/usr/sbin/named: FAILED
/usr/sbin/named-checkconf: FAILED
/usr/sbin/named-checkzone: FAILED
/usr/sbin/ndc: FAILED
/usr/sbin/newusers: FAILED
/usr/sbin/nfsstat: FAILED
/usr/sbin/nhfsstone: FAILED
/usr/sbin/nscd: FAILED
/usr/sbin/pvchange: FAILED
/usr/sbin/pvck: FAILED
/usr/sbin/pvcreate: FAILED
/usr/sbin/pvdisplay: FAILED
/usr/sbin/pvmove: FAILED
/usr/sbin/pvremove: FAILED
/usr/sbin/pvresize: FAILED
/usr/sbin/pvs: FAILED
/usr/sbin/pvscan: FAILED
/usr/sbin/pwck: FAILED
/usr/sbin/pwconv: FAILED
/usr/sbin/pwunconv: FAILED
/usr/sbin/ramsize: FAILED
/usr/sbin/rdev: FAILED
/usr/sbin/readprofile: FAILED
/usr/sbin/reject: FAILED
/usr/sbin/rndc: FAILED
/usr/sbin/rndc-confgen: FAILED
/usr/sbin/rootflags: FAILED
/usr/sbin/rpc.gssd: FAILED
/usr/sbin/rpc.idmapd: FAILED
/usr/sbin/rpcinfo: FAILED
/usr/sbin/rpc.mountd: FAILED
/usr/sbin/rpc.nfsd: FAILED
/usr/sbin/rpc.svcgssd: FAILED
/usr/sbin/showmount: FAILED
/usr/sbin/sosreport: FAILED
/usr/sbin/sshd: FAILED
/usr/sbin/sysreport: FAILED
/usr/sbin/tunelp: FAILED
/usr/sbin/tzdata-update: FAILED
/usr/sbin/useradd: FAILED
/usr/sbin/userdel: FAILED
/usr/sbin/usermod: FAILED
/usr/sbin/usernetctl: FAILED
/usr/sbin/vgcfgbackup: FAILED
/usr/sbin/vgcfgrestore: FAILED
/usr/sbin/vgchange: FAILED
/usr/sbin/vgck: FAILED
/usr/sbin/vgconvert: FAILED
/usr/sbin/vgcreate: FAILED
/usr/sbin/vgdisplay: FAILED
/usr/sbin/vgexport: FAILED
/usr/sbin/vgextend: FAILED
/usr/sbin/vgimport: FAILED
/usr/sbin/vgmerge: FAILED
/usr/sbin/vgmknodes: FAILED
/usr/sbin/vgreduce: FAILED
/usr/sbin/vgremove: FAILED
/usr/sbin/vgrename: FAILED
/usr/sbin/vgs: FAILED
/usr/sbin/vgscan: FAILED
/usr/sbin/vgsplit: FAILED
/usr/sbin/vidmode: FAILED
/usr/sbin/vigr: FAILED
/usr/sbin/vipw: FAILED
/usr/sbin/visudo: FAILED
/usr/sbin/yppoll: FAILED
/usr/sbin/ypserv_test: FAILED
/usr/sbin/ypset: FAILED
/usr/sbin/yptest: FAILED
/usr/sbin/zdump: FAILED
/usr/sbin/zic: FAILED
/bin/arch: FAILED
/bin/dmesg: FAILED
/bin/dnsdomainname: FAILED
/bin/doexec: FAILED
/bin/domainname: FAILED
/bin/hostname: FAILED
/bin/ipcalc: FAILED
/bin/kill: FAILED
/bin/ksh: FAILED
/bin/ksh93: FAILED
/bin/logger: FAILED
/bin/login: FAILED
/bin/more: FAILED
/bin/mount: FAILED
/bin/netstat: FAILED
/bin/nisdomainname: FAILED
/bin/raw: FAILED
/bin/rpm: FAILED
/bin/taskset: FAILED
/bin/umount: FAILED
/bin/usleep: FAILED
/bin/ypdomainname: FAILED
/sbin/addpart: FAILED
/sbin/agetty: FAILED
/sbin/arp: FAILED
/sbin/audispd: FAILED
/sbin/auditctl: FAILED
/sbin/auditd: FAILED
/sbin/aureport: FAILED
/sbin/ausearch: FAILED
/sbin/autrace: FAILED
/sbin/blockdev: FAILED
/sbin/brcm_iscsiuio: FAILED open or read
/sbin/cciss_id: FAILED
/sbin/clock: FAILED
/sbin/consoletype: FAILED
/sbin/ctrlaltdel: FAILED
/sbin/delpart: FAILED
/sbin/dhclient: FAILED
/sbin/dmeventd: FAILED
/sbin/dmsetup: FAILED
/sbin/dmsetup.static: FAILED
/sbin/ether-wake: FAILED
/sbin/extlinux: FAILED
/sbin/fdisk: FAILED
/sbin/fsck.cramfs: FAILED
/sbin/fstab-decode: FAILED
/sbin/genhostid: FAILED
/sbin/getkey: FAILED
/sbin/grubby: FAILED
/sbin/hwclock: FAILED
/sbin/ifconfig: FAILED
/sbin/initlog: FAILED
/sbin/ip6tables: FAILED
/sbin/ip6tables-restore: FAILED
/sbin/ip6tables-save: FAILED
/sbin/ipmaddr: FAILED
/sbin/iptables: FAILED
/sbin/iptables-restore: FAILED
/sbin/iptables-save: FAILED
/sbin/iptunnel: FAILED
/sbin/iscsiadm: FAILED
/sbin/iscsid: FAILED
/sbin/iscsi-iname: FAILED
/sbin/iscsistart: FAILED
/sbin/kpartx: FAILED
/sbin/kpartx.static: FAILED
/sbin/kudzu: FAILED
/sbin/ldconfig: FAILED
/sbin/losetup: FAILED
/sbin/lspci: FAILED
/sbin/lvm: FAILED
/sbin/lvm.static: FAILED
/sbin/microcode_ctl: FAILED
/sbin/mii-diag: FAILED
/sbin/mii-tool: FAILED
/sbin/mkfs: FAILED
/sbin/mkfs.cramfs: FAILED
/sbin/mkinitrd: FAILED
/sbin/mkswap: FAILED
/sbin/mount.nfs: FAILED
/sbin/mount.nfs4: FAILED
/sbin/mpath_ctl: FAILED
/sbin/mpath_prio_alua: FAILED
/sbin/mpath_prio_alua.static: FAILED
/sbin/mpath_prio_emc: FAILED
/sbin/mpath_prio_emc.static: FAILED
/sbin/mpath_prio_hds_modular: FAILED
/sbin/mpath_prio_hds_modular.static: FAILED
/sbin/mpath_prio_hp_sw: FAILED
/sbin/mpath_prio_hp_sw.static: FAILED
/sbin/mpath_prio_intel: FAILED
/sbin/mpath_prio_intel.static: FAILED
/sbin/mpath_prio_netapp: FAILED
/sbin/mpath_prio_netapp.static: FAILED
/sbin/mpath_prio_ontap: FAILED
/sbin/mpath_prio_ontap.static: FAILED
/sbin/mpath_prio_rdac: FAILED
/sbin/mpath_prio_rdac.static: FAILED
/sbin/mpath_prio_tpc: FAILED
/sbin/mpath_prio_tpc.static: FAILED
/sbin/mpath_prio_weighted: FAILED
/sbin/mpath_prio_weighted.static: FAILED
/sbin/multipath: FAILED
/sbin/multipathd: FAILED
/sbin/multipath.static: FAILED
/sbin/nameif: FAILED
/sbin/nash: FAILED
/sbin/netplugd: FAILED
/sbin/netreport: FAILED
/sbin/nologin: FAILED
/sbin/parted: FAILED
/sbin/partprobe: FAILED
/sbin/partx: FAILED
/sbin/pivot_root: FAILED
/sbin/plipconfig: FAILED
/sbin/ppp-watch: FAILED
/sbin/pvscan: FAILED
/sbin/rngd: FAILED
/sbin/route: FAILED
/sbin/rpc.lockd: FAILED
/sbin/rpc.statd: FAILED
/sbin/setpci: FAILED
/sbin/sfdisk: FAILED
/sbin/slattach: FAILED
/sbin/sln: FAILED
/sbin/swapoff: FAILED
/sbin/swapon: FAILED
/sbin/umount.nfs: FAILED
/sbin/umount.nfs4: FAILED
/sbin/vgchange: FAILED
/sbin/vgscan: FAILED
/etc/init.d/crond: FAILED
/etc/init.d/ip6tables: FAILED
/etc/init.d/iptables: FAILED
/etc/init.d/iscsid: FAILED
/etc/init.d/lvm2-monitor: FAILED
/etc/init.d/microcode_ctl: FAILED
/etc/init.d/netconsole: FAILED
/etc/init.d/nfs: FAILED
/etc/init.d/nfslock: FAILED
/etc/init.d/rpcgssd: FAILED
/etc/init.d/rpcidmapd: FAILED
/etc/init.d/rpcsvcgssd: FAILED

0

sozotech

May 16, 2014 17:35

Well, I could not find a log of ssh/sshd being updated in /var/log/yum.log or anything in the previous mornings upcp log which made me suspect the compromise. Eric

0

quizknows

May 16, 2014 18:14

[quote="sozotech, post: 1646432">Well, I could not find a log of ssh/sshd being updated in /var/log/yum.log or anything in the previous mornings upcp log which made me suspect the compromise. Eric
As you should. If yum / upcp had updated the RPM then the CSF email is expected and wouldn't be cause for concern. However, if there's no record of sshd being updated by yum or upcp, then you're probably right to assume root-level compromise. If you haven't already reinstalled, try "rpmverify -v openssh-server". If you see a '5' next to /usr/sbin/sshd, then it's compromised. If it's just a row of dots, then the MD5 matches the RPM. Verify the RPM has a valid signature using "rpm -qi openssh-server" If it's signed, match the signature to another package like coreutils. If those match you're good. If they don't, or openssh-server is unsigned, then it's compromised even if it passes rpmverify. If you're NOT hacked, your verify and -qi should look like this:


[root@new ~]# rpmverify -v openssh-server
.........  c /etc/pam.d/ssh-keycat
S.5....T.  c /etc/pam.d/sshd
.........    /etc/rc.d/init.d/sshd
SM5....T.  c /etc/ssh/sshd_config
.........  c /etc/sysconfig/sshd
.........    /usr/libexec/openssh/sftp-server
.........    /usr/libexec/openssh/ssh-keycat
.........    /usr/sbin/.sshd.hmac
.........    /usr/sbin/sshd
.........    /usr/share/doc/openssh-server-5.3p1
.........  d /usr/share/doc/openssh-server-5.3p1/HOWTO.ssh-keycat
.........  d /usr/share/man/man5/moduli.5.gz
.........  d /usr/share/man/man5/sshd_config.5.gz
.........  d /usr/share/man/man8/sftp-server.8.gz
.........  d /usr/share/man/man8/sshd.8.gz
.........    /var/empty/sshd
[root@new ~]# rpm -qi openssh-server
Name        : openssh-server               Relocations: (not relocatable)
Version     : 5.3p1                             Vendor: CentOS
Release     : 94.el6                        Build Date: Fri 22 Nov 2013 05:40:05 PM EST
Install Date: Thu 15 May 2014 04:39:41 PM EDT      Build Host: c6b8.bsys.dev.centos.org
Group       : System Environment/Daemons    Source RPM: openssh-5.3p1-94.el6.src.rpm
Size        : 689757                           License: BSD
Signature   : RSA/SHA1, Sun 24 Nov 2013 02:32:56 PM EST, Key ID 0946fca2c105b9de
Packager    : CentOS BuildSystem 
URL         : http://www.openssh.com/portable.html
Summary     : An open source SSH server daemon
Description :
OpenSSH is a free version of SSH (Secure SHell), a program for logging
into and executing commands on a remote machine. This package contains
the secure shell daemon (sshd). The sshd daemon allows SSH clients to
securely connect to your SSH server.

0

Infopro

May 16, 2014 22:42

Excellent post, quizknows.

0

quizknows

May 16, 2014 22:57

[quote="Infopro, post: 1646542">Excellent post, quizknows.
Thanks :) Partly due to the numerous ebury variants, I have plenty of practice verifying openssh-server and keyutils-libs RPMs >_<

0

ThinIce

May 18, 2014 17:45

If I recall correctly, depending on your OS / whether it is set to run in cron, it's also worth checking your prelink log to see if this has modified things after an update to a library (i.e. glibc) - while rpm is prelink aware if memory serves, CSF isn't

0

Reinstall compromised SSHD via ssh

Comments

Didn't find what you were looking for?