Cron Jobs and /usr/lib/systemd/systemd --user in Almalinux
Answered
Hello,
I have multiple websites with PHP cron jobs, on cron job run, the following command run too:
/usr/lib/systemd/systemd --user which is cauding server load spikes and the server becomes unstable.
(Screenshots attached)
It happens with noshell or jailshell, it is only gone when I give shell access to the users.
this started after migrating from CentOs to Almalinux.
and CSF started to catch those with alerts like:
[QUOTE]
Time: Mon Aug 14 12:28:13 2023 +0200
Account: xxxxx
Resource: Process Time
Exceeded: 3908 > 1800 (seconds)
Executable: /usr/lib/systemd/systemd
Command Line: /usr/lib/systemd/systemd --user
PID: 48066 (Parent PID:48066)
Killed: No
I am running PHP 7.4 with DSO and Apache prefork. I already checked those thread who have the same problem, but no fix helped so far. I would really appreciate any help with this. Thanks
I am running PHP 7.4 with DSO and Apache prefork. I already checked those thread who have the same problem, but no fix helped so far. I would really appreciate any help with this. Thanks
-
Hey there! Can you let me know specifically where those screenshots are from? If that is from WHM >> Process Manager, that's showing very little CPU usage and almost no Memory usage, so it would be odd if that alone was making the server unstable. The CSF notice is normal as it's just telling you the process has run longer than 1800 seconds, which is the default for that software. 0 -
Thanks for you answer, yes, the screenshot is from WHM >> Process Manager The numbers sometimes go as high as 70-80 I am not worried about CSF as I can whitelist those processes, I am worried about the spikes. here is another screenshot and this is the log from /var/log/messages which are logged for every website which has a cron job [QUOTE] Aug 15 16:56:14 server systemd[2311719]: Stopped target Default. Aug 15 16:56:14 server systemd[2311719]: Stopped target Basic System. Aug 15 16:56:14 server systemd[2311719]: Stopped target Timers. Aug 15 16:56:14 server systemd[2311719]: Stopped target Paths. Aug 15 16:56:14 server systemd[2311719]: Stopped target Sockets. Aug 15 16:56:14 server systemd[2311719]: Stopped Mark boot as successful after the user session has run 2 minutes. Aug 15 16:56:14 server systemd[2311719]: Closed D-Bus User Message Bus Socket. Aug 15 16:56:14 server systemd[2311719]: Reached target Shutdown. Aug 15 16:56:14 server systemd[2311719]: Started Exit the Session. Aug 15 16:56:14 server systemd[2311719]: Reached target Exit the Session. Aug 15 16:56:14 server systemd[1]: user@1046.service: Succeeded. Aug 15 16:56:14 server systemd[1]: Stopped User Manager for UID 1046. Aug 15 16:56:14 server systemd[1]: Stopping User runtime directory /run/user/1046... Aug 15 16:56:14 server systemd[1]: run-user-1046.mount: Succeeded. Aug 15 16:56:14 server systemd[1]: user-runtime-dir@1046.service: Succeeded. Aug 15 16:56:14 server systemd[1]: Stopped User runtime directory /run/user/1046. Aug 15 16:56:14 server systemd[1]: Removed slice User Slice of UID 1046. Aug 15 16:56:15 server systemd[1]: Stopping User Manager for UID 1016... Aug 15 16:56:15 server systemd[2311732]: Stopped target Default. Aug 15 16:56:15 server systemd[2311732]: Stopped target Basic System. Aug 15 16:56:15 server systemd[2311732]: Stopped target Sockets. Aug 15 16:56:15 server systemd[2311732]: Stopped target Timers. Aug 15 16:56:15 server systemd[2311732]: Stopped Mark boot as successful after the user session has run 2 minutes. Aug 15 16:56:15 server systemd[2311732]: Stopped target Paths. Aug 15 16:56:15 server systemd[2311732]: Closed D-Bus User Message Bus Socket. Aug 15 16:56:15 server systemd[2311732]: Reached target Shutdown. Aug 15 16:56:15 server systemd[2311732]: Started Exit the Session. Aug 15 16:56:15 server systemd[2311732]: Reached target Exit the Session. Aug 15 16:56:15 server systemd[1]: user@1016.service: Succeeded. Aug 15 16:56:15 server systemd[1]: Stopped User Manager for UID 1016. Aug 15 16:56:15 server systemd[1]: Stopping User runtime directory /run/user/1016... Aug 15 16:56:15 server systemd[1]: run-user-1016.mount: Succeeded. Aug 15 16:56:15 server systemd[1]: user-runtime-dir@1016.service: Succeeded. Aug 15 16:56:15 server systemd[1]: Stopped User runtime directory /run/user/1016. Aug 15 16:56:15 server systemd[1]: Removed slice User Slice of UID 1016. Aug 15 16:56:17 server systemd[1]: session-81270.scope: Succeeded. 0 -
Interesting - we may need to see this in action, so could you open a ticket so we can check the server directly? 0 -
Thanks cPRex, Ticket ID: 95106429 I thought the problem was with sd-pam, which also fired along with the cron jobs. but sd-pam are not taking that much process and I am more concerned about /usr/lib/systemd/systemd --user 0 -
Thanks - I'm following along with that ticket now. 0 -
Our team was able to provide the following details in the ticket: On the RHEL 8 family of operating systems (probably ubuntu too?) the pam_systemd module is enabled in /etc/pam.d/system-auth. As a result systemd spawns a user level process to manage services, crons, etc upon login, or any time a user cron is executed. On systems with a large number of users, this can amount to a staggering level of additional %sys or %si level CPU usage. This new user level process management is not required on cPanel servers. Systems running a graphical user environment where gdm is in use will not work with this disabled. This can be disabled by default by running the following as the root user: systemctl mask user@.service Then any existing user sessions would need to be stopped or restarted. To then enable this on a per user basis, use the following command, replacing the myuser portion with the desired cPanel username: ln -s /usr/lib/systemd/system/user@.service /etc/systemd/system/user@$(id -u myuser).service0 -
Thanks cPRex, I really appreciate your efforts along with cPanel team. I just want to say that this worked for me, in case someone has the same issue. The server went from a load average of 6 to 2! Thanks a lot 0 -
I'm glad to hear it!!! 0 -
Hi. I'm experiencing the same issue and have ended up in this thread. Is there any command to "restart" or safely kill those systemd --user processes without needing to restart the server?
Thanks,
Ignacio
0 -
I've tried killing those processes and then executing the recommended command:
# systemctl mask user@.service
Created symlink /etc/systemd/system/user@.service → /dev/null.
# ps axo user:30,pid,comm:100 | grep systemd | grep -v "root\|grep" | awk '{ print $2 }' | xargs killbut they keep coming back:
# ps aux | grep systemd | grep "\-\-user"
admin-i+ 3935965 0.1 0.0 89964 9896 ? Ss 14:09 0:00 /usr/lib/systemd/systemd --user
autolig+ 3962687 0.4 0.0 89888 9968 ? Ss 14:15 0:00 /usr/lib/systemd/systemd --user
cespotco 3968108 1.3 0.0 89968 10020 ? Ss 14:16 0:00 /usr/lib/systemd/systemd --user
tatitasc 3968135 1.3 0.0 89964 9840 ? Ss 14:16 0:00 /usr/lib/systemd/systemd --user0 -
imorandin - I don't believe they should be coming back if those processes have been disabled globally on the machine, so we may need to see a ticket on this.
0 -
Hi,
Sorry for the delay, busy week.
It appears that after a while, the processes cease to appear, so the workaround functions perfectly!
Thanks,
Ignacio
0 -
I'm glad to hear it!
0 -
Hello can i know if it has any side effect?
0 -
Hi JUST CAUSE: I implemented the workaround 2 months ago with no side effects.
0
Please sign in to leave a comment.
Comments
15 comments