Skip to main content

MariaDB service down

rs200
Hi, i have an issue on a VPS with cPanel installed. Today i noticed that one of my app can't write session data on disk, so after some troubleshooting, i saw disk space full on 100%. I made the error to activate backup on WHM with too many retentions, so disk become full last night. I rebooted the VPS from WHM but some service were down (cpanelsolr, mariadb and another). To speed up, i deleted directly from SSH the older backups in order to free some space. After that, i rebooted again and now only mariadb is down. I can't start it neither from SSH. I got this errors # systemctl start mariadb Job for mariadb.service failed because the control process exited with error code. See "systemctl status mariadb.service" and "journalctl -xe" for details.
# systemctl -l status mariadb mariadb.service - MariaDB 10.1.41 database server Loaded: loaded (/usr/lib/systemd/system/mariadb.service; enabled; vendor preset: disabled) Drop-In: /etc/systemd/system/mariadb.service.d ??migrated-from-my.cnf-settings.conf, protecthome.conf Active: failed (Result: exit-code) since Fri 2019-10-11 12:53:32 CEST; 1min 37s ago Docs: man:mysqld(8) https://mariadb.com/kb/en/library/systemd/ Process: 7786 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE) Process: 7756 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= || VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS) Process: 7754 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS) Main PID: 7786 (code=exited, status=1/FAILURE) Status: "MariaDB server is down" Oct 11 12:53:29 server.gnnserver.com systemd[1]: Starting MariaDB 10.1.41 database server... Oct 11 12:53:29 server.gnnserver.com mysqld[7786]: 2019-10-11 12:53:29 139964404709632 [Note] /usr/sbin/mysqld (mysqld 10.1.41-MariaDB) starting as process 7786 ... Oct 11 12:53:32 server.gnnserver.com systemd[1]: mariadb.service: main process exited, code=exited, status=1/FAILURE Oct 11 12:53:32 server.gnnserver.com systemd[1]: Failed to start MariaDB 10.1.41 database server. Oct 11 12:53:32 server.gnnserver.com systemd[1]: Unit mariadb.service entered failed state. Oct 11 12:53:32 server.gnnserver.com systemd[1]: mariadb.service failed.
What have i do to fix the issue?

Comments

7 comments

Date Votes
  • GOT
    You probably need to look in the hostname.err log in /var/lib/mysql to really see why its failing, what you have included here does not actually tell us.
    0
  • rs200
    You probably need to look in the hostname.err log in /var/lib/mysql to really see why its failing, what you have included here does not actually tell us.

    Hi GOT, these are the last 200 lines of the error log 2019-10-11 13:46:19 140535658191104 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2019-10-11 13:46:19 140535658191104 [Note] InnoDB: Completed initialization of buffer pool 2019-10-11 13:46:19 140535658191104 [Note] InnoDB: Highest supported file format is Barracuda. 2019-10-11 13:46:19 140535658191104 [Note] InnoDB: 128 rollback segment(s) are active. 2019-10-11 13:46:19 140535658191104 [Note] InnoDB: Waiting for purge to start 2019-10-11 13:46:19 140535658191104 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.44-86.0 started; log sequence number 446457190 2019-10-11 13:46:19 140534869915392 [Note] InnoDB: Dumping buffer pool(s) not yet started 2019-10-11 13:46:19 140535658191104 [Note] Plugin 'FEEDBACK' is disabled. 2019-10-11 13:46:19 140535658191104 [Note] Recovering after a crash using tc.log 2019-10-11 13:46:19 140535658191104 [ERROR] Can't init tc log 2019-10-11 13:46:19 140535658191104 [ERROR] Aborting 2019-10-11 13:51:34 140710433106176 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB. 2019-10-11 13:51:34 140710433106176 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2019-10-11 13:51:34 140710433106176 [Note] InnoDB: The InnoDB memory heap is disabled 2019-10-11 13:51:34 140710433106176 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2019-10-11 13:51:34 140710433106176 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2019-10-11 13:51:34 140710433106176 [Note] InnoDB: Compressed tables use zlib 1.2.7 2019-10-11 13:51:34 140710433106176 [Note] InnoDB: Using Linux native AIO 2019-10-11 13:51:34 140710433106176 [Note] InnoDB: Using SSE crc32 instructions 2019-10-11 13:51:34 140710433106176 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2019-10-11 13:51:34 140710433106176 [Note] InnoDB: Completed initialization of buffer pool 2019-10-11 13:51:34 140710433106176 [Note] InnoDB: Highest supported file format is Barracuda. 2019-10-11 13:51:34 140710433106176 [Note] InnoDB: 128 rollback segment(s) are active. 2019-10-11 13:51:34 140710433106176 [Note] InnoDB: Waiting for purge to start 2019-10-11 13:51:34 140710433106176 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.44-86.0 started; log sequence number 446457200 2019-10-11 13:51:34 140710433106176 [Note] Plugin 'FEEDBACK' is disabled. 2019-10-11 13:51:34 140710433106176 [Note] Recovering after a crash using tc.log 2019-10-11 13:51:34 140710433106176 [ERROR] Can't init tc log 2019-10-11 13:51:34 140710433106176 [ERROR] Aborting 2019-10-11 13:56:36 139808954480896 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB. 2019-10-11 13:56:36 139808954480896 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2019-10-11 13:56:36 139808954480896 [Note] InnoDB: The InnoDB memory heap is disabled 2019-10-11 13:56:36 139808954480896 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2019-10-11 13:56:36 139808954480896 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2019-10-11 13:56:36 139808954480896 [Note] InnoDB: Compressed tables use zlib 1.2.7 2019-10-11 13:56:36 139808954480896 [Note] InnoDB: Using Linux native AIO 2019-10-11 13:56:36 139808954480896 [Note] InnoDB: Using SSE crc32 instructions 2019-10-11 13:56:36 139808954480896 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2019-10-11 13:56:36 139808954480896 [Note] InnoDB: Completed initialization of buffer pool 2019-10-11 13:56:36 139808954480896 [Note] InnoDB: Highest supported file format is Barracuda. 2019-10-11 13:56:36 139808954480896 [Note] InnoDB: 128 rollback segment(s) are active. 2019-10-11 13:56:36 139808954480896 [Note] InnoDB: Waiting for purge to start 2019-10-11 13:56:36 139808954480896 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.44-86.0 started; log sequence number 446457210 2019-10-11 13:56:36 139808164804352 [Note] InnoDB: Dumping buffer pool(s) not yet started 2019-10-11 13:56:36 139808954480896 [Note] Plugin 'FEEDBACK' is disabled. 2019-10-11 13:56:36 139808954480896 [Note] Recovering after a crash using tc.log 2019-10-11 13:56:36 139808954480896 [ERROR] Can't init tc log 2019-10-11 13:56:36 139808954480896 [ERROR] Aborting 2019-10-11 14:01:41 140146544408832 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB. 2019-10-11 14:01:41 140146544408832 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2019-10-11 14:01:41 140146544408832 [Note] InnoDB: The InnoDB memory heap is disabled 2019-10-11 14:01:41 140146544408832 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2019-10-11 14:01:41 140146544408832 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2019-10-11 14:01:41 140146544408832 [Note] InnoDB: Compressed tables use zlib 1.2.7 2019-10-11 14:01:41 140146544408832 [Note] InnoDB: Using Linux native AIO 2019-10-11 14:01:41 140146544408832 [Note] InnoDB: Using SSE crc32 instructions 2019-10-11 14:01:41 140146544408832 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2019-10-11 14:01:41 140146544408832 [Note] InnoDB: Completed initialization of buffer pool 2019-10-11 14:01:41 140146544408832 [Note] InnoDB: Highest supported file format is Barracuda. 2019-10-11 14:01:41 140146544408832 [Note] InnoDB: 128 rollback segment(s) are active. 2019-10-11 14:01:41 140146544408832 [Note] InnoDB: Waiting for purge to start 2019-10-11 14:01:41 140146544408832 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.44-86.0 started; log sequence number 446457220 2019-10-11 14:01:41 140145755944704 [Note] InnoDB: Dumping buffer pool(s) not yet started 2019-10-11 14:01:41 140146544408832 [Note] Plugin 'FEEDBACK' is disabled. 2019-10-11 14:01:41 140146544408832 [Note] Recovering after a crash using tc.log 2019-10-11 14:01:41 140146544408832 [ERROR] Can't init tc log 2019-10-11 14:01:41 140146544408832 [ERROR] Aborting 2019-10-11 14:07:51 140002881116416 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB. 2019-10-11 14:07:51 140002881116416 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2019-10-11 14:07:51 140002881116416 [Note] InnoDB: The InnoDB memory heap is disabled 2019-10-11 14:07:51 140002881116416 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2019-10-11 14:07:51 140002881116416 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2019-10-11 14:07:51 140002881116416 [Note] InnoDB: Compressed tables use zlib 1.2.7 2019-10-11 14:07:51 140002881116416 [Note] InnoDB: Using Linux native AIO 2019-10-11 14:07:51 140002881116416 [Note] InnoDB: Using SSE crc32 instructions 2019-10-11 14:07:51 140002881116416 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2019-10-11 14:07:51 140002881116416 [Note] InnoDB: Completed initialization of buffer pool 2019-10-11 14:07:51 140002881116416 [Note] InnoDB: Highest supported file format is Barracuda. 2019-10-11 14:07:51 140002881116416 [Note] InnoDB: 128 rollback segment(s) are active. 2019-10-11 14:07:51 140002881116416 [Note] InnoDB: Waiting for purge to start 2019-10-11 14:07:51 140002881116416 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.44-86.0 started; log sequence number 446457230 2019-10-11 14:07:51 140002092640000 [Note] InnoDB: Dumping buffer pool(s) not yet started 2019-10-11 14:07:51 140002881116416 [Note] Plugin 'FEEDBACK' is disabled. 2019-10-11 14:07:51 140002881116416 [Note] Recovering after a crash using tc.log 2019-10-11 14:07:51 140002881116416 [ERROR] Can't init tc log 2019-10-11 14:07:51 140002881116416 [ERROR] Aborting 2019-10-11 14:14:00 140437970098432 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB. 2019-10-11 14:14:00 140437970098432 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2019-10-11 14:14:00 140437970098432 [Note] InnoDB: The InnoDB memory heap is disabled 2019-10-11 14:14:00 140437970098432 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2019-10-11 14:14:00 140437970098432 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2019-10-11 14:14:00 140437970098432 [Note] InnoDB: Compressed tables use zlib 1.2.7 2019-10-11 14:14:00 140437970098432 [Note] InnoDB: Using Linux native AIO 2019-10-11 14:14:00 140437970098432 [Note] InnoDB: Using SSE crc32 instructions 2019-10-11 14:14:00 140437970098432 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2019-10-11 14:14:00 140437970098432 [Note] InnoDB: Completed initialization of buffer pool 2019-10-11 14:14:00 140437970098432 [Note] InnoDB: Highest supported file format is Barracuda. 2019-10-11 14:14:00 140437970098432 [Note] InnoDB: 128 rollback segment(s) are active. 2019-10-11 14:14:00 140437970098432 [Note] InnoDB: Waiting for purge to start 2019-10-11 14:14:00 140437970098432 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.44-86.0 started; log sequence number 446457240 2019-10-11 14:14:01 140437176186624 [Note] InnoDB: Dumping buffer pool(s) not yet started 2019-10-11 14:14:01 140437970098432 [Note] Plugin 'FEEDBACK' is disabled. 2019-10-11 14:14:01 140437970098432 [Note] Recovering after a crash using tc.log 2019-10-11 14:14:01 140437970098432 [ERROR] Can't init tc log 2019-10-11 14:14:01 140437970098432 [ERROR] Aborting 2019-10-11 14:17:20 140679455901952 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB. 2019-10-11 14:17:20 140679455901952 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2019-10-11 14:17:20 140679455901952 [Note] InnoDB: The InnoDB memory heap is disabled 2019-10-11 14:17:20 140679455901952 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2019-10-11 14:17:20 140679455901952 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2019-10-11 14:17:20 140679455901952 [Note] InnoDB: Compressed tables use zlib 1.2.7 2019-10-11 14:17:20 140679455901952 [Note] InnoDB: Using Linux native AIO 2019-10-11 14:17:20 140679455901952 [Note] InnoDB: Using SSE crc32 instructions 2019-10-11 14:17:20 140679455901952 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2019-10-11 14:17:20 140679455901952 [Note] InnoDB: Completed initialization of buffer pool 2019-10-11 14:17:20 140679455901952 [Note] InnoDB: Highest supported file format is Barracuda. 2019-10-11 14:17:20 140679455901952 [Note] InnoDB: 128 rollback segment(s) are active. 2019-10-11 14:17:20 140679455901952 [Note] InnoDB: Waiting for purge to start 2019-10-11 14:17:20 140679455901952 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.44-86.0 started; log sequence number 446457250 2019-10-11 14:17:20 140678667433728 [Note] InnoDB: Dumping buffer pool(s) not yet started 2019-10-11 14:17:20 140679455901952 [Note] Plugin 'FEEDBACK' is disabled. 2019-10-11 14:17:20 140679455901952 [Note] Recovering after a crash using tc.log 2019-10-11 14:17:20 140679455901952 [ERROR] Can't init tc log 2019-10-11 14:17:20 140679455901952 [ERROR] Aborting 2019-10-11 14:19:21 140531384215808 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB. 2019-10-11 14:19:21 140531384215808 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2019-10-11 14:19:21 140531384215808 [Note] InnoDB: The InnoDB memory heap is disabled 2019-10-11 14:19:21 140531384215808 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2019-10-11 14:19:21 140531384215808 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2019-10-11 14:19:21 140531384215808 [Note] InnoDB: Compressed tables use zlib 1.2.7 2019-10-11 14:19:21 140531384215808 [Note] InnoDB: Using Linux native AIO 2019-10-11 14:19:21 140531384215808 [Note] InnoDB: Using SSE crc32 instructions 2019-10-11 14:19:21 140531384215808 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2019-10-11 14:19:21 140531384215808 [Note] InnoDB: Completed initialization of buffer pool 2019-10-11 14:19:21 140531384215808 [Note] InnoDB: Highest supported file format is Barracuda. 2019-10-11 14:19:21 140531384215808 [Note] InnoDB: 128 rollback segment(s) are active. 2019-10-11 14:19:21 140531384215808 [Note] InnoDB: Waiting for purge to start 2019-10-11 14:19:21 140531384215808 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.44-86.0 started; log sequence number 446457260 2019-10-11 14:19:21 140530604304128 [Note] InnoDB: Dumping buffer pool(s) not yet started 2019-10-11 14:19:21 140531384215808 [Note] Plugin 'FEEDBACK' is disabled. 2019-10-11 14:19:21 140531384215808 [Note] Recovering after a crash using tc.log 2019-10-11 14:19:21 140531384215808 [ERROR] Can't init tc log 2019-10-11 14:19:21 140531384215808 [ERROR] Aborting 2019-10-11 14:24:23 140606924716288 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB. 2019-10-11 14:24:23 140606924716288 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2019-10-11 14:24:23 140606924716288 [Note] InnoDB: The InnoDB memory heap is disabled 2019-10-11 14:24:23 140606924716288 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2019-10-11 14:24:23 140606924716288 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2019-10-11 14:24:23 140606924716288 [Note] InnoDB: Compressed tables use zlib 1.2.7 2019-10-11 14:24:23 140606924716288 [Note] InnoDB: Using Linux native AIO 2019-10-11 14:24:23 140606924716288 [Note] InnoDB: Using SSE crc32 instructions 2019-10-11 14:24:23 140606924716288 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2019-10-11 14:24:23 140606924716288 [Note] InnoDB: Completed initialization of buffer pool 2019-10-11 14:24:23 140606924716288 [Note] InnoDB: Highest supported file format is Barracuda. 2019-10-11 14:24:23 140606924716288 [Note] InnoDB: 128 rollback segment(s) are active. 2019-10-11 14:24:23 140606924716288 [Note] InnoDB: Waiting for purge to start 2019-10-11 14:24:23 140606924716288 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.44-86.0 started; log sequence number 446457270 2019-10-11 14:24:23 140606139528960 [Note] InnoDB: Dumping buffer pool(s) not yet started 2019-10-11 14:24:23 140606924716288 [Note] Plugin 'FEEDBACK' is disabled. 2019-10-11 14:24:23 140606924716288 [Note] Recovering after a crash using tc.log 2019-10-11 14:24:23 140606924716288 [ERROR] Can't init tc log 2019-10-11 14:24:23 140606924716288 [ERROR] Aborting 2019-10-11 14:29:27 140643549071616 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB. 2019-10-11 14:29:27 140643549071616 [Note] InnoDB: Using mutexes to ref count buffer pool pages 2019-10-11 14:29:27 140643549071616 [Note] InnoDB: The InnoDB memory heap is disabled 2019-10-11 14:29:27 140643549071616 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins 2019-10-11 14:29:27 140643549071616 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier 2019-10-11 14:29:27 140643549071616 [Note] InnoDB: Compressed tables use zlib 1.2.7 2019-10-11 14:29:27 140643549071616 [Note] InnoDB: Using Linux native AIO 2019-10-11 14:29:27 140643549071616 [Note] InnoDB: Using SSE crc32 instructions 2019-10-11 14:29:27 140643549071616 [Note] InnoDB: Initializing buffer pool, size = 128.0M 2019-10-11 14:29:27 140643549071616 [Note] InnoDB: Completed initialization of buffer pool 2019-10-11 14:29:27 140643549071616 [Note] InnoDB: Highest supported file format is Barracuda. 2019-10-11 14:29:27 140643549071616 [Note] InnoDB: 128 rollback segment(s) are active. 2019-10-11 14:29:27 140643549071616 [Note] InnoDB: Waiting for purge to start 2019-10-11 14:29:27 140643549071616 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.44-86.0 started; log sequence number 446457280 2019-10-11 14:29:27 140642755802880 [Note] InnoDB: Dumping buffer pool(s) not yet started 2019-10-11 14:29:27 140643549071616 [Note] Plugin 'FEEDBACK' is disabled. 2019-10-11 14:29:27 140643549071616 [Note] Recovering after a crash using tc.log 2019-10-11 14:29:27 140643549071616 [ERROR] Can't init tc log 2019-10-11 14:29:27 140643549071616 [ERROR] Aborting
    0
  • GOT
    This looks like your failure 2019-10-11 13:56:36 139808954480896 [Note] Recovering after a crash using tc.log 2019-10-11 13:56:36 139808954480896 [ERROR] Can't init tc log 2019-10-11 13:56:36 139808954480896 [ERROR] Aborting Basically innodb needs to go into recovery because the disk got full and you likely have corrupted data. First make sure that your partition that holds /var is not full. If it is not full see if this file exists and mv it to a backup: mv /var/lib/mysql/tc.log /var/lib/mysql/tc_bkp.log then try starting mariadb again. But you gotta have a reasonable amount of free space on /var and /tmp
    0
  • rs200
    Sorry but should I remove that file and start mariadb? I have about 11 GB of free disk space. WHM shows me "/" and "/var/tmp" as partitions.
    0
  • rs200
    Sorry again i'm not so expert with linux commands and i made confusion :) I renamed the tc.log file into tc_bkp.log with mv command and then restart mariadb. Now it works fine! Thank you very much GOT :)
    0
  • GOT
    Sure thing glad it worked!
    0
  • cPanelLauren
    I'm glad to see your issue was able to be resolved! Thank you @GOT
    0

Please sign in to leave a comment.

Didn't find what you were looking for?

New post