Unusual load MySQL
I tried the mysqlmymonlite.sh to gather the info but apparently it was stuck at (see below).
Anyways, i tried to fetch most info that may help you identify the problem. The problem and overview: 1. Mysql load was normal at 7%-8% historically. 2. Suddenly for the past one week, i see it consistently at 25%, server load too has increased hence 3. Since then it has kept increasing incrementally to now 32%. 4. No major codebase changes. 5. Standard Apache 2.4 installation with Php 5.4 + Node + Redis + Mongo. 6. 2 Operational Sites, with average combine traffic of 15K pageviews daily. 7. Mysql Cache was turned off as over the months i found that it was generating more overhead than providing any help. 8. Queries not-using-indexes has remained high forever (pardon, programmatic handicap) tuning-primer.sh
mysqltuner.pl
mysql config
Load Averages sar -p
hardware info
memory info
I just want to know is there any particular reason for sudden increase in mysql load? Thank you guys.
Running make install
F/FD/FDALY/Test-Tester-0.109.tar.gz is just needed temporarily during building or testing. Do you want to install it permanently? [yes] Anyways, i tried to fetch most info that may help you identify the problem. The problem and overview: 1. Mysql load was normal at 7%-8% historically. 2. Suddenly for the past one week, i see it consistently at 25%, server load too has increased hence 3. Since then it has kept increasing incrementally to now 32%. 4. No major codebase changes. 5. Standard Apache 2.4 installation with Php 5.4 + Node + Redis + Mongo. 6. 2 Operational Sites, with average combine traffic of 15K pageviews daily. 7. Mysql Cache was turned off as over the months i found that it was generating more overhead than providing any help. 8. Queries not-using-indexes has remained high forever (pardon, programmatic handicap) tuning-primer.sh
-- MYSQL PERFORMANCE TUNING PRIMER --
- By: Matthew Montgomery -
MySQL Version 5.5.36-cll x86_64
Uptime = 13 days 18 hrs 56 min 12 sec
Avg. qps = 196
Total Questions = 233818121
Threads Connected = 5
Server has been running for over 48hrs.
It should be safe to follow these recommendations
To find out more information on how each of these
runtime variables effects performance visit:
http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html
Visit http://www.mysql.com/products/enterprise/advisors.html
for info about MySQL's Enterprise Monitoring and Advisory Service
SLOW QUERIES
The slow query log is NOT enabled.
Current long_query_time = 1.000000 sec.
You have 101 out of 233818159 that take longer than 1.000000 sec. to complete
Your long_query_time seems to be fine
BINARY UPDATE LOG
The binary update log is NOT enabled.
You will not be able to do point in time recovery
See http://dev.mysql.com/doc/refman/5.5/en/point-in-time-recovery.html
WORKER THREADS
Current thread_cache_size = 16
Current threads_cached = 12
Current threads_per_sec = 0
Historic threads_per_sec = 0
Your thread_cache_size is fine
MAX CONNECTIONS
Current max_connections = 151
Current threads_connected = 5
Historic max_used_connections = 29
The number of used connections is 19% of the configured maximum.
Your max_connections variable seems to be fine.
INNODB STATUS
Current InnoDB index space = 13 M
Current InnoDB data space = 558 M
Current InnoDB buffer pool free = 4 %
Current innodb_buffer_pool_size = 560 M
Depending on how much space your innodb indexes take up it may be safe
to increase this value to up to 2 / 3 of total system memory
MEMORY USAGE
Max Memory Ever Allocated : 1021 M
Configured Max Per-thread Buffers : 2.10 G
Configured Max Global Buffers : 608 M
Configured Max Memory Limit : 2.69 G
Physical Memory : 23.59 G
Max memory limit seem to be within acceptable norms
KEY BUFFER
Current MyISAM index space = 104 M
Current key_buffer_size = 32 M
Key cache miss rate is 1 : 305208
Key buffer free ratio = 40 %
Your key_buffer_size seems to be too high.
Perhaps you can use these resources elsewhere
QUERY CACHE
Query cache is supported but not enabled
Perhaps you should set the query_cache_size
SORT OPERATIONS
Current sort_buffer_size = 2 M
Current read_rnd_buffer_size = 8 M
Sort buffer seems to be fine
JOINS
Current join_buffer_size = 2.00 M
You have had 53101252 queries where a join could not use an index properly
You have had 347 joins without keys that check for key usage after each row
You should enable "log-queries-not-using-indexes"
Then look for non indexed joins in the slow query log.
If you are unable to optimize your queries you may want to increase your
join_buffer_size to accommodate larger joins in one pass.
Note! This script will still suggest raising the join_buffer_size when
ANY joins not using indexes are found.
OPEN FILES LIMIT
Current open_files_limit = 32768 files
The open_files_limit should typically be set to at least 2x-3x
that of table_cache if you have heavy MyISAM usage.
Your open_files_limit value seems to be fine
TABLE CACHE
Current table_open_cache = 10240 tables
Current table_definition_cache = 10240 tables
You have a total of 1491 tables
You have 1819 open tables.
The table_cache value seems to be fine
TEMP TABLES
Current max_heap_table_size = 32 M
Current tmp_table_size = 32 M
Of 53364610 temp tables, 33% were created on disk
Perhaps you should increase your tmp_table_size and/or max_heap_table_size
to reduce the number of disk-based temporary tables
Note! BLOB and TEXT columns are not allow in memory tables.
If you are using these columns raising these values might not impact your
ratio of on disk temp tables.
TABLE SCANS
Current read_buffer_size = 2 M
Current table scan ratio = 699 : 1
read_buffer_size seems to be fine
TABLE LOCKING
Current Lock Wait ratio = 1 : 3632
You may benefit from selective use of InnoDB.
If you have long running SELECT's against MyISAM tables and perform
frequent updates consider setting 'low_priority_updates=1'
If you have a high concurrency of inserts on Dynamic row-length tables
consider setting 'concurrent_insert=ALWAYS'.
mysqltuner.pl
root@some-network [~/scripts]# perl mysqltuner*
>> MySQLTuner 1.2.0 - Major Hayden
>> Bug reports, feature requests, and downloads at http://mysqltuner.com/
>> Run with '--help' for additional options and output filtering
-------- General Statistics --------------------------------------------------
[--] Skipped version check for MySQLTuner script
[OK] Currently running supported MySQL version 5.5.36-cll
[OK] Operating on 64-bit architecture
-------- Storage Engine Statistics -------------------------------------------
[--] Status: +Archive -BDB -Federated +InnoDB -ISAM -NDBCluster
[--] Data in MyISAM tables: 311M (Tables: 1113)
[--] Data in InnoDB tables: 558M (Tables: 324)
[--] Data in PERFORMANCE_SCHEMA tables: 0B (Tables: 17)
[--] Data in MEMORY tables: 14M (Tables: 13)
[!!] Total fragmented tables: 74
-------- Security Recommendations -------------------------------------------
[OK] All database users have passwords assigned
-------- Performance Metrics -------------------------------------------------
[--] Up for: 13d 18h 57m 24s (233M q [196.269 qps], 11M conn, TX: 4220B, RX: 117B)
[--] Reads / Writes: 96% / 4%
[--] Total buffers: 640.0M global + 14.2M per thread (151 max threads)
[OK] Maximum possible memory usage: 2.7G (11% of installed RAM)
[OK] Slow queries: 0% (101/233M)
[OK] Highest usage of available connections: 19% (29/151)
[OK] Key buffer size / total MyISAM indexes: 32.0M/104.6M
[OK] Key buffer hit rate: 100.0% (6B cached / 21K reads)
[!!] Query cache is disabled
[OK] Sorts requiring temporary tables: 0% (6K temp sorts / 66M sorts)
[!!] Joins performed without indexes: 53107745
[!!] Temporary tables created on disk: 33% (27M on disk / 80M total)
[OK] Thread cache hit rate: 99% (339 created / 11M connections)
[!!] Table cache hit rate: 8% (1K open / 21K opened)
[OK] Open file limit used: 7% (2K/32K)
[OK] Table locks acquired immediately: 99% (424M immediate / 424M locks)
[OK] InnoDB data size / buffer pool: 558.4M/560.0M
-------- Recommendations -----------------------------------------------------
General recommendations:
Run OPTIMIZE TABLE to defragment tables for better performance
Enable the slow query log to troubleshoot bad queries
Adjust your join queries to always utilize indexes
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries without LIMIT clauses
Increase table_cache gradually to avoid file descriptor limits
Variables to adjust:
query_cache_size (>= 8M)
join_buffer_size (> 2.0M, or always use indexes with joins)
tmp_table_size (> 32M)
max_heap_table_size (> 32M)
table_cache (> 10240)
mysql config
[mysqld]
bind-address=127.0.0.1
innodb_file_per_table=1
innodb_buffer_pool_size=560M
key_buffer_size=32M
join_buffer_size=2M
sort_buffer_size=2M
read_buffer_size=2M
read_rnd_buffer_size=8M
myisam_sort_buffer_size=32M
table_open_cache=10240
table_definition_cache=10240
thread_cache_size=16
tmp_table_size=32M
max_heap_table_size=32M
long_query_time=1
open_files_limit=32768
Load Averages sar -p
root@some-network [~/scripts]# sar -p
Linux 3.10.9-xxxx-grs-ipv6-64 (some.hostname.net) Thursday 20 February 2014 _x86_64_ (8 CPU)
12:00:01 IST CPU %user %nice %system %iowait %steal %idle
12:10:01 IST all 2.20 0.00 0.35 0.42 0.00 97.03
12:20:01 IST all 2.34 0.00 0.37 0.33 0.00 96.95
12:30:01 IST all 2.15 0.00 0.34 0.36 0.00 97.15
12:40:01 IST all 1.93 0.05 0.39 0.39 0.00 97.24
12:50:01 IST all 2.26 0.00 0.47 0.38 0.00 96.89
01:00:01 IST all 1.89 0.00 0.34 0.38 0.00 97.40
01:10:01 IST all 1.71 0.00 0.31 0.33 0.00 97.65
01:20:01 IST all 2.14 0.00 0.36 0.41 0.00 97.09
01:30:01 IST all 1.71 0.01 0.27 0.53 0.00 97.49
01:40:01 IST all 1.82 0.00 0.28 0.38 0.00 97.51
01:50:01 IST all 1.88 0.00 0.31 0.36 0.00 97.44
02:00:01 IST all 1.66 0.00 0.25 0.35 0.00 97.75
02:10:01 IST all 1.46 0.00 0.22 0.35 0.00 97.97
02:20:01 IST all 1.94 0.01 0.38 0.37 0.00 97.31
02:30:01 IST all 1.55 0.00 0.22 0.33 0.00 97.91
02:40:01 IST all 1.67 0.00 0.25 0.31 0.00 97.77
02:50:01 IST all 1.78 0.04 0.31 0.37 0.00 97.50
03:00:01 IST all 4.30 0.00 1.19 0.42 0.00 94.09
03:10:01 IST all 1.79 0.01 0.28 0.32 0.00 97.61
03:20:01 IST all 1.54 0.00 0.20 0.33 0.00 97.92
03:30:01 IST all 1.87 0.00 0.29 0.31 0.00 97.53
03:40:01 IST all 1.92 0.01 0.30 0.36 0.00 97.41
03:50:01 IST all 2.45 0.01 0.44 0.44 0.00 96.67
04:00:01 IST all 1.67 0.00 0.31 0.46 0.00 97.56
04:10:01 IST all 1.79 0.00 0.30 0.39 0.00 97.52
04:20:01 IST all 1.97 0.01 0.29 0.37 0.00 97.36
04:30:01 IST all 1.54 0.00 0.24 0.35 0.00 97.87
04:40:01 IST all 1.74 0.04 0.29 0.33 0.00 97.60
04:50:01 IST all 1.63 0.04 0.24 0.33 0.00 97.75
05:00:01 IST all 1.59 0.00 0.22 0.31 0.00 97.87
05:10:01 IST all 1.87 0.01 0.27 0.34 0.00 97.51
05:20:01 IST all 1.56 0.00 0.25 0.31 0.00 97.87
05:30:01 IST all 1.46 0.00 0.22 0.31 0.00 98.02
05:40:01 IST all 1.56 0.00 0.25 0.30 0.00 97.89
05:50:01 IST all 1.47 0.00 0.22 0.27 0.00 98.04
06:00:01 IST all 1.70 0.01 0.25 0.30 0.00 97.74
06:10:01 IST all 1.65 0.00 0.27 0.31 0.00 97.77
06:20:01 IST all 1.44 0.00 0.22 0.30 0.00 98.04
06:30:01 IST all 1.82 0.00 0.30 0.28 0.00 97.60
06:40:01 IST all 1.61 0.00 0.25 0.26 0.00 97.88
06:50:01 IST all 1.83 0.04 0.37 0.27 0.00 97.49
07:00:01 IST all 1.59 0.00 0.25 0.32 0.00 97.83
07:00:01 IST CPU %user %nice %system %iowait %steal %idle
07:10:01 IST all 1.64 0.00 0.26 0.33 0.00 97.77
07:20:01 IST all 1.67 0.00 0.31 0.30 0.00 97.72
07:30:01 IST all 1.63 0.00 0.34 0.30 0.00 97.73
07:40:01 IST all 1.36 0.01 0.24 0.29 0.00 98.11
07:50:01 IST all 1.99 0.00 0.35 0.35 0.00 97.31
08:00:01 IST all 2.58 0.00 0.61 0.36 0.00 96.44
08:10:01 IST all 2.49 0.00 0.53 0.35 0.00 96.63
08:20:01 IST all 2.90 0.00 0.73 0.35 0.00 96.02
08:30:01 IST all 2.33 0.01 0.42 0.34 0.00 96.90
08:40:01 IST all 3.06 0.04 0.75 0.36 0.00 95.79
08:50:01 IST all 2.43 0.00 0.40 0.36 0.00 96.80
09:00:01 IST all 3.08 0.00 0.65 0.36 0.00 95.90
09:10:01 IST all 2.96 0.00 0.58 0.41 0.00 96.04
09:20:01 IST all 3.28 0.01 0.62 0.39 0.00 95.70
09:30:01 IST all 3.11 0.00 0.73 0.38 0.00 95.79
09:40:01 IST all 2.40 0.00 0.42 0.45 0.00 96.73
09:50:01 IST all 2.51 0.00 0.45 0.35 0.00 96.69
10:00:01 IST all 2.26 0.00 0.39 0.39 0.00 96.96
10:10:01 IST all 2.10 0.01 0.34 0.38 0.00 97.17
10:20:01 IST all 2.08 0.00 0.39 0.43 0.00 97.10
10:30:01 IST all 2.70 0.00 0.51 0.35 0.00 96.44
10:40:01 IST all 3.09 0.04 0.57 0.38 0.00 95.91
10:50:01 IST all 3.05 0.00 0.58 0.37 0.00 96.00
11:00:01 IST all 2.91 0.01 0.50 0.37 0.00 96.21
11:10:01 IST all 3.22 0.00 0.70 0.44 0.00 95.64
11:20:01 IST all 3.27 0.00 0.59 0.42 0.00 95.70
11:30:01 IST all 2.94 0.00 0.50 0.39 0.00 96.17
11:40:01 IST all 3.10 0.00 0.55 0.38 0.00 95.96
11:50:01 IST all 3.52 0.01 0.63 0.41 0.00 95.43
12:00:01 IST all 3.06 0.00 0.49 0.37 0.00 96.08
12:10:01 IST all 3.81 0.00 0.74 0.45 0.00 95.00
12:20:01 IST all 3.76 0.00 0.71 0.37 0.00 95.16
12:30:02 IST all 3.82 0.01 0.61 0.37 0.00 95.20
12:40:01 IST all 3.15 0.05 0.50 0.38 0.00 95.92
12:50:01 IST all 2.91 0.00 0.41 0.36 0.00 96.32
01:00:01 IST all 3.08 0.00 0.46 0.35 0.00 96.12
01:10:01 IST all 3.25 0.00 0.51 0.41 0.00 95.82
01:20:01 IST all 3.38 0.00 0.63 0.40 0.00 95.59
01:30:01 IST all 3.79 0.01 0.71 0.37 0.00 95.12
01:40:01 IST all 3.10 0.00 0.45 0.34 0.00 96.11
01:50:01 IST all 2.87 0.00 0.40 0.34 0.00 96.38
02:00:01 IST all 3.37 0.00 0.57 0.41 0.00 95.64
02:00:01 IST CPU %user %nice %system %iowait %steal %idle
02:10:01 IST all 4.25 0.00 0.87 0.42 0.00 94.46
02:20:01 IST all 3.54 0.01 0.61 0.38 0.00 95.46
02:30:01 IST all 3.63 0.00 0.73 0.38 0.00 95.25
02:40:01 IST all 3.56 0.00 0.53 0.43 0.00 95.47
02:50:01 IST all 3.92 0.05 0.68 0.40 0.00 94.95
03:00:01 IST all 3.51 0.00 0.56 0.35 0.00 95.59
03:10:01 IST all 4.27 0.01 0.73 0.41 0.00 94.58
03:20:01 IST all 4.03 0.00 0.76 0.41 0.00 94.80
03:30:01 IST all 3.75 0.01 0.61 0.36 0.00 95.27
03:40:01 IST all 3.86 0.00 0.72 0.43 0.00 94.99
03:50:01 IST all 4.30 0.00 0.76 0.40 0.00 94.54
04:00:01 IST all 3.53 0.00 0.59 0.38 0.00 95.50
04:10:01 IST all 6.22 0.01 0.65 0.37 0.00 92.75
04:20:01 IST all 2.61 0.00 0.43 0.34 0.00 96.61
04:30:01 IST all 3.40 0.00 0.63 0.34 0.00 95.63
04:40:01 IST all 3.69 0.00 0.56 0.39 0.00 95.35
04:50:01 IST all 3.30 0.05 0.54 0.35 0.00 95.76
05:00:01 IST all 3.48 0.01 0.52 0.35 0.00 95.64
05:10:01 IST all 3.72 0.00 0.60 0.39 0.00 95.29
05:20:01 IST all 3.86 0.00 0.65 0.36 0.00 95.13
05:30:01 IST all 2.82 0.00 0.49 0.38 0.00 96.31
05:40:01 IST all 2.73 0.00 0.47 0.37 0.00 96.43
05:50:01 IST all 2.70 0.34 0.43 0.37 0.00 96.16
06:00:01 IST all 2.61 0.00 0.39 0.32 0.00 96.68
06:10:01 IST all 2.67 0.00 0.41 0.37 0.00 96.54
06:20:01 IST all 2.65 0.00 0.41 0.36 0.00 96.58
06:30:01 IST all 2.78 0.00 0.44 0.30 0.00 96.47
06:40:01 IST all 2.81 0.05 0.42 0.33 0.00 96.39
06:50:01 IST all 2.92 0.00 0.50 0.32 0.00 96.26
07:00:01 IST all 2.72 0.00 0.49 0.34 0.00 96.44
07:10:01 IST all 2.81 0.00 0.75 0.34 0.00 96.10
07:20:01 IST all 2.29 0.00 0.42 0.29 0.00 96.99
07:30:01 IST all 2.34 0.01 0.38 0.30 0.00 96.97
07:40:01 IST all 2.61 0.00 0.42 0.33 0.00 96.64
07:50:01 IST all 2.67 0.00 0.51 0.32 0.00 96.50
08:00:01 IST all 2.25 0.00 0.37 0.31 0.00 97.07
08:10:01 IST all 2.32 0.00 0.36 0.33 0.00 96.98
08:20:01 IST all 2.07 0.01 0.36 0.32 0.00 97.25
08:30:01 IST all 3.06 0.00 0.63 0.35 0.00 95.96
08:40:01 IST all 2.70 0.04 0.57 0.29 0.00 96.40
08:50:01 IST all 2.47 0.00 0.40 0.34 0.00 96.78
09:00:01 IST all 2.10 0.00 0.36 0.34 0.00 97.20
09:00:01 IST CPU %user %nice %system %iowait %steal %idle
09:10:01 IST all 2.33 0.01 0.37 0.35 0.00 96.95
09:20:01 IST all 2.97 0.00 0.51 0.32 0.00 96.18
09:30:01 IST all 2.35 0.00 0.37 0.36 0.00 96.92
09:40:01 IST all 2.22 0.00 0.33 0.31 0.00 97.13
09:50:01 IST all 2.44 0.00 0.42 0.35 0.00 96.79
10:00:01 IST all 2.69 0.01 0.44 0.36 0.00 96.50
10:10:01 IST all 2.78 0.00 0.56 0.36 0.00 96.29
10:20:01 IST all 2.89 0.00 0.59 0.35 0.00 96.17
10:30:01 IST all 2.42 0.00 0.44 0.34 0.00 96.79
10:40:01 IST all 2.52 0.04 0.40 0.33 0.00 96.71
10:50:01 IST all 2.30 0.01 0.36 0.33 0.00 97.00
11:00:01 IST all 2.12 0.00 0.32 0.32 0.00 97.23
11:10:01 IST all 2.13 0.00 0.36 0.35 0.00 97.15
11:20:01 IST all 2.45 0.00 0.39 0.34 0.00 96.82
11:30:01 IST all 2.61 0.00 0.49 0.36 0.00 96.54
11:40:01 IST all 2.09 0.01 0.34 0.34 0.00 97.22
11:50:01 IST all 2.27 0.00 0.38 0.34 0.00 97.00
12:00:01 IST all 13.58 0.02 1.90 0.49 0.00 84.01
12:10:01 IST all 15.27 0.00 2.27 0.58 0.00 81.88
12:20:01 IST all 15.82 0.00 2.33 0.50 0.00 81.34
12:30:01 IST all 15.66 0.00 2.25 0.47 0.00 81.61
12:40:01 IST all 15.36 0.00 2.26 0.52 0.00 81.86
12:50:01 IST all 15.75 0.11 2.27 0.51 0.00 81.38
01:00:01 IST all 16.36 0.00 2.44 0.55 0.00 80.65
01:10:01 IST all 15.83 0.00 2.34 0.64 0.00 81.19
01:20:01 IST all 15.98 0.00 2.29 0.71 0.00 81.01
01:30:01 IST all 16.05 0.00 2.39 0.67 0.00 80.89
01:40:01 IST all 15.98 0.01 2.36 0.61 0.00 81.04
01:50:01 IST all 16.54 0.00 2.63 0.61 0.00 80.22
02:00:01 IST all 16.67 0.00 2.38 0.61 0.00 80.33
02:10:01 IST all 15.86 0.00 2.33 0.54 0.00 81.27
02:20:01 IST all 15.89 0.00 2.28 0.48 0.00 81.35
02:30:01 IST all 15.74 0.01 2.35 0.57 0.00 81.32
02:40:01 IST all 15.68 0.10 2.30 0.61 0.00 81.32
02:50:01 IST all 15.10 0.00 2.28 0.49 0.00 82.13
03:00:01 IST all 15.41 0.00 2.23 0.51 0.00 81.85
03:10:01 IST all 16.04 0.00 2.33 0.63 0.00 81.00
03:20:01 IST all 16.51 0.01 2.40 0.58 0.00 80.50
03:30:01 IST all 15.20 0.00 2.20 0.43 0.00 82.16
03:40:01 IST all 15.96 0.02 2.33 0.48 0.00 81.21
03:50:01 IST all 16.38 0.00 2.35 0.47 0.00 80.79
04:00:01 IST all 15.37 0.04 2.23 0.45 0.00 81.91
04:00:01 IST CPU %user %nice %system %iowait %steal %idle
04:10:01 IST all 15.55 0.01 2.26 0.54 0.00 81.64
04:20:01 IST all 15.62 0.00 2.24 0.62 0.00 81.51
04:30:01 IST all 15.77 0.00 2.26 0.57 0.00 81.39
04:40:01 IST all 15.74 0.09 2.32 0.56 0.00 81.29
04:50:01 IST all 15.45 0.00 2.26 0.43 0.00 81.85
05:00:01 IST all 16.45 0.01 2.44 0.54 0.00 80.56
05:10:01 IST all 15.25 0.00 2.26 0.51 0.00 81.98
05:20:01 IST all 16.03 0.00 2.33 0.49 0.00 81.15
05:30:01 IST all 16.10 0.00 2.31 0.49 0.00 81.09
05:40:01 IST all 15.48 0.00 2.25 0.54 0.00 81.73
05:50:01 IST all 14.72 0.01 2.27 0.44 0.00 82.57
06:00:01 IST all 15.40 0.00 2.20 0.47 0.00 81.92
06:10:01 IST all 15.94 0.00 2.31 0.51 0.00 81.23
06:20:01 IST all 14.39 0.00 2.05 0.49 0.00 83.06
06:30:01 IST all 13.76 0.00 1.95 0.52 0.00 83.77
06:40:01 IST all 15.10 0.10 2.22 0.61 0.00 81.96
06:50:01 IST all 14.88 0.00 2.22 0.51 0.00 82.38
07:00:01 IST all 14.71 0.00 2.14 0.52 0.00 82.62
07:10:01 IST all 14.42 0.00 2.11 0.46 0.00 83.01
07:20:01 IST all 13.62 0.00 1.97 0.56 0.00 83.86
07:30:01 IST all 13.82 0.01 1.99 0.52 0.00 83.66
07:40:01 IST all 13.88 0.00 1.97 0.46 0.00 83.69
07:50:01 IST all 13.34 0.00 1.92 0.54 0.00 84.20
08:00:01 IST all 14.61 0.00 2.21 0.42 0.00 82.76
08:10:01 IST all 14.48 0.00 2.11 0.51 0.00 82.89
08:20:01 IST all 15.40 0.01 2.24 0.50 0.00 81.85
08:30:01 IST all 15.30 0.00 2.19 0.37 0.00 82.13
08:40:01 IST all 16.15 0.00 2.40 0.60 0.00 80.84
08:50:01 IST all 16.31 0.11 2.39 0.54 0.00 80.65
09:00:01 IST all 14.46 0.00 2.10 0.57 0.00 82.86
09:10:01 IST all 15.64 0.01 2.30 0.57 0.00 81.49
09:20:01 IST all 16.32 0.00 2.43 0.61 0.00 80.64
09:30:01 IST all 15.15 0.00 2.18 0.54 0.00 82.13
09:40:01 IST all 15.94 0.00 2.31 0.53 0.00 81.21
09:50:01 IST all 16.27 0.00 2.43 0.57 0.00 80.73
10:00:01 IST all 15.19 0.01 2.24 0.54 0.00 82.02
10:10:01 IST all 15.37 0.00 2.22 0.54 0.00 81.87
10:20:01 IST all 15.77 0.00 2.27 0.62 0.00 81.34
10:30:01 IST all 15.80 0.00 2.26 0.63 0.00 81.31
10:40:01 IST all 16.02 0.00 2.33 0.60 0.00 81.04
10:50:01 IST all 16.08 0.10 2.34 0.66 0.00 80.82
11:00:01 IST all 14.85 0.00 2.15 0.57 0.00 82.43
11:00:01 IST CPU %user %nice %system %iowait %steal %idle
11:10:01 IST all 15.40 0.00 2.31 0.52 0.00 81.77
11:20:01 IST all 15.86 0.00 2.29 0.56 0.00 81.29
11:30:01 IST all 15.54 0.00 2.29 0.59 0.00 81.58
11:40:01 IST all 15.70 0.01 2.27 0.54 0.00 81.48
11:50:01 IST all 15.81 0.00 2.29 0.77 0.00 81.12
12:00:01 IST all 16.26 0.00 2.37 0.75 0.00 80.62
12:10:01 IST all 14.53 0.00 2.04 0.75 0.00 82.68
12:20:01 IST all 14.98 0.00 2.13 0.51 0.00 82.37
12:30:01 IST all 16.39 0.01 2.39 0.51 0.00 80.70
12:40:01 IST all 16.57 0.10 2.41 0.54 0.00 80.39
12:50:01 IST all 14.42 0.00 2.10 0.65 0.00 82.83
01:00:01 IST all 16.87 0.00 2.44 0.86 0.00 79.82
01:10:01 IST all 15.01 0.00 2.22 0.67 0.00 82.09
01:20:01 IST all 15.13 0.01 2.13 0.49 0.00 82.24
01:30:01 IST all 16.26 0.00 2.49 0.58 0.00 80.65
01:40:01 IST all 15.42 0.00 2.22 0.55 0.00 81.80
01:50:01 IST all 15.60 0.00 2.25 0.86 0.00 81.28
02:00:01 IST all 14.82 0.00 2.10 0.79 0.00 82.28
02:10:01 IST all 19.04 0.01 2.52 0.72 0.00 77.72
02:20:01 IST all 15.42 0.00 2.20 0.75 0.00 81.62
02:30:01 IST all 15.51 0.00 2.22 0.77 0.00 81.50
02:40:01 IST all 15.28 0.09 2.26 0.54 0.00 81.82
02:50:01 IST all 15.96 0.00 2.33 0.68 0.00 81.03
03:00:01 IST all 16.47 0.00 2.42 0.83 0.00 80.27
03:10:01 IST all 16.25 0.01 2.39 0.79 0.00 80.57
03:20:01 IST all 16.27 0.00 2.39 0.88 0.00 80.46
03:30:02 IST all 17.21 0.00 2.58 0.68 0.00 79.53
03:40:01 IST all 15.56 0.00 2.22 0.87 0.00 81.35
03:50:01 IST all 16.05 0.00 2.33 0.90 0.00 80.71
04:00:01 IST all 16.19 0.01 2.34 0.85 0.00 80.60
04:10:01 IST all 16.06 0.00 2.34 0.83 0.00 80.76
04:20:02 IST all 16.80 0.00 2.44 0.75 0.00 80.01
04:30:01 IST all 8.06 0.00 1.09 0.57 0.00 90.28
04:40:01 IST all 15.70 0.10 2.26 0.52 0.00 81.43
04:50:01 IST all 16.19 0.01 2.33 0.53 0.00 80.95
05:00:01 IST all 15.34 0.00 2.22 0.53 0.00 81.91
05:10:01 IST all 15.89 0.00 2.28 0.54 0.00 81.30
05:20:01 IST all 15.75 0.00 2.28 0.54 0.00 81.43
05:30:01 IST all 15.95 0.00 2.35 0.68 0.00 81.02
05:40:01 IST all 16.17 1.63 2.40 0.55 0.00 79.25
05:50:01 IST all 17.11 0.01 2.54 0.59 0.00 79.76
06:00:01 IST all 17.19 0.00 2.55 0.55 0.00 79.71
06:00:01 IST CPU %user %nice %system %iowait %steal %idle
06:10:01 IST all 16.83 0.00 2.51 0.65 0.00 80.02
06:20:01 IST all 16.78 0.00 2.49 0.63 0.00 80.10
06:30:01 IST all 15.85 0.00 2.35 0.59 0.00 81.20
06:40:01 IST all 16.74 0.10 2.45 0.55 0.00 80.16
06:50:01 IST all 15.78 0.00 2.33 0.59 0.00 81.29
07:00:01 IST all 16.33 0.00 2.45 0.76 0.00 80.45
07:10:01 IST all 17.40 0.00 2.56 0.83 0.00 79.20
07:20:01 IST all 17.14 0.00 2.50 0.66 0.00 79.69
07:30:01 IST all 18.79 0.01 2.76 0.61 0.00 77.83
07:40:01 IST all 17.23 0.00 2.49 0.62 0.00 79.65
07:50:01 IST all 18.18 0.00 2.66 0.63 0.00 78.53
08:00:01 IST all 17.46 0.00 2.50 0.58 0.00 79.45
08:10:01 IST all 17.51 0.00 2.59 0.75 0.00 79.14
08:20:01 IST all 16.55 0.01 2.42 0.78 0.00 80.25
08:30:01 IST all 16.15 0.00 2.32 0.56 0.00 80.97
08:40:01 IST all 17.59 0.00 2.50 0.72 0.00 79.18
Average: all 12.60 0.02 1.78 0.49 0.00 85.11
--------------------------------------------------------------------------------------------
hardware info
root@some-network [~/scripts]# cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU W3530 @ 2.80GHz
stepping : 5
microcode : 0x11
cpu MHz : 2801.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.32
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 1
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU W3530 @ 2.80GHz
stepping : 5
microcode : 0x11
cpu MHz : 2801.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 1
cpu cores : 4
apicid : 2
initial apicid : 2
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.32
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 2
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU W3530 @ 2.80GHz
stepping : 5
microcode : 0x11
cpu MHz : 2801.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 2
cpu cores : 4
apicid : 4
initial apicid : 4
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.32
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 3
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU W3530 @ 2.80GHz
stepping : 5
microcode : 0x11
cpu MHz : 2801.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 4
apicid : 6
initial apicid : 6
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.32
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 4
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU W3530 @ 2.80GHz
stepping : 5
microcode : 0x11
cpu MHz : 2801.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 0
cpu cores : 4
apicid : 1
initial apicid : 1
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.32
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 5
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU W3530 @ 2.80GHz
stepping : 5
microcode : 0x11
cpu MHz : 2801.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 1
cpu cores : 4
apicid : 3
initial apicid : 3
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.32
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 6
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU W3530 @ 2.80GHz
stepping : 5
microcode : 0x11
cpu MHz : 2801.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 2
cpu cores : 4
apicid : 5
initial apicid : 5
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.32
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
processor : 7
vendor_id : GenuineIntel
cpu family : 6
model : 26
model name : Intel(R) Xeon(R) CPU W3530 @ 2.80GHz
stepping : 5
microcode : 0x11
cpu MHz : 2801.000
cache size : 8192 KB
physical id : 0
siblings : 8
core id : 3
cpu cores : 4
apicid : 7
initial apicid : 7
fpu : yes
fpu_exception : yes
cpuid level : 11
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf pni dtes64 monitor ds_cpl vmx est tm2 ssse3 cx16 xtpr pdcm dca sse4_1 sse4_2 popcnt lahf_lm ida dtherm tpr_shadow vnmi flexpriority ept vpid
bogomips : 5600.32
clflush size : 64
cache_alignment : 64
address sizes : 36 bits physical, 48 bits virtual
power management:
memory info
root@some-network [~/scripts]# cat /proc/meminfo
MemTotal: 24737800 kB
MemFree: 16371108 kB
Buffers: 788680 kB
Cached: 4937140 kB
SwapCached: 0 kB
Active: 5279624 kB
Inactive: 1600700 kB
Active(anon): 1157900 kB
Inactive(anon): 116256 kB
Active(file): 4121724 kB
Inactive(file): 1484444 kB
Unevictable: 0 kB
Mlocked: 0 kB
SwapTotal: 525308 kB
SwapFree: 525308 kB
Dirty: 1004 kB
Writeback: 0 kB
AnonPages: 1153496 kB
Mapped: 166080 kB
Shmem: 120240 kB
Slab: 1254664 kB
SReclaimable: 995104 kB
SUnreclaim: 259560 kB
KernelStack: 3104 kB
PageTables: 11188 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
WritebackTmp: 0 kB
CommitLimit: 12894208 kB
Committed_AS: 3086672 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 326396 kB
VmallocChunk: 34359409652 kB
HardwareCorrupted: 0 kB
DirectMap4k: 5632 kB
DirectMap2M: 25151488 kBI just want to know is there any particular reason for sudden increase in mysql load? Thank you guys.
-
key_buffer_size set over 110M also increase innodb_buffer_pool_size=560M InnoDB can use more and decreate table cache to 2 In your case I would turn on query cache Maybe it's time to optimize that code and add some caching [!!] Joins performed without indexes: 53107745 You should review database structure if it needs some indexes I would suggest here simple solution, add log-queries-not-using-indexes track those queries not using indexes, and add those indexes, or if necessary optimize queries to utilize indexes In most cases when basic mysql optimizations (few key settings) are done, the biggest part that can impact performance is the code People tend to search some magic solutions in mysql, some people have tens of variables in their my.cnf, but they dont look into their code 0 -
Thank you for your reply. [quote="thinkbot, post: 1600862"> key_buffer_size set over 110M also increase innodb_buffer_pool_size=560M InnoDB can use more and decreate table cache to 2
key_buffer_size is now set to 128M (~myisam database size) Innodb ~ i haven't changed. decreate table cache to 2? you mean 2048? [quote="thinkbot, post: 1600862"> In your case I would turn on query cache
Query cache doesnt help my case at all. There is frequent defragmentation and a constant overhead. I have checked with various values from 128M to 32M, and cache_min_res. [quote="thinkbot, post: 1600862"> Maybe it's time to optimize that code and add some caching [!!] Joins performed without indexes: 53107745 You should review database structure if it needs some indexes I would suggest here simple solution, add log-queries-not-using-indexes track those queries not using indexes, and add those indexes, or if necessary optimize queries to utilize indexes
Yes, i am aware of non-index using joins. Its is not a new thing for our setting, i have always had non-index using joins (i knows its bad for performance). Anyways, My problem is investigating the "sudden" rise in mysql load over last one week.0 -
You may also want to try using a command such as "mysqladmin processlist" to get a better idea of what's happening when you notice higher than normal MySQL usage (since you stated this has only started recently). Thank you. 0 -
A quick update: 1. key_buffer_size now set at 128M 2. File open limit (auto set by cpanel ~8250) 3. Table Cache ~ 2048 4. Current Mysql load ~ 55% 5. http traffic checked for any attacks (nothing unusual) 6. Had done hard restarts, checked hardwares for r/w faults none. I am not sure why the load keeps increasing. Its as if some kinda overhead is building on. And it seemingly is incremental. My next plan of action is just try disabling one db at a time and probably isolate the menace. @thinkbot - i wonder if fractional slow_query times can be set. @cPanelMichael - have done that already. I was hoping to find sleeping queries or anything unusual but none there either. 0
Please sign in to leave a comment.
Comments
5 comments