I have a small server with Digital Ocean running Ubuntu 20.04 with 16GB of RAM and 1GB of swap space. The server is used to run various web applications and services.
The problem is that sometimes the server runs out of memory and becomes unresponsive, where the oom-killer kills the mysql process. This usually happens when there is a sudden spike in traffic or when I run some memory-intensive tasks.
At any one time we have roughly 50 users, 4 ubuntu users with VS Code and Remote SSH.
I want to know what variables or settings I can tweak to avoid these memory crashes and improve the performance and stability of my server. Or am I simply in a situation where I need to get additional memory.
Ran MySQLTuner and received the following:
[--] Skipped version check for MySQLTuner script
[OK] Logged in using credentials from Debian maintenance account.
[!!] Your MySQL version 10.2.44-MariaDB-1:10.2.44+maria~bionic-log is EOL software. Upgrade soon!
[OK] Operating on 64-bit architecture
-------- Log file Recommendations ------------------------------------------------------------------
[OK] Log file /var/lib/mysql/mysql-error.log exists
[--] Log file: /var/lib/mysql/mysql-error.log (3M)
[OK] Log file /var/lib/mysql/mysql-error.log is not empty
[OK] Log file /var/lib/mysql/mysql-error.log is smaller than 32 Mb
[OK] Log file /var/lib/mysql/mysql-error.log is readable.
[!!] /var/lib/mysql/mysql-error.log contains 10584 warning(s).
[!!] /var/lib/mysql/mysql-error.log contains 9939 error(s).
[--] 89 start(s) detected in /var/lib/mysql/mysql-error.log
[--] 1) 2024-02-02 15:54:13 140708901082816 [Note] /usr/sbin/mysqld: ready for connections.
[--] 2) 2024-02-02 15:42:47 140543570908864 [Note] /usr/sbin/mysqld: ready for connections.
[--] 3) 2024-02-01 12:21:55 140302234851008 [Note] /usr/sbin/mysqld: ready for connections.
[--] 4) 2024-01-30 14:08:31 139672760092352 [Note] /usr/sbin/mysqld: ready for connections.
[--] 5) 2024-01-24 14:05:35 139926124615360 [Note] /usr/sbin/mysqld: ready for connections.
[--] 6) 2024-01-23 13:42:55 140439711549120 [Note] /usr/sbin/mysqld: ready for connections.
[--] 7) 2024-01-21 16:06:07 139863368140480 [Note] /usr/sbin/mysqld: ready for connections.
[--] 8) 2023-11-07 10:22:30 140584071567040 [Note] /usr/sbin/mysqld: ready for connections.
[--] 9) 2023-11-06 13:23:52 140407900714688 [Note] /usr/sbin/mysqld: ready for connections.
[--] 10) 2023-11-06 12:12:41 140413023254208 [Note] /usr/sbin/mysqld: ready for connections.
[--] 18 shutdown(s) detected in /var/lib/mysql/mysql-error.log
[--] 1) 2024-01-21 16:05:45 140576751769344 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 2) 2023-11-07 10:22:28 140401446020864 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 3) 2023-11-06 11:19:38 140555894138624 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 4) 2023-10-06 9:58:48 140567196006144 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 5) 2023-10-05 11:37:42 140316381210368 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 6) 2023-09-14 10:10:34 140544218359552 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 7) 2023-09-01 7:34:06 140624616740608 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 8) 2023-08-01 9:40:33 139756556404480 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 9) 2023-08-01 9:03:19 140042846336768 [Note] /usr/sbin/mysqld: Shutdown complete
[--] 10) 2023-08-01 7:00:13 140447310087936 [Note] /usr/sbin/mysqld: Shutdown complete
-------- Storage Engine Statistics -----------------------------------------------------------------
[--] Status: +Aria +CSV +InnoDB +MEMORY +MRG_MyISAM +MyISAM +PERFORMANCE_SCHEMA +SEQUENCE
[--] Data in InnoDB tables: 15.1G (Tables: 9900)
[--] Data in MyISAM tables: 1.3G (Tables: 18)
[!!] Total fragmented tables: 1
-------- Analysis Performance Metrics --------------------------------------------------------------
[--] innodb_stats_on_metadata: OFF
[OK] No stat updates during querying INFORMATION_SCHEMA.
-------- Views Metrics -----------------------------------------------------------------------------
-------- Triggers Metrics --------------------------------------------------------------------------
-------- Routines Metrics --------------------------------------------------------------------------
-------- Security Recommendations ------------------------------------------------------------------
[OK] There are no anonymous accounts for any database users
[OK] All database users have passwords assigned
[--] There are 620 basic passwords in the list.
-------- CVE Security Recommendations --------------------------------------------------------------
[OK] NO SECURITY CVE FOUND FOR YOUR VERSION
-------- Performance Metrics -----------------------------------------------------------------------
[--] Up for: 3m 51s (27K q [121.078 qps], 971 conn, TX: 78M, RX: 5M)
[--] Reads / Writes: 97% / 3%
[--] Binary logging is enabled (GTID MODE: ON)
[--] Physical Memory : 15.7G
[--] Max MySQL memory : 8.8G
[--] Other process memory: 0B
[--] Total buffers: 5.3G global + 70.3M per thread (50 max threads)
[--] Performance_schema Max memory usage: 0B
[--] Galera GCache Max memory usage: 0B
[OK] Maximum reached memory usage: 6.2G (39.38% of installed RAM)
[OK] Maximum possible memory usage: 8.8G (56.03% of installed RAM)
[OK] Overall possible memory usage with other process is compatible with memory available
[OK] Slow queries: 0% (0/27K)
[OK] Highest usage of available connections: 24% (12/50)
[OK] Aborted connections: 0.00% (0/971)
[!!] Name resolution is active: a reverse name resolution is made for each new connection which can reduce performance
[!!] Query cache may be disabled by default due to mutex contention.
[OK] Query cache efficiency: 33.2% (11K cached / 34K selects)
[OK] Query cache prunes per day: 0
[OK] Sorts requiring temporary tables: 0% (3 temp sorts / 4K sorts)
[!!] Joins performed without indexes: 31
[!!] Temporary tables created on disk: 35% (155 on disk / 441 total)
[OK] Thread cache hit rate: 98% (12 created / 971 connections)
[OK] Table cache hit rate: 99% (914 hits / 923 requests)
[!!] table_definition_cache (8192) is less than number of tables (10078)
[OK] Open file limit used: 0% (93/32K)
[OK] Table locks acquired immediately: 100% (97 immediate / 97 locks)
[OK] Binlog cache memory access: 94.63% (141 Memory / 149 Total)
-------- Performance schema ------------------------------------------------------------------------
[!!] Performance_schema should be activated.
[--] Sys schema is not installed.
-------- ThreadPool Metrics ------------------------------------------------------------------------
[--] ThreadPool stat is disabled.
-------- MyISAM Metrics ----------------------------------------------------------------------------
[!!] Key buffer used: 20.0% (25.6M used / 128.0M cache)
[!!] Key buffer size / total MyISAM indexes: 128.0M/507.6M
[!!] Read Key buffer hit rate: 34.3% (142K cached / 93K reads)
[!!] Write Key buffer hit rate: 73.8% (439 cached / 324 writes)
-------- InnoDB Metrics ----------------------------------------------------------------------------
[--] InnoDB is enabled.
[--] InnoDB Thread Concurrency: 0
[OK] InnoDB File per table is activated
[!!] InnoDB buffer pool / data size: 5.0G / 15.1G
[!!] Ratio InnoDB log file size / InnoDB Buffer pool size (10%): 256.0M * 2 / 5.0G should be equal to 25%
[OK] InnoDB buffer pool instances: 5
[--] Number of InnoDB Buffer Pool Chunk: 5 for 5 Buffer Pool Instance(s)
[OK] Innodb_buffer_pool_size aligned with Innodb_buffer_pool_chunk_size & Innodb_buffer_pool_instances
[OK] InnoDB Read buffer efficiency: 99.20% (28233223 hits / 28461906 total)
[!!] InnoDB Write Log efficiency: 88.46% (2469 hits / 2791 total)
[OK] InnoDB log waits: 0.00% (0 waits / 322 writes)
-------- Aria Metrics ------------------------------------------------------------------------------
[--] Aria Storage Engine is enabled.
[OK] Aria pagecache size / total Aria indexes: 128.0M/0B
[OK] Aria pagecache hit rate: 100.0% (787K cached / 132 reads)
-------- TokuDB Metrics ----------------------------------------------------------------------------
[--] TokuDB is disabled.
-------- XtraDB Metrics ----------------------------------------------------------------------------
[--] XtraDB is disabled.
-------- Galera Metrics ----------------------------------------------------------------------------
[--] Galera is disabled.
-------- Replication Metrics -----------------------------------------------------------------------
[--] Galera Synchronous replication: NO
[--] No replication slave(s) for this server.
[--] Binlog format: MIXED
[--] XA support enabled: ON
[--] Semi synchronous replication Master: Not Activated
[--] Semi synchronous replication Slave: Not Activated
[--] This is a standalone server
-------- Recommendations ---------------------------------------------------------------------------
General recommendations:
You are using an unsupported version for production environments
Upgrade as soon as possible to a supported version !
Check warning line(s) in /var/lib/mysql/mysql-error.log file
Check error line(s) in /var/lib/mysql/mysql-error.log file
Run ALTER TABLE ... FORCE or OPTIMIZE TABLE to defragment tables for better performance
OPTIMIZE TABLE `____`.`__global_search`; -- can free 148.100666046143 MiB
Total freed space after defragmentation : 148.100666046143 MiB
MySQL was started within the last 24 hours: recommendations may be inaccurate
Configure your accounts with ip or subnets only, then update your configuration with skip-name-resolve=1
We will suggest raising the 'join_buffer_size' until JOINs not using indexes are found.
See
https://dev.mysql.com/doc/internals/en/join-buffer-size.html
(specially the conclusions at the bottom of the page).
When making adjustments, make tmp_table_size/max_heap_table_size equal
Reduce your SELECT DISTINCT queries which have no LIMIT clause
Performance schema should be activated for better diagnostics
Consider installing Sys schema from
https://github.com/FromDual/mariadb-sys
for MariaDB
Before changing innodb_log_file_size and/or innodb_log_files_in_group read this:
Variables to adjust:
skip-name-resolve=1
query_cache_size (=0)
query_cache_type (=0)
join_buffer_size (> 2.0M, or always use indexes with JOINs)
tmp_table_size (> 32M)
max_heap_table_size (> 32M)
table_definition_cache (8192) > 10078 or -1 (autosizing if supported)
performance_schema=ON
key_buffer_size (~ 26M)
key_buffer_size (> 507.6M)
innodb_buffer_pool_size (>= 15.1G) if possible.
innodb_log_file_size should be (=640M) if possible, so InnoDB total log file size equals 25% of buffer pool size.
For the most part, we've tweaked the innodb pool variables to try fix in addition to occasionally monitoring and killing processes that spike (very manual though).