I'm moving an existing 5GB database to a new server (Intel quad core, 8Gb RAM) with software RAID 1 disks, CentOS 6 and MySQL 5.1.52-64 bit installed. The normal CLI import of the database onto the new server went off without a hitch, no significant delays on the INSERT statements.
However, when doing a large batch update via a PHP (5.3.8) script that includes updates to INNODB tables and inserts to MyISAM tables, a few hundred or thousand records will process quickly for about 2 to 10 seconds and then all of a sudden get hung up on a single insert or update. The hangups mainly occur for MyISAM tables (inserts) on the status 'closing tables' and for INNODB tables (updates) on 'freeing items'. The hangup last from 2 to 20 seconds and then things proceeds normally. There is no other activity on the server.
I thought the problem might lie with the software RAID, so I did some random access read/write tests (shown below). But the setup performs slightly better than my test server. Also, I can run this same batch update on my test box (an old box with 2GB ram and with almost the same my.cnf) and see the same effects, though the hangups are significanly shorter and less frequent. However on my Windows development laptop I see no problems at all.
FYI, there seems to be no problem with normal SELECTs. They proceed without hangup and I can get 4,000 to 5,000 a second on tables with millions of rows. My test box and Win7 laptop with the same benchmark test get 3,000 to 4,000. The inserts/updates in question are the simple single table variety. Eg: INSERT IGNORE INTO some_myisam_table (fld1, fld2, fld3, fld4, fld5) VALUES (val1, val2, val3, val4, val5). And for updates: UPDATE some_innodb_table SET field='abcd' WHERE key=12345. The tables they are operating on have between 1 million to 20 million rows and three or four keys per table.
It almost feels like server is flushing a cache ever few seconds when I see a single update hanging, and then proceeding normally. I've poured over the my.cnf as well as searched all potential bugs online. Any thoughts?
However, when doing a large batch update via a PHP (5.3.8) script that includes updates to INNODB tables and inserts to MyISAM tables, a few hundred or thousand records will process quickly for about 2 to 10 seconds and then all of a sudden get hung up on a single insert or update. The hangups mainly occur for MyISAM tables (inserts) on the status 'closing tables' and for INNODB tables (updates) on 'freeing items'. The hangup last from 2 to 20 seconds and then things proceeds normally. There is no other activity on the server.
I thought the problem might lie with the software RAID, so I did some random access read/write tests (shown below). But the setup performs slightly better than my test server. Also, I can run this same batch update on my test box (an old box with 2GB ram and with almost the same my.cnf) and see the same effects, though the hangups are significanly shorter and less frequent. However on my Windows development laptop I see no problems at all.
FYI, there seems to be no problem with normal SELECTs. They proceed without hangup and I can get 4,000 to 5,000 a second on tables with millions of rows. My test box and Win7 laptop with the same benchmark test get 3,000 to 4,000. The inserts/updates in question are the simple single table variety. Eg: INSERT IGNORE INTO some_myisam_table (fld1, fld2, fld3, fld4, fld5) VALUES (val1, val2, val3, val4, val5). And for updates: UPDATE some_innodb_table SET field='abcd' WHERE key=12345. The tables they are operating on have between 1 million to 20 million rows and three or four keys per table.
It almost feels like server is flushing a cache ever few seconds when I see a single update hanging, and then proceeding normally. I've poured over the my.cnf as well as searched all potential bugs online. Any thoughts?
## HD READ RANDOM ACCESS TEST (done with SeekMark - http://learnitwithme.com/) ## ----------------------- ## Threads: 2 ## Seeks: 1000 ## Read from File: test.file (4.8Gb) [mm@server]# ./seekmark -t2 -s1000 -f ./test.file READ benchmarking against ./test.file 4834 MB threads to spawn: 2 seeks per thread: 1000 io size in bytes: 512 Spawning worker 0 to do 1000 seeks Spawning worker 1 to do 1000 seeks thread 1 completed, time: 6.03, 165.92 seeks/sec, 6.0ms per request thread 0 completed, time: 6.41, 156.01 seeks/sec, 6.4ms per request total time: 6.41, time per READ request(ms): 3.205 311.96 total seeks per sec, 155.98 READ seeks per sec per thread ## HD WRITE RANDOM ACCESS TEST ## ------------------------ ## Threads: 2 ## Seeks: 1000 ## Write to File: test.file (4.8Gb) [mm@server]# ./seekmark -t2 -s1000 -w destroy-data -f ./test.file WRITE benchmarking against ./test.file 4834 MB threads to spawn: 2 seeks per thread: 1000 io size in bytes: 512 write data is randomly generated Spawning worker 0 to do 1000 seeks Spawning worker 1 to do 1000 seeks thread 0 completed, time: 38.95, 25.67 seeks/sec, 39.0ms per request thread 1 completed, time: 39.10, 25.57 seeks/sec, 39.1ms per request total time: 39.10, time per WRITE request(ms): 19.552 51.14 total seeks per sec, 25.57 WRITE seeks per sec per thread ## MY.CNF ## ------ [client] port = 3306 socket = /var/lib/mysql/mysql.sock [mysql] default-character-set=utf8 no-auto-rehash [mysqld] port = 3306 datadir=/var/lib/mysql socket=/var/lib/mysql/mysql.sock user=mysql symbolic-links=0 character-set-server=utf8 max_connections = 250 max_user_connections = 40 key_buffer = 384M myisam_sort_buffer_size = 64M join_buffer_size = 2M read_buffer_size = 2M sort_buffer_size = 2M table_open_cache = 1024 thread_cache_size = 16 wait_timeout = 120 connect_timeout = 10 max_allowed_packet = 4M max_connect_errors = 10 tmp_table_size = 16M query_cache_limit = 1M query_cache_size = 64M query_cache_type = 1 myisam_max_sort_file_size=40G read_rnd_buffer_size=8M innodb_additional_mem_pool_size = 20M innodb_flush_method = O_DIRECT innodb_log_buffer_size = 4M innodb_buffer_pool_size = 2048M innodb_log_file_size = 256M innodb_thread_concurrency = 32 innodb_flush_log_at_trx_commit = 2 log_output = FILE slow_query_log = 1 slow_query_log_file = /var/log/mysql-slow.log long_query_time = 1 [mysqld_safe] open_files_limit = 8192 log-error=/var/log/mysqld.log pid-file=/var/run/mysqld/mysqld.pid [mysqldump] quick max_allowed_packet = 4M [myisamchk] key_buffer = 256M sort_buffer = 256M read_buffer = 2M write_buffer = 2M [mysqlhotcopy] interactive-timeout