I just want to share about a strange behavior of one of our MySQL server yesterday.
This server is a 5.1.50 MySQL server on debian 4.0 (Yes, I know…)
When a “mysqld got signal 6” error occurred yesterday, the MySQL server crashed and didn’t want to restart.
Then, I found these informations in the error log file :
/usr/local/mysql/bin/mysqld: File '*** glibc detected *** malloc():memory corruption: 0x00002aac2d5ab460 ***' not found (Errcode: 2) 120306 17:19:47 [ERROR] Failed to open log (file '*** glibc detected *** malloc():memory corruption: 0x00002aac2d5ab460 ***', errno 2) 120306 17:19:47 [ERROR] Could not open log file 120306 17:19:47 [ERROR] Can't init tc log 120306 17:19:47 [ERROR] Aborting
120306 17:19:47 InnoDB: Starting shutdown... 120306 17:19:53 InnoDB: Shutdown completed; log sequence number 55 1061584593 120306 17:19:53 [Note] /usr/local/mysql/bin/mysqld: Shutdown complete
Great, let see what is the Errcode 2 through the perror utility :
# perror 2 OS error code 2: No such file or directory
But where is this missing file ? And have you ever heard about a file called “*** glibc detected *** malloc():memory corruption: 0x00002aac2d5ab460 ***” ?!
No, of course not. But this error message was more meaningful : Could not open log file
The InnoDB log files and binary log files seemed to be ok (file permissions and other checks showed no failure)
But when I did a cat on the binary log index file… surprise ! :
# cat mysql-bin.index /var/database/bin-log/mysql-bin.126242 /var/database/bin-log/mysql-bin.126243 /var/database/bin-log/mysql-bin.126244 /var/database/bin-log/mysql-bin.126245 *** glibc detected *** malloc(): memory corruption: 0x00002aac2d5ab460 ***
The index file was corrupted during the last crash and an error message was inserted inside this file because of a memory corruption.
Deleting files helped to restart the MySQL server but I found very strange that MySQL wasn’t able to deliver a more accurate message about that.
PS : A MySQL bug report was created : http://bugs.mysql.com/bug.php?id=64579