A (little) MySQL bug story…

I just want to share about a strange behavior of one of our MySQL server yesterday.
This server is a 5.1.50 MySQL server on debian 4.0 (Yes, I know…)

When a “mysqld got signal 6” error occurred yesterday, the MySQL server crashed and didn’t want to restart.
Then, I found these informations in the error log file :

/usr/local/mysql/bin/mysqld: File '*** glibc detected ***
malloc():memory corruption: 0x00002aac2d5ab460 ***' not found (Errcode: 2)
120306 17:19:47 [ERROR] Failed to open log (file '*** glibc detected ***
malloc():memory corruption: 0x00002aac2d5ab460 ***', errno 2)
120306 17:19:47 [ERROR] Could not open log file
120306 17:19:47 [ERROR] Can't init tc log
120306 17:19:47 [ERROR] Aborting
120306 17:19:47 InnoDB: Starting shutdown...
120306 17:19:53 InnoDB: Shutdown completed; log sequence number 55 1061584593
120306 17:19:53 [Note] /usr/local/mysql/bin/mysqld: Shutdown complete

Great, let see what is the Errcode 2 through the perror utility :

# perror 2
OS error code 2: No such file or directory

But where is this missing file ? And have you ever heard about a file called “*** glibc detected *** malloc():memory corruption: 0x00002aac2d5ab460 ***” ?!
No, of course not. But this error message  was more meaningful : Could not open log file

The InnoDB log files and binary log files seemed to be ok (file permissions and other checks showed no failure)
But when I did a cat on the binary log index file… surprise ! :

# cat mysql-bin.index
/var/database/bin-log/mysql-bin.126242
/var/database/bin-log/mysql-bin.126243
/var/database/bin-log/mysql-bin.126244
/var/database/bin-log/mysql-bin.126245
*** glibc detected *** malloc(): memory corruption: 0x00002aac2d5ab460 ***

The index file was corrupted during the last crash and an error message was inserted inside this file because of a memory corruption.
Deleting files helped to restart the MySQL server but I found very strange that MySQL wasn’t able to deliver a more accurate message about that.

PS : A MySQL bug report was created : http://bugs.mysql.com/bug.php?id=64579




Share the love!

Subscribe to RSS feed or by email to automatically receive real-time notifications
Oracle, MySQL, and InnoDB are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective owners

4 Comments

  1. This is how a memory corruption error looks like.
    Do you know what have you done to go into this state ?
    If you can repeat the transition from the working state to this corrupted state please file a bug with your findings.

    Reply

Leave a Comment.