Temporary file behavior… (and how lsof save my life)

I would like to share this story based on a true event about the temporary files behavior in MySQL.

MONyog reports this error to my already full mailbox several times a day :


 

1 – Catch the query (if you can) !

 

I don’t have access to the client logs but I would like to know which query is involved in this error.

Let me explain how I can retrieve informations about this query with MONyog and a very simple shell loop :

  •  Enable the query sniffer in MONyog (based on processlist) : Edit server -> Advanced settings -> Sniffer settings

 

  • Monitor your MySQL TMPDIR directory with this simple shell loop :
  • [ Use this command to retrieve the MySQL temporary directory : show variables like 'tmpdir'; ]
  • [ In this case : tmpdir=/database/tmp ]
while [ 1 = 1 ]; do { date;ls -artlh /database/tmp ; df -h /database/tmp;
  lsof | grep mysql | grep /database/tmp;sleep 1; }; done
 
With this loop, I can follow in real time the informations about the files created in the MySQL temporary directory and it will be very useful to find how exactly the error happens (see below).

With the MONyog query sniffer, I found the query involved by comparing the two emails sent by MONyog (Error message above and the emails about long queries).
The explain plan of this query was as follow :

 

And finaly, the real error was :
ERROR 3 (HY000): Error writing file '/database/tmp/MYakhJC5' (Errcode: 28)
 Error (Code 3): Error writing file '/database/tmp/MYakhJC5' (Errcode: 28)
 Error (Code 1028): Sort aborted
# perror 28
 OS error code  28:  No space left on device

 

2 – No space left on device, really ?!

 

Let see the filesystem definition for /database/tmp with the df command :
And here is a list view of the files in the directory when the error occured :
 total 2,3G
 drwxrwx--- 11 mysql mysql 4,0K 16 févr. 10:37 ..
 -rw-rw----  1 mysql mysql    6  6 avril 15:39 mysql.pid
 srwxrwxrwx  1 mysql mysql    0  6 avril 15:39 mysql.sock
 -rw-rw----  1 mysql mysql 1,0K 12 avril 15:41 #sql_7237_0.MYI
 -rw-rw----  1 mysql mysql 2,3G 12 avril 15:42 #sql_7237_0.MYD
 -rw-rw----  1 mysql mysql 1,0K 12 avril 15:42 #sql_7237_5.MYI
 -rw-rw----  1 mysql mysql    0 12 avril 15:42 #sql_7237_5.MYD
 drwxrwx---  2 mysql mysql 4,0K 12 avril 15:43 .
The /database/tmp filesystem has a size of 6,5Go and there was a total of 2,3Go of files when the error occured !

 

3 - “Was passiert ?!”

 

The lsof command was my very good friend to discover what happened.
Let see the trace of my tiny loop when the error occured (lsof part only) :

 

 

I can see a 2,3Go temporary file but there are two other temporary files which continue to grow until the end of the world :-(
And these two (deleted) files appears to be the cause of my issue.

 

4 – What next ?

 

The killer question : why did you have only 6.5 GB for your temporary space ?
Answer : Why not :-)

You have to consider this behavior to set your temporary space properly, specially if this space is a ramdisk filesystem.

Tschüs !

PS : I let the experts explains why these two additional files are created (hint : look at the Exta column)

Vote for MySQL[plus] awards 2011 !

First of all, I wish you a happy new year.
Many things happened last year, it was really exciting to be involved in the MySQL ecosystem.
I hope this enthusiasm will be increased this year, up to you !

To start the year, I propose the MySQL[plus] Awards 2011
It will only take 5 minutes to fill out these polls.
Answer with your heart first and then with your experience with some of these tools or services.

Polls will be closed January 31, so, vote now !
For “other” answers, please,  let me a comment with details.

Don’t hesitate to submit proposal for tools or services in the comments.
And, please, share these polls !

 

Best MySQL GUI client Tool

  • SQLyog (47%, 85 Votes)
  • MySQL Workbench (20%, 36 Votes)
  • HeidiSQL (8%, 14 Votes)
  • Navicat for MySQL (7%, 12 Votes)
  • MySQL Administrator / Query browser (6%, 10 Votes)
  • Other (5%, 9 Votes)
  • phpMyAdmin (4%, 8 Votes)
  • SQLWave (1%, 1 Votes)
  • SIDU DB Web GUI (1%, 1 Votes)
  • EMS SQL Manager for MySQL (1%, 1 Votes)
  • Sequel Pro (1%, 1 Votes)
  • sqldeveloper (1%, 1 Votes)
  • dbForge Studio (1%, 1 Votes)
  • MyQuery (1%, 1 Votes)
  • DBTools Manager (0%, 0 Votes)
  • SQL Maestro MySQL Tools (0%, 0 Votes)
  • Toad for MySQL (0%, 0 Votes)

Total Voters: 181

Loading ... Loading ...

 

Best MySQL Monitoring Tool

  • MONyog (35%, 135 Votes)
  • MySQL Enterprise Monitor (29%, 113 Votes)
  • Nagios (13%, 52 Votes)
  • Cacti (10%, 38 Votes)
  • Other (6%, 25 Votes)
  • Zabbix (5%, 19 Votes)
  • Ganglia (2%, 4 Votes)

Total Voters: 386

Loading ... Loading ...

 

Best MySQL Backup Tool (two possible answers)

  • Xtrabackup (46%, 151 Votes)
  • mysqldump (29%, 94 Votes)
  • MySQL Enterprise Backup (28%, 91 Votes)
  • mydumper (9%, 30 Votes)
  • LVM Backup (FS snapshot) (6%, 19 Votes)
  • mylvmbackup (5%, 17 Votes)
  • ZRM Enterprise (5%, 15 Votes)
  • ZRM Community (5%, 15 Votes)
  • Other (5%, 15 Votes)
  • Cold backup (2%, 6 Votes)
  • r1soft CDP for MySQL (1%, 2 Votes)

Total Voters: 329

Loading ... Loading ...

 

Best MySQL Replication/Cluster Tools (two possible answers)

  • MySQL Replication (37%, 183 Votes)
  • Tungsten Replicator (21%, 105 Votes)
  • MySQL NDB Cluster (17%, 83 Votes)
  • ShoonerSQL (16%, 76 Votes)
  • Galera Replication (13%, 66 Votes)
  • DRBD (9%, 45 Votes)
  • Other (2%, 11 Votes)
  • ScaleDB (2%, 9 Votes)

Total Voters: 490

Loading ... Loading ...

 

Best MySQL community Tool

  • Percona toolkit (37%, 117 Votes)
  • MySQL Sandbox (28%, 89 Votes)
  • MHA for MySQL (mysql-master-ha) (14%, 45 Votes)
  • Other (6%, 18 Votes)
  • Xtrabackup Manager (5%, 15 Votes)
  • mysqltuner (3%, 11 Votes)
  • Innotop (3%, 10 Votes)
  • mycheckpoint (2%, 5 Votes)
  • openark kit (1%, 3 Votes)
  • MMM (1%, 3 Votes)
  • Common schema (0%, 2 Votes)

Total Voters: 318

Loading ... Loading ...

 

Best MySQL Support Service

  • SkySQL (38%, 130 Votes)
  • Percona (28%, 96 Votes)
  • Oracle (26%, 90 Votes)
  • Other (4%, 15 Votes)
  • FromDual (3%, 9 Votes)
  • Pythian (1%, 4 Votes)
  • Blue Gecko (0%, 1 Votes)
  • PalominoDB (0%, 0 Votes)

Total Voters: 345

Loading ... Loading ...

 

Best MySQL based distribution

  • Oracle MySQL (36%, 122 Votes)
  • MariaDB (33%, 114 Votes)
  • Percona server (27%, 91 Votes)
  • Drizzle (3%, 11 Votes)
  • Other (1%, 4 Votes)
  • TokuDB (0%, 0 Votes)
  • InfiniDB (0%, 0 Votes)

Total Voters: 342

Loading ... Loading ...

 

Best NoSQL solution (two possible answers)

  • Memcached (46%, 120 Votes)
  • Hadoop / HBase (28%, 72 Votes)
  • MongoDB (25%, 65 Votes)
  • Cassandra (10%, 27 Votes)
  • Other (10%, 27 Votes)
  • HandlerSocket (9%, 24 Votes)
  • CouchDB (6%, 16 Votes)
  • Redis (5%, 13 Votes)
  • Tarantool (4%, 11 Votes)
  • Riak (3%, 7 Votes)
  • Amazon SimpleDB (2%, 5 Votes)
  • Tokyo Cabinet / Tyrant (1%, 3 Votes)

Total Voters: 259

Loading ... Loading ...

 

Best MySQL 5.6 improvement

  • Performances (27%, 77 Votes)
  • Replication (26%, 74 Votes)
  • Fulltext (16%, 47 Votes)
  • memcached API (13%, 36 Votes)
  • Backup/Restore buffer pool (8%, 22 Votes)
  • Partitioning (6%, 16 Votes)
  • Other (4%, 11 Votes)

Total Voters: 285

Loading ... Loading ...

 

Happy 2012 !
Cédric

This article is obviously not sponsored !
(MySQL is a trademark of Oracle Corporation and/or its affiliates)

Monitoring your monitoring tools (MONyog inside) !

Regardless of the monitoring tool you use to monitor your databases, it can be better to monitor this tool.
No, it’s not a joke ! Do you think you can have a benefit with a monitoring tool not connected to your servers ? ( without being alerted )

I choose to talk about MONyog here but this can apply to all existing monitoring tools.
I just want to share the message, the tool does not matter, so, do it !

So, let me explain how to control if you have fresh data with MONyog.
With MONyog it’s easy because it’s an agentless monitoring tool.

There are two ways to check that :

Per server general info :

 

For each server, you can find the last data updated date in the Monitors/Advisors tab.
The MySQL Availability in the dashboard is also usefull.

But it’s a per server information, so, that can be boring if you have a lot of databases servers.

General log file :

The default location for this file is MONyog_PATH/MONyog.log on your MONyog server.
You can find usefull informations there about MySQL, SFTP or SSH connexion failures for all your databases servers, in one place.

This log can also be used to detect client side errors if you don’t have access to the application servers.

Hope that can help, tell us what is the best method to do that with others monitoring tools like MySQL Enterprise Monitor.

Have a nice week-end.