Why a new memory engine may change everything ?

I’m sure you are aware that the last Percona server release includes a new improved MEMORY storage engine for MySQL.
This new engine is based on Dynamic Row Format and offers some of great features, specialy for VARCHAR, VARBINARY, TEXT and BLOB fields in MEMORY tables.

But because this new MEMORY engine by Percona has some limitations and because Percona server hasn’t used it for its internal temporary tables yet, I would like to talk about what can be the real benefits to have a brand new MEMORY engine based on Dynamic row format specialy for internal memory tables.

Just remember or discover how MySQL uses internal memory tables

And the MEMORY storage engine characteristics and its limitations

So, the memory storage engine transforms all varchar fields in char fields for internal temporary tables or for user created memory tables. 

1. Let me explain what is the problem with a simple exemple :

I’ve created an InnoDB table (without index) with two varchar fields (50 & 100) :

mysql> show table status like 'test_memory5'\G
*************************** 1. row ***************************
           Name: test_memory5
         Engine: InnoDB
        Version: 10
     Row_format: Compact
           Rows: 621089
 Avg_row_length: 66
    Data_length: 41484288
Max_data_length: 0
   Index_length: 0
      Data_free: 5242880
 Auto_increment: NULL
    Create_time: 2011-09-21 13:10:17
    Update_time: NULL
     Check_time: NULL
      Collation: latin1_swedish_ci
       Checksum: NULL

The size of this table is about 48Mb (and more than 600.000 rows) :

-rw-rw---- 1 mysql mysql  48M 2011-09-21 13:11 test_memory5.ibd

Now, I’m creating a new memory table with exactly the same structure and I’m setting paramters for memory tables like this :

  • set tmp_table_size=50*1024*1024;
  • set max_heap_table_size=50*1024*1024;

That means I can create a 50Mb max memory table.
Thus, let me insert my 600.000 rows in this table :

mysql> insert into test_memory6 select * from test_memory5;
ERROR 1114 (HY000): The table 'test_memory6' is full

My 50Mb memory table can’t  contain the 48Mb of the InnoDB table !
Let’s try with 80Mb :

mysql> set tmp_table_size=80*1024*1024;
Query OK, 0 rows affected (0.00 sec)
mysql> set max_heap_table_size=80*1024*1024;
Query OK, 0 rows affected (0.00 sec)
mysql> insert into test_memory6 select * from test_memory5;
ERROR 1114 (HY000): The table 'test_memory6' is full

And this error occurs until the memory tables can have a 110Mb maximum size !

Why ? Because the two varchar fields of the InnoDB table are converted in char fields with the memory storage engine.
Let’s see this example from MySQL documentation :

Value CHAR(4) Storage Required VARCHAR(4) Storage Required
'' '    ' 4 bytes '' 1 byte
'ab' 'ab  ' 4 bytes 'ab' 3 bytes
'abcd' 'abcd' 4 bytes 'abcd' 5 bytes
'abcdefgh' 'abcd' 4 bytes 'abcd' 5 bytes

By the way, this is a very good reason to take care of your varchar fields.

Conclusion : A memory table can be really bigger than an InnoDB table

2. Let  me explain why a new memory engine may change everything :

Changing the rules for memory tables may change everything for, at least, two reasons :

  • VARCHARVARBINARYTEXT and BLOB fields will be supported by this new engine for user created memory tables (Percona server can do it with restrictions)
  • Internal memory tables could be more efficient with Dynamic Row Format
The real benefits will come with internal memory tables, how often do you see that when you “explain” your queries :
Extra: Using where; Using temporary; Using filesort

Thus, for each query using a temporary table, MySQL could use less of memory (RAM) !
I don’t know what would be the real benefit but I am convinced that it can be really significant.

I look forward to see more benchmark about that with last percona server release and I hope that Percona server or MariaDB will support dynamic row format for internal memory tables soon.

Please, let us know if you have already tested this new Percona Memory engine.

[UPDATED] Why you should go to Percona Live (London) ?

[ UPDATED 2011-09-22 ] :
For French attendees : Vous participez ou souhaitez participer à cette conférence, rejoignez le groupe de la communauté francophone pour partager autour de cet évènement
For All : Download the take away PDF of the conference here : PLL2011.pdf
For All : Download the schedules : Percona_Live_London2011.ics

I don’t want to debate about the next MySQL Conference here, you can find a great summary post here : http://rpbouman.blogspot.com/2011/08/regarding-mysql-conference-and-expo.html

But just consider the Robert’s point of view which I fully agree with  : Why not put the energy into organizing a Fall 2012 MySQL Community Conference in Europe ?

I just want to talk about the Percona Live MySQL Conference whose the next summit will be the 24th & 25th of October and explain why you should go there (But it’s just an advice)

MySQL is not only a product owned by Oracle, this is a great community, and this community provides some of the best informations about MySQL and its ecosystem.
You can read every days a lot of useful informations and tips about MySQL and these informations are provided by the community.

So, let me explain why the Percona Live is a great place to be :

  • There are several conferences almost all over the world : NY,  Santa Clara, San Francisco, London and more to come !
  • ( I just would like to say that this is not a “San Francisco” only conference, thanks to Percona for that ! )
  • You can easily spend two days for a conference close to you ( and so don’t give up your job and your family for a long time )
  • This is an easy way to meet other members of the MySQL Community
  • These others members work close to you. You can easily keep in touch after the conference ( and you must do it ! )
  • You can meet people behind the blogs you read every days
  • There isn’t only speakers from Percona but speakers from some of the most innovative companies with MySQL
  • Be sure that the tutorial and conference will be at a very high level
  • And finally, the price is really affordable ( but not the hotel in London, sorry )
  • Your comments…

Be sure that I write this post independently and without ever having attended such a conference.
Just read here my expectations about the next Percona Live in London.
Glad to meet you there !

Stay tuned for my feeling after the conference…