Why a new memory engine may change everything ?
I’m sure you are aware that the last Percona server release includes a new improved MEMORY storage engine for MySQL.
This new engine is based on Dynamic Row Format and offers some of great features, specialy for VARCHAR, VARBINARY, TEXT and BLOB fields in MEMORY tables.
But because this new MEMORY engine by Percona has some limitations and because Percona server hasn’t used it for its internal temporary tables yet, I would like to talk about what can be the real benefits to have a brand new MEMORY engine based on Dynamic row format specialy for internal memory tables.
Just remember or discover how MySQL uses internal memory tables
And the MEMORY storage engine characteristics and its limitations
So, the memory storage engine transforms all varchar fields in char fields for internal temporary tables or for user created memory tables.
1. Let me explain what is the problem with a simple exemple :
I’ve created an InnoDB table (without index) with two varchar fields (50 & 100) :
mysql> show table status like 'test_memory5'\G *************************** 1. row *************************** Name: test_memory5 Engine: InnoDB Version: 10 Row_format: Compact Rows: 621089 Avg_row_length: 66 Data_length: 41484288 Max_data_length: 0 Index_length: 0 Data_free: 5242880 Auto_increment: NULL Create_time: 2011-09-21 13:10:17 Update_time: NULL Check_time: NULL Collation: latin1_swedish_ci Checksum: NULL Create_options: Comment:
The size of this table is about 48Mb (and more than 600.000 rows) :
-rw-rw---- 1 mysql mysql 48M 2011-09-21 13:11 test_memory5.ibd
Now, I’m creating a new memory table with exactly the same structure and I’m setting paramters for memory tables like this :
- set tmp_table_size=50*1024*1024;
- set max_heap_table_size=50*1024*1024;
That means I can create a 50Mb max memory table.
Thus, let me insert my 600.000 rows in this table :
mysql> insert into test_memory6 select * from test_memory5; ERROR 1114 (HY000): The table 'test_memory6' is full
My 50Mb memory table can’t contain the 48Mb of the InnoDB table !
Let’s try with 80Mb :
mysql> set tmp_table_size=80*1024*1024; Query OK, 0 rows affected (0.00 sec) mysql> set max_heap_table_size=80*1024*1024; Query OK, 0 rows affected (0.00 sec) mysql> insert into test_memory6 select * from test_memory5; ERROR 1114 (HY000): The table 'test_memory6' is full
And this error occurs until the memory tables can have a 110Mb maximum size !
Why ? Because the two varchar fields of the InnoDB table are converted in char fields with the memory storage engine.
Let’s see this example from MySQL documentation :
By the way, this is a very good reason to take care of your varchar fields.
Conclusion : A memory table can be really bigger than an InnoDB table
2. Let me explain why a new memory engine may change everything :
Changing the rules for memory tables may change everything for, at least, two reasons :
- VARCHAR, VARBINARY, TEXT and BLOB fields will be supported by this new engine for user created memory tables (Percona server can do it with restrictions)
- Internal memory tables could be more efficient with Dynamic Row Format
Extra: Using where; Using temporary; Using filesort
Thus, for each query using a temporary table, MySQL could use less of memory (RAM) !
I don’t know what would be the real benefit but I am convinced that it can be really significant.
I look forward to see more benchmark about that with last percona server release and I hope that Percona server or MariaDB will support dynamic row format for internal memory tables soon.
Please, let us know if you have already tested this new Percona Memory engine.