InnoDB has an oft-unused parameter innodb_concurrency_tickets that seems widely misunderstood. From the docs: “The number of threads that can enter InnoDB concurrently is determined by the innodb_thread_concurrency variable. A thread is placed in a queue when it tries to enter InnoDB if the number of threads has already reached the concurrency limit. When a thread is allowed to enter InnoDB, it is given a number of “free tickets†equal to the value of innodb_concurrency_tickets, and the thread can enter and leave InnoDB freely until it has used up its tickets. After that point, the thread again becomes subject to the concurrency check (and possible queuing) the next time it tries to enter InnoDB. The default value is 500…”
What this means from a practical perspective is that each query is allocated 500 tickets when it begins executing. Each time it enters InnoDB, this number is decremented until it reaches zero (“entering InnoDB” appears only to occur when a row is accessed). When it reaches zero, it may-or-may-not be put into a queue and wait to continue execution. InnoDB doesn’t provide us a way in which to determine how many concurrency tickets a query uses, making this parameter notoriously difficult to tune. It is important to note that this variable only comes in to play when innodb_thread_concurrency is greater than zero.
On a stock install of MySQL, here are some example queries and the corresponding number of concurrency tickets used for each:
mysql> CREATE TABLE test_table (
-> id int
-> ) ENGINE=InnoDB; -- 0 Tickets Used
Query OK, 0 rows affected (0.36 sec)
mysql> INSERT INTO test_table (id) values (1); -- 0 Tickets Used
Query OK, 1 row affected (0.00 sec)
mysql> SELECT * FROM test_table; -- 1 Ticket Used
+------+
| id |
+------+
| 1 |
+------+
1 row in set (0.00 sec)
mysql> INSERT INTO test_table (id) values (2),(3); -- 0 Tickets Used
Query OK, 2 rows affected (0.00 sec)
Records: 2 Duplicates: 0 Warnings: 0
mysql> SELECT COUNT(*) FROM test_table; -- 3 Tickets Used
+----------+
| COUNT(*) |
+----------+
| 3 |
+----------+
1 row in set (0.00 sec)
mysql> UPDATE test_table SET id=4 WHERE id=1; -- 4 Tickets Used (because no index, a table scan is performed)
Query OK, 1 row affected (0.00 sec)
Rows matched: 1 Changed: 1 Warnings: 0
mysql> ALTER TABLE test_table ADD INDEX (id); -- 5 Tickets Used
Query OK, 3 rows affected (0.01 sec)
Records: 3 Duplicates: 0 Warnings: 0And now on to a more interesting scenario: foreign keys
mysql> CREATE TABLE parent (id INT NOT NULL,
-> PRIMARY KEY (id)
-> ) ENGINE=INNODB; -- 0 Tickets Used
Query OK, 0 rows affected (0.01 sec)
mysql> CREATE TABLE child (id INT, parent_id INT,
-> INDEX par_ind (parent_id),
-> FOREIGN KEY (parent_id) REFERENCES parent(id)
-> ON DELETE CASCADE
-> ) ENGINE=INNODB; -- 0 Tickets Used
Query OK, 0 rows affected (0.00 sec)
mysql> INSERT INTO parent (id) VALUES (1),(2),(3),(4); -- 3 Tickets Used
Query OK, 4 rows affected (0.03 sec)
mysql> INSERT INTO child (id, parent_id) VALUES (1,1),(1,1),(2,1); -- 2 Tickets Used
Query OK, 3 rows affected (0.00 sec)
Records: 3 Duplicates: 0 Warnings: 0
mysql> DELETE FROM child WHERE 1; -- 6 Tickets Used
Query OK, 3 rows affected (0.02 sec)
mysql> ALTER TABLE `child` ADD PRIMARY KEY (`id`,`parent_id`); -- 0 Tickets Used
Query OK, 0 rows affected (0.02 sec)
Records: 0 Duplicates: 0 Warnings: 0
mysql> INSERT INTO `child` (`id`,`parent_id`) VALUES (1,1), (1,2), (2,1),(2,2); -- 3 Tickets Used
Query OK, 4 rows affected (0.01 sec)
Records: 4 Duplicates: 0 Warnings: 0So, how can we put this into practice, since this information isn’t available to most users?
INSERT w/PRIMARY KEY defined: Number of rows inserted – 1
INSERT w/FOREIGN KEY constraint: Number of rows inserted – 1
SELECT: 1 ticket per row returned
UPDATE: 1 ticket per row examined + 1 ticket per row updated
DELETE: 1 ticket per row examined + 1 ticket per row deleted
ALTER: (2 * rows in the table) – 1
As with any performance optimization effort, you will want to optimize for the common case. If you have a very simple workload, you can calculate these values by hand. But for most workloads with a complex access pattern, we’ll need to estimate or wait for InnoDB to expose this information to us.
What happens in the case where I have two distinct access patterns: single row primary-key lookups and SELECT statements that examine 900 rows? If innodb_concurrency_tickets is set to 500, then all of the single row PK lookups will execute without ever being subject to an additional concurrency check (there is always one when a thread first enters InnoDB) while the 900-row SELECT statements will always be subject to one additional concurrency check (we actually care less about the concurrency check itself than the possibility that it may become queued). Your first instinct may be to increase innodb_concurrency_tickets to >=900 in this case, but that isn’t necessarily the best decision. As stated in the docs, the number of threads that can enter InnoDB is limited by innodb_thread_concurrency (which is why these two variables are most often tuned in concert). To continue the example, if innodb_thread_concurrency is set to 8 and eight 900-row-SELECT statements come in, they will effectively block the PK lookups until one of them is subject to a concurrency check or complete execution and exit InnoDB. If innodb_concurrency_tickets had been increased to >= 900, then ALL of the PK lookups would be blocked until the 900-row-SELECT statements complete execution.
With a maximum value of 4,294,967,295 this has the potential to block other queries for a significant amount of time. Setting innodb_concurrency_tickets too high can have startlingly negative performance implications. On the other hand, if we determine that 99% of the traffic are these single row PK lookups and only 1% are the 900-row SELECTs, we may be tempted to lower the setting to 1 to accommodate the “typical case”. The effects of this, though, would be to cause the 900-row SELECT statements to be subject to 899 concurrency checks. This means 899 potential opportunities to be queued! So, as with most other parameters, this is a balancing act.
It really comes down to the importance of the applicable queries. Imagine those 900-row SELECT statements were actually 10,000 row selects, this would become a more pressing issue. If they are reporting queries used only internally, then it is not so much of an issue and you can leave innodb_concurrency_tickets rather small. If, on the other hand, these are the queries that lead to revenue generation, you may want to give them a bit more dedicated CPU time so they execute that much faster (even at the expense of the PK lookups). In other words, if you’re optimizing for throughput in this scenario, you will tune innodb_concurrency_tickets to the 99th percentile of small PK lookups. If you’re optimizing for response time, you would set it larger to accommodate the larger (important) select statements.
A quick sysbench run gives us the following results (X-axis is innodb_concurrency_tickets, Y-axis is txn/sec. More is better). Since all sysbench queries are 10 rows or less, we don’t really expect to see much of a difference here:
Details:
sysbench --test=oltp --oltp-table-size=80000000 --oltp-read-only=off --init-rng=on --num-threads=16 --max-requests=0 --oltp-dist-type=uniform --max-time=300 --mysql-user=root --mysql-socket=/var/lib/mysql/mysql.sock run
Applicable my.cnf settings:
innodb_buffer_pool_size=24G innodb_data_file_path=ibdata1:10M:autoextend innodb_file_per_table=1 innodb_flush_log_at_trx_commit = 1 innodb_log_buffer_size = 8M innodb_log_files_in_group=2 innodb_log_file_size=1900M innodb_thread_concurrency=16 innodb_flush_method = O_DIRECT innodb_write_io_threads=8 innodb_read_io_threads=8 innodb_io_capacity=500 innodb_max_dirty_pages_pct=90 max_connections=3000 query_cache_size=0 skip-name-resolve table_cache=10000
The post Tuning InnoDB Concurrency Tickets appeared first on MySQL Performance Blog.