码迷,mamicode.com
首页 > 数据库 > 详细

020:InnoDB Buffer Pool

时间:2018-01-05 18:28:04      阅读:242      评论:0      收藏:0      [点我收藏+]

标签:rds   方法   show   热点   3.2   容量   读写   ase   超过   

一. 缓冲池(Buffer Pool)

1. 缓冲池介绍

  • 每次读写数据都是通过 Buffer Pool
  • Buffer Pool 中没有用户所需要的数据时,才去硬盘中获取;
  • 通过 innodb_buffer_pool_size进行设置总容量,该值设置的越大越好;
  • innodb_buffer_pool_instances 设置为多个缓冲池;

  • 总容量还是innodb_buffer_pool_size
  • 设置多个instance 可将热点打散,提高并发性能(建议设置成CPU个数值)

  • Buffer Pool也是以 页(page)为单位的,且大小和innodb_page_size一致;

2. Buffer Pool 性能测试

技术分享图片

  • 18G的测试数据,80M条记录;

  • Buffer Pool使用的内存超过数据库的大小 时,比如20G(库中所有数据都在内存中),此时的性能有了很大的提升;
  • 该图测试的是`TPS (每秒事物数),sysbench中一个事物由18条SQL语句组成,即这里的QPS为4.5W
  • 内存减少 10% ,性能下降 60%

3. Buffer Pool的管理

3.1 Buffer Pool 的组成

技术分享图片

1.Free List

  • Buffer Pool 刚启动时,有一个个16K的空白的页,这些页就存放(链表串联)在 Free List 中

2.LRU List

  • 当读取一个数据页的时候,就从 Free List 中取出一个页,存入数据,并将该页放入到 LRU List 中

3.Flush List

  • 当 LRU List 中的页 第一次 被修改了,就将该页的 指针(page number) 放入了 Flush List (只要修改过,就放入,不管修改几次)
  • Flush List 中包含脏页(数据经过修改,但是未刷入磁盘的页)
  • Flush list 中存放的不是一个页,而是页的指针(page number)

3.2 查看Buffer Pool的状态

  1. 使用命令 show engine innodb status\G 配合 pager less
mysql gcdb@localhost:(none)> show engine innodb status;
+--------+------+----------------------------------------------------------------------------------+
| Type   | Name | Status                                                                           |
+--------+------+----------------------------------------------------------------------------------+
| InnoDB |      |                                                                                  |
|        |      | =====================================                                            |
|        |      | 2018-01-05 14:18:27 0x7fa8b4649700 INNODB MONITOR OUTPUT                         |
|        |      | =====================================                                            |

-- ---------------省略其他输出-----------------

|        |      | ----------------------                                                           |
|        |      | BUFFER POOL AND MEMORY                                                           |
|        |      | ----------------------                                                           |
|        |      | Total large memory allocated 10994319360                                         |
|        |      | Dictionary memory allocated 14685357                                             |
|        |      | Buffer pool size   655280                                                        |
|        |      | Free buffers       648346                                                        |
|        |      | Database pages     6904                                                          |
|        |      | Old database pages 2662                                                          |
|        |      | Modified db pages  0                                                             |
|        |      | Pending reads      0                                                             |
|        |      | Pending writes: LRU 0, flush list 0, single page 0                               |
|        |      | Pages made young 2, not young 0                                                  |
|        |      | 0.00 youngs/s, 0.00 non-youngs/s                                                 |
|        |      | Pages read 6663, created 241, written 988                                        |
|        |      | 0.00 reads/s, 0.00 creates/s, 0.39 writes/s                                      |
|        |      | Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000        |
|        |      | Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s |
|        |      | LRU len: 6904, unzip_LRU len: 0                                                  |
|        |      | I/O sum[0]:cur[0], unzip sum[0]:cur[0]                                           |

--在输出的结果中可以定位到BUFFER POOL AND MEMORY ,以下部分显示了缓冲地的状态以及占用内存的情况:
? Total large memory allocated 10994319360
lnnoDB 所分配的内存总量为10994319360 字节。
? Dictionary memory allocated 14685357
数据字典内存区使用 14685357字节
? Buffer pool size 16384
缓冲地中页的数量,这里是 655280 个页,总共占用10G 内存( 16K*655280) 。
? Free buffers 648346
缓冲地中free 链表中空闲页的数量。
? Database pages 41
Buffer Pool中使用了多少页(LRU List)
? Old database pages 2662
最近不被访问的数据
? Modified db pages
赃页的页数
? Pending reads 0
正在读取的页的数量,这里为0 。
? Pending writes: LRU 0, flush list 0, single page 0
正在进行刷新页的数量,刷新可以分为LRU 、flush list 、single page 三种类型,这在后面的
? Pages made young 2, not young 0
 0.00 youngs/s, 0.00 non-youngs/s
 -- young表示old-->new的状态
?  Pages read 6663, created 241, written 988
缓冲地中已经读取、创建和刷新页的次数,这里分别为6663 、241998 。
?  0.00 reads/s, 0.00 creates/s, 0.39 writes/s
过去一段时间内,每秒页的读取、也IJ 建和刷新的次数。注意: SHOW INNODB STATUS 输出
的是过去一段时间内的结果。
? Buffer pool hit rate 1000 / 1000
缓冲地的命中率,这是监控最为关注的一个性能指标。命中率越高,数据库的性能越好,这里为1 00% ,表示缓冲地能够缓存所有InnoDB 存储引擎表。

|        |      | ----------------------                                                           |
|        |      | INDIVIDUAL BUFFER POOL INFO                                                      |
|        |      | ----------------------                                                           |
|        |      | ---BUFFER POOL 0                                                                 |
|        |      | Buffer pool size   81910                                                         |
|        |      | Free buffers       80996                                                         |
|        |      | Database pages     910                                                           |
|        |      | Old database pages 339                                                           |
|        |      | Modified db pages  0                                                             |
|        |      | Pending reads      0                                                             |
|        |      | Pending writes: LRU 0, flush list 0, single page 0                               |
|        |      | Pages made young 0, not young 0                                                  |
|        |      | 0.00 youngs/s, 0.00 non-youngs/s                                                 |
|        |      | Pages read 865, created 45, written 104                                          |
|        |      | 0.00 reads/s, 0.00 creates/s, 0.00 writes/s                                      |
|        |      | Buffer pool hit rate 1000 / 1000, young-making rate 0 / 1000 not 0 / 1000        |
|        |      | Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s |
|        |      | LRU len: 910, unzip_LRU len: 0                                                   |
|        |      | I/O sum[0]:cur[0], unzip sum[0]:cur[0]                                           |
--同上
-- ---------------省略其他输出-----------------

|        |      | ----------------------------                                                     |
|        |      | END OF INNODB MONITOR OUTPUT                                                     |
|        |      | ============================                                                     |
|        |      |                                                                                  |
+--------+------+----------------------------------------------------------------------------------+
(END)
  1. 使用元数据表 information_schema.INNODB_BUFFER_POOL_STATS
mysql gcdb@localhost:(none)> select * from information_schema.INNODB_BUFFER_POOL_STATS\G
***************************[ 1. row ]***************************
POOL_ID                          | 0
POOL_SIZE                        | 81910
FREE_BUFFERS                     | 80996
DATABASE_PAGES                   | 910
OLD_DATABASE_PAGES               | 339
MODIFIED_DATABASE_PAGES          | 0
PENDING_DECOMPRESS               | 0
PENDING_READS                    | 0
PENDING_FLUSH_LRU                | 0
PENDING_FLUSH_LIST               | 0
PAGES_MADE_YOUNG                 | 0
PAGES_NOT_MADE_YOUNG             | 0
PAGES_MADE_YOUNG_RATE            | 0.0
PAGES_MADE_NOT_YOUNG_RATE        | 0.0
NUMBER_PAGES_READ                | 865
NUMBER_PAGES_CREATED             | 45
NUMBER_PAGES_WRITTEN             | 104
PAGES_READ_RATE                  | 0.0
PAGES_CREATE_RATE                | 0.0
PAGES_WRITTEN_RATE               | 0.0
NUMBER_PAGES_GET                 | 69856
HIT_RATE                         | 0
YOUNG_MAKE_PER_THOUSAND_GETS     | 0
NOT_YOUNG_MAKE_PER_THOUSAND_GETS | 0
NUMBER_PAGES_READ_AHEAD          | 768
NUMBER_READ_AHEAD_EVICTED        | 0
READ_AHEAD_RATE                  | 0.0
READ_AHEAD_EVICTED_RATE          | 0.0
LRU_IO_TOTAL                     | 0
LRU_IO_CURRENT                   | 0
UNCOMPRESS_TOTAL                 | 0
UNCOMPRESS_CURRENT               | 0
***************************[ 2. row ]***************************
POOL_ID                          | 1
POOL_SIZE                        | 81910
FREE_BUFFERS                     | 80843
DATABASE_PAGES                   | 1063
OLD_DATABASE_PAGES               | 400
MODIFIED_DATABASE_PAGES          | 0
PENDING_DECOMPRESS               | 0
PENDING_READS                    | 0

-- ---------------省略其他输出-----------------

mysql gcdb@localhost:(none)> select * from information_schema.INNODB_BUFFER_PAGE_LRU limit 1\G
***************************[ 1. row ]***************************
POOL_ID             | 0
LRU_POSITION        | 0
SPACE               | 0                 -- space id 表空间号
PAGE_NUMBER         | 7                 -- 对应的页号
PAGE_TYPE           | SYSTEM
FLUSH_TYPE          | 1
FIX_COUNT           | 0
IS_HASHED           | NO
NEWEST_MODIFICATION | 32993864040       -- 该页最近一次(最新)被修改的LSN值
OLDEST_MODIFICATION | 0                 -- 该页在Buffer Pool中第一次被修改的LSN值,FLushList是根据该值进行排序的
                                        -- 该值越小,表示该页应该最先被刷新
ACCESS_TIME         | 2688054668
TABLE_NAME          | <null>
INDEX_NAME          | <null>
NUMBER_RECORDS      | 0
DATA_SIZE           | 0
COMPRESSED_SIZE     | 0
COMPRESSED          | NO
IO_FIX              | IO_NONE
IS_OLD              | YES
FREE_PAGE_CLOCK     | 0
1 row in set
Time: 0.143s
mysql gcdb@localhost:(none)>

-----------------省略其他输出-----------------

3.3 Buffer Pool 在线调整

  • 从 MySQL 5.7 开始,可以在线修改 innodb_buffer_pool_size
mysql gcdb@localhost:(none)> show variables like "%innodb_buffer_pool_size%";
+-------------------------+-------------+
| Variable_name           | Value       |
+-------------------------+-------------+
| innodb_buffer_pool_size | 10737418240 |    -- innodb_buffer_pool_size为10G
+-------------------------+-------------+
1 row in set
Time: 0.012s
mysql gcdb@localhost:(none)> set global innodb_buffer_pool_size=16*1024*1024*1024;    --调整为innodb_buffer_pool_size为16G
Query OK, 0 rows affected
Time: 0.008s
mysql gcdb@localhost:(none)> show variables like "%innodb_buffer_pool_size%";
+-------------------------+-------------+
| Variable_name           | Value       |
+-------------------------+-------------+
| innodb_buffer_pool_size | 17179869184 |   --调整为innodb_buffer_pool_size为16G
+-------------------------+-------------+
1 row in set
Time: 0.012s
mysql gcdb@localhost:(none)> show engine innodb status;
+--------+------+----------------------------------------------------------------------------------+
| Type   | Name | Status                                                                           |
+--------+------+----------------------------------------------------------------------------------+
| InnoDB |      |                                                                                  |
|        |      | =====================================                                            |
|        |      | 2018-01-05 15:23:16 0x7fa8b4649700 INNODB MONITOR OUTPUT                         |
|        |      | =====================================                                            |
--  ---------------省略其他输出-----------------
|        |      | ----------------------                                                           |
|        |      | BUFFER POOL AND MEMORY                                                           |
|        |      | ----------------------                                                           |
|        |      | Total large memory allocated 17590910976                                         | --lnnoDB 所分配的内存总量为17590910976字节
|        |      | Dictionary memory allocated 14685357                                             |
|        |      | Buffer pool size   1048496                                                       |
|        |      | Free buffers       1041477                                                       |
|        |      | Database pages     7019                                                          |
|        |      | Old database pages 2662                                                          |
|        |      | Modified db pages  0                                                             |
|        |      | Pending reads      0                                                             |
|        |      | Pending writes: LRU 0, flush list 0, single page 0                               |
|        |      | Pages made young 2, not young 0                                                  |
|        |      | 0.00 youngs/s, 0.00 non-youngs/s                                                 |
|        |      | Pages read 6663, created 356, written 1158                                       |
|        |      | 0.00 reads/s, 0.00 creates/s, 0.00 writes/s                                      |
|        |      | No buffer pool page gets since the last printout                                 |
|        |      | Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s |
|        |      | LRU len: 7019, unzip_LRU len: 0                                                  |
|        |      | I/O sum[0]:cur[0], unzip sum[0]:cur[0]                                           |
|        |      | ----------------------                                                           |
|        |      | INDIVIDUAL BUFFER POOL INFO                                                      |
|        |      | ----------------------                                                           |
|        |      | ---BUFFER POOL 0                                                                 |
|        |      | Buffer pool size   131062                                                        | --BUFFER POOL 0 原先分配 81910页变为131062
|        |      | Free buffers       130152                                                        |
|        |      | Database pages     910                                                           |
|        |      | Old database pages 339                                                           |
|        |      | Modified db pages  0                                                             |
|        |      | Pending reads      0                                                             |
|        |      | Pending writes: LRU 0, flush list 0, single page 0                               |
|        |      | Pages made young 0, not young 0                                                  |
|        |      | 0.00 youngs/s, 0.00 non-youngs/s                                                 |
|        |      | Pages read 865, created 45, written 104                                          |
|        |      | 0.00 reads/s, 0.00 creates/s, 0.00 writes/s                                      |
|        |      | No buffer pool page gets since the last printout                                 |
|        |      | Pages read ahead 0.00/s, evicted without access 0.00/s, Random read ahead 0.00/s |
|        |      | LRU len: 910, unzip_LRU len: 0                                                   |
|        |      | I/O sum[0]:cur[0], unzip sum[0]:cur[0]                                           |
|        |      | ---BUFFER POOL 1                                                                 |
|        |      | Buffer pool size   131062                                                        |
|        |      | Free buffers       129999                                                        |
|        |      | Database pages     1063                                                          |
|        |      | Old database pages 400                                                           |
|        |      | Modified db pages  0                                                             |
|        |      | Pending reads      0                                                             |


mysql gcdb@localhost:(none)> set global innodb_buffer_pool_size=8*1024*1024*1024;   -- 缩小,没修改的页被丢弃,修改的需要刷回磁盘
Query OK, 0 rows affected
Time: 0.001s

mysql gcdb@localhost:(none)> show variables like "%innodb_buffer_pool_size%";
+-------------------------+------------+
| Variable_name           | Value      |
+-------------------------+------------+
| innodb_buffer_pool_size | 8589934592 |
+-------------------------+------------+
1 row in set
Time: 0.012s
mysql gcdb@localhost:(none)>
  • MySQL 5.7之前的版本,修改innodb_buffer_pool_size,需要重启

3.4 LRU List 的管理

  • 3.4.1使用mid point 的LRU算法

  • LRU是Least Recently Used的缩写,即最近最久未使用,常用于页面置换算法,是为虚拟页式存储管理服务的。
  • 当该页被第一次读取 时,将该页先放在 mid point的位置(因为无法保证一定是活跃);
  • 取当被读到第二次 时,才将改页放入到 new page 的首部;
  • innodb_old_blocks_pct 参数控制 mid point 的位置,默认是 37 ,即 3/8 的位置

技术分享图片

LRU 中new page和old page是在一个链表上的,想像成排队,访问多的就从mid point排到了链表的前面然后后冷的页就慢慢被挤到了old page中,如果old中的数据继续被多次访问,还是会回到new中

1 : mid --> new

2 : mid --> old --> new

3 : mid --> old --> 刷回磁盘

4 : new --> old --> 刷回磁盘

  • 当Free List中没有空余的页时,就需要从 old page 中最后的页(被淘汰的页)给取出,给新的查询所使用
  • 如果被淘汰的页是脏页(page number在Flush List中),则需要先刷回磁盘后,再给新的查询使用
mysql gcdb@localhost:(none)> show variables like "%innodb_old_blocks_pct%"
+-----------------------+-------+
| Variable_name         | Value |
+-----------------------+-------+
| innodb_old_blocks_pct | 37    |   --调整为innodb_buffer_pool_size为8G
+-----------------------+-------+
1 row in set
Time: 0.012s
  • 避免扫描语句污染LRU
  • 当使用 select * from tablename; 时,该语句会读取某个页很多次(即该页可能被读取了两次以上,读取一条记录,就需要读一次页 )
  • innodb_old_blocks_time
mysql gcdb@localhost:(none)> show variables like"%innodb_old_blocks_time%"
+------------------------+-------+
| Variable_name          | Value |
+------------------------+-------+
| innodb_old_blocks_time | 1000  |   --设置为1s
+------------------------+-------+
1 row in set
Time: 0.012s
mysql gcdb@localhost:(none)>
  1. 当该页被第一次 读取时,将该页放在mid point 位置,但是随后无论你读多少次 ,我在这 innodb_old_blocks_time 的时间内都不管(都视作只读取了一次 ),等这个时间过去了(时间到),如果该页还是被读取了,我才把这个页放到 new page 的首部。

  2. 通常 select * 扫描操作不会高于1秒,一个页很快就被扫完了。

4. Buffer Pool 的预热

  • Buffer Pool预热

  • 在MySQL启动后(MySQL5.6之前),Buffer Pool中页的数据是空的,需要大量的时间才能把磁盘中的页读入到内存中,导致启动后的一段时间性能很差。

  • 使用该方法预热,强制扫描,将数据刷入buffer pool,但是不能真正将热点数据放入buffer pool ;select count(1) from table force index(PRIMARY) ;select count(1) from table FORCE index(index name);

  • 在 MySQL 5.6 以后,可以在 停机 的时候 dumpbuffer pool 的数据(space,page number),然后在 启动 的时候 Loadbuffer pool,该功能可以让MySQL启动时 自动预热 ,无需人工干预。

mysql gcdb@localhost:(none)> show variables like "%innodb_buffer_pool%";
+-------------------------------------+----------------+
| Variable_name                       | Value          |
+-------------------------------------+----------------+
| innodb_buffer_pool_chunk_size       | 134217728      |
| innodb_buffer_pool_dump_at_shutdown | ON             | -- 在停机时dump出buffer pool中的(space,page)
| innodb_buffer_pool_dump_now         | OFF            | -- set 一下,表示现在就从buffer pool中dump
| innodb_buffer_pool_dump_pct         | 25             | -- dump的百分比,是每个buffer pool文件,而不是整体
| innodb_buffer_pool_filename         | ib_buffer_pool | -- dump出的文件的名字
| innodb_buffer_pool_instances        | 8              |
| innodb_buffer_pool_load_abort       | OFF            |
| innodb_buffer_pool_load_at_startup  | ON             | -- 启动时加载dump的文件,恢复到buffer pool中
| innodb_buffer_pool_load_now         | OFF            | -- set一下,表示现在加载 dump的文件
| innodb_buffer_pool_size             | 8589934592     |
+-------------------------------------+----------------+
10 rows in set
Time: 0.013s
mysql gcdb@localhost:(none)>
[root@localhost-m(252) /r2/mysqldata]# head ib_buffer_pool   --dump出来的文件
2,7560
2,7557
2,7552
2,7100
2,7096
2,7092
2,7090
2,7084
2,7082
2,7077
[root@localhost-m(252) /r2/mysqldata]#

1.dump的越多,启动的越慢
2.频繁的手工dump( set innodb_buffer_pool_dump_now = 1 ),会导致Buffer Pool中的数据越来越少,是因为设置了 innodb_buffer_pool_dump_pct
3.如果做了高可用,可以定期dump,然后将该dump的文件传送到slave上,然后直接load( set innodb_buffer_pool_load_now = 1 )``(slave上的(Space,Page)和Master上的 大致相同 )

  1. load now 和 dumpnow都是 异步在后台加载的,返回的速度很
--
--mysql 启动
--
sehll> cat error.log
## ---------------省略其他输出-----------------
2017-11-24T10:45:22.008199+08:00 0 [Note] InnoDB: Loading buffer pool(s) from /r2/mysqldata/ib_buffer_pool 
## ---------------省略其他输出-----------------
2017-11-24T10:45:25.716362+08:00 0 [Note] InnoDB: Buffer pool(s) load completed at 171124 10:45:25 --速度还是很快的
--
--mysql 停机
--
shell> cat error.log
## ---------------省略其他输出-----------------
2017-12-29T10:31:47.844235+08:00 0 [Note] InnoDB: Dumping buffer pool(s) to /r2/mysqldata/ib_buffer_pool --dump buffer 
2017-12-29T10:31:47.844597+08:00 0 [Note] InnoDB: Buffer pool(s) dump completed at 171229 10:31:47
## ---------------省略其他输出-----------------
  • 查看当前buffer pool中的数据的条数
[root@localhost-m(252) /r2/mysqldata]#  wc -l ib_buffer_pool
129 ib_buffer_pool
mysql gcdb@localhost:(none)> set global innodb_buffer_pool_dump_now=1;
Query OK, 0 rows affected
Time: 0.001s
mysql gcdb@localhost:(none)>  show status like 'Innodb_buffer_pool_dump_status';
+--------------------------------+--------------------------------------------------+
| Variable_name                  | Value                                            |
+--------------------------------+--------------------------------------------------+
| Innodb_buffer_pool_dump_status | Buffer pool(s) dump completed at 180105 17:49:54 |
+--------------------------------+--------------------------------------------------+
1 row in set
Time: 0.011s
mysql gcdb@localhost:(none)>
-- 已经完成
[root@localhost-m(252) /r2/mysqldata]# wc -l ib_buffer_pool
1751 ib_buffer_pool   --变为1751条
  • innodb_buffer_pool_dump_pct

    该百分比(N<100)不是你当前buffer pool总的数据(总页数)N%,而是你每个buffer pool实例中最近使用的页N%

if there are 4 buffer pools with 100 pages each, and innodb_buffer_pool_dump_pct is set to 25, the 25 most recently used pages from each buffer pool are dumped

020:InnoDB Buffer Pool

标签:rds   方法   show   热点   3.2   容量   读写   ase   超过   

原文地址:https://www.cnblogs.com/gczheng/p/8206122.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!