码迷,mamicode.com
首页 > 数据库 > 详细

Mysql:5.5-5.6:Components & Plugins

时间:2020-03-13 21:04:57      阅读:85      评论:0      收藏:0      [点我收藏+]

标签:ttl   hang   instr   result   desire   deadlock   run   permanent   reject   

 

5.5 MySQL Server Components

MySQL Server includes a component-based infrastructure for extending server capabilities. A component provides services that are available to the server and other components. (With respect to service use, the server is a component, equal to other components.) Components interact with each other only through the services they provide.

MySQL distributions include several components that implement server extensions:

System and status variables implemented by a server component are exposed when the component is installed and have names that begin with a component-specific prefix. For example, the log_filter_dragnet error log filter component implements a system variable named log_error_filter_rules, the full name of which is dragnet.log_error_filter_rules. To refer to this variable, use the full name.

The following sections describe how to install and uninstall components, and how to determine at runtime which components are installed and obtain information about them.

For information about the internal implementation of components, see the MySQL Server Doxygen documentation, available at https://dev.mysql.com/doc/index-other.html. For example, if you intend to write your own components, this information is important for understanding how components work.

5.5.1 Installing and Uninstalling Components

Server components must be loaded into the server before they can be used. MySQL supports component loading at runtime.

The INSTALL COMPONENT and UNINSTALL COMPONENT SQL statements enable component loading and unloading. For example:

INSTALL COMPONENT ‘file://component_validate_password‘;
UNINSTALL COMPONENT ‘file://component_validate_password‘;

A loader service handles component loading and unloading, and also lists loaded components in the component table of the mysql system database that serves as a registry.

The SQL statements for component manipulation affect server operation and the mysql.component system table as follows:

  • INSTALL COMPONENT loads components into the server. The components become active immediately. The loader service also registers loaded components in the mysql.component system table. For subsequent server restarts, the loader service loads any components listed in mysql.component during the startup sequence. This occurs even if the server is started with the --skip-grant-tables option.

  • UNINSTALL COMPONENT deactivates components and unloads them from the server. The loader service also unregisters the components from the mysql.component system table so that they are no longer loaded during the startup sequence for subsequent server restarts.

Compared to the corresponding INSTALL PLUGIN statement for server plugins, the INSTALL COMPONENT statement for components offers the significant advantage that it is not necessary to know any platform-specific file name suffix for naming the component. This means that a given INSTALL COMPONENT statement can be executed uniformly across platforms.

5.5.2 Obtaining Server Component Information

The component table in the mysql system database contains information about currently loaded components and shows which components have been registered with INSTALL COMPONENT. To see which components are installed, use this statement:

SELECT * FROM mysql.component;

5.5.3 Error Log Components

This section describes the characteristics of individual error log components. For general information about configuring error logging, see Section 5.4.2, “The Error Log”.

A log component can be a filter or a sink:

  • A filter processes log events, to add, remove, or modify event fields, or to delete events entirely. The resulting events pass to the next log component named in the log_error_services system variable value.

  • A sink is a destination (writer) for log events. Typically, a sink processes log events into log messages that have a particular format and writes these messages to its associated output, such as a file or the system log.

The server executes filters and sinks in the log_error_services value in the order they are named. The rightmost component should therefore be a sink. If the rightmost component is a filter, any changes it has on events have no effect on output.

The following sections describe individual log components, grouped by component type:

Component descriptions include these types of information:

  • The component name and intended purpose.

  • Whether the component is built in or must be loaded. For a loadable component, the description specifies the URN to use to load and unload the component with the INSTALL COMPONENT and UNINSTALL COMPONENT statements.

  • Whether the component can be listed multiple times in the log_error_services value.

  • For a sink component, the destination to which the component writes output.

Error Log Filter Components

Error log filter components implement filtering of error log events:

  • If no filter component is enabled, no filtering occurs.

  • Any enabled filter component affects log events only for components listed later in the log_error_services value. In particular, for any log sink component listed in log_error_services earlier than any filter component, no log event filtering occurs.

The log_filter_internal Component

If log_filter_internal is disabled, log_error_verbosity and log_error_suppression_list have no effect.

The log_filter_dragnet Component

Error Log Sink Components

Error log sink components are writers that implement error log output. If no sink component is enabled, no log output occurs.

Some sink component descriptions refer to the default error log destination. This is the console or a file and is indicated by the fault of the log_error system variable, determined as described in Section 5.4.2.2, “Default Error Log Destination Configuration”.

The log_sink_internal Component
  • Purpose: Implements traditional error log message output format.

  • URN: This component is built in and need not be loaded with INSTALL COMPONENT before use.

  • Multiple uses permitted: No.

  • Output destination: Writes to the default error log destination.

The log_sink_json Component
  • Purpose: Implements JSON-format error logging. See Section 5.4.2.6, “Error Logging in JSON Format”.

  • URN: file://component_log_sink_json

  • Multiple uses permitted: Yes.

  • Output destination: The JSON log writer determines its output destination based on the default error log destination, which is given by the log_error system variable:

    • If log_error names a file, the JSON writer bases output file naming on that file name, plus a numbered .NN.json suffix, with NN starting at 00. For example, if log_error is file_name, successive instances of log_sink_json named in the log_error_services value write to file_name.00.json, file_name.01.json, and so forth.

    • If log_error is stderr, the JSON writer writes to the console. If log_json_writer is named multiple times in the log_error_services value, they all write to the console, which is likely not useful.

The log_sink_syseventlog Component
  • Purpose: Implements error logging to the system log. This is the Event Log on Windows, and syslog on Unix and Unix-like systems. See Section 5.4.2.7, “Error Logging to the System Log”.

  • URN: file://component_log_sink_syseventlog

  • Multiple uses permitted: No.

  • Output destination: Writes to the system log. Does not use the default error log destination.

The log_sink_test Component
  • Purpose: Intended for internal use in writing test cases. Not intended for production use.

  • URN: file://component_log_sink_test

  • Multiple uses permitted: Yes.

  • Output destination: Writes to the default error log destination.

5.6 MySQL Server Plugins

MySQL supports a plugin API that enables creation of server components. Plugins can be loaded at server startup, or loaded and unloaded at runtime without restarting the server. The components supported by this interface include, but are not limited to, storage engines, INFORMATION_SCHEMA tables, full-text parser plugins, and server extensions.

MySQL distributions include several plugins that implement server extensions:

The following sections describe how to install and uninstall plugins, and how to determine at runtime which plugins are installed and obtain information about them. For information about writing plugins, see Section 29.2, “The MySQL Plugin API”.

5.6.1 Installing and Uninstalling Plugins

Server plugins must be loaded into the server before they can be used. MySQL supports plugin loading at server startup and runtime. It is also possible to control the activation state of loaded plugins at startup, and to unload them at runtime.

While a plugin is loaded, information about it is available from the INFORMATION_SCHEMA.PLUGINS table and the SHOW PLUGINS statement. See Section 5.6.2, “Obtaining Server Plugin Information”.

Installing Plugins

Before a server plugin can be used, it must be installed using one of the following methods. In the descriptions, plugin_name stands for a plugin name such as innodb, csv, or validate_password.

Built-in plugins:

A built-in plugin is known by the server automatically. Normally, the server enables the plugin at startup. Some built-in plugins permit this to be changed with the --plugin_name[=activation_state] option.

Plugins registered in the mysql.plugin system table:

The plugin table in the mysql system database serves as a registry of plugins (other than built-in plugins, which need not be registered). At startup, the server loads each plugin listed in the table. Normally, for a plugin loaded from the mysql.plugin table, the server also enables the plugin. This can be changed with the --plugin_name[=activation_state] option.

If the server is started with the --skip-grant-tables option, it does not consult the mysql.plugin table and does not load the plugins listed there.

Plugins named with command-line options:

A plugin located in a plugin library file can be loaded at server startup with the --plugin-load, --plugin-load-add, or --early-plugin-load option. Normally, for a plugin loaded at startup, the server also enables the plugin. This can be changed with the --plugin_name[=activation_state] option.

The --plugin-load and --plugin-load-add options load plugins after built-in plugins and storage engines have initialized during the server startup sequence. The --early-plugin-load option is used to load plugins that must be available prior to initialization of built-in plugins and storage engines.

The value of each plugin-loading option is a semicolon-separated list of name=plugin_library and plugin_library values. Each name is the name of a plugin to load, and plugin_library is the name of the library file that contains the plugin code. If a plugin library is named without any preceding plugin name, the server loads all plugins in the library. The server looks for plugin library files in the directory named by the plugin_dir system variable.

Plugin-loading options do not register any plugin in the mysql.plugin table. For subsequent restarts, the server loads the plugin again only if --plugin-load, --plugin-load-add, or --early-plugin-load is given again. That is, the option produces a one-time plugin-installation operation that persists for a single server invocation.

--plugin-load, --plugin-load-add, and --early-plugin-load enable plugins to be loaded even when --skip-grant-tables is given (which causes the server to ignore the mysql.plugin table). --plugin-load, --plugin-load-add, and --early-plugin-load also enable plugins to be loaded at startup that cannot be loaded at runtime.

The --plugin-load-add option complements the --plugin-load option:

For example, these options:

--plugin-load=x --plugin-load-add=y

are equivalent to this option:

--plugin-load="x;y"

But these options:

--plugin-load-add=y --plugin-load=x

are equivalent to this option:

--plugin-load=x

Plugins installed with the INSTALL PLUGIN statement:

A plugin located in a plugin library file can be loaded at runtime with the INSTALL PLUGIN statement. The statement also registers the plugin in the mysql.plugin table to cause the server to load it on subsequent restarts. For this reason, INSTALL PLUGIN requires the INSERT privilege for the mysql.plugin table.

The plugin library file base name depends on your platform. Common suffixes are .so for Unix and Unix-like systems, .dll for Windows.

Example: The --plugin-load option installs a plugin at server startup. To install a plugin named myplugin from a plugin library file named somepluglib.so, use these lines in a my.cnf file:

[mysqld]
plugin-load=myplugin=somepluglib.so

In this case, the plugin is not registered in mysql.plugin. Restarting the server without the --plugin-load option causes the plugin not to be loaded at startup.

Alternatively, the INSTALL PLUGIN statement causes the server to load the plugin code from the library file at runtime:

INSTALL PLUGIN myplugin SONAME ‘somepluglib.so‘;

INSTALL PLUGIN also causes permanent” plugin registration: The plugin is listed in the mysql.plugin table to ensure that the server loads it on subsequent restarts.

Many plugins can be loaded either at server startup or at runtime. However, if a plugin is designed such that it must be loaded and initialized during server startup, attempts to load it at runtime using INSTALL PLUGIN produce an error:

mysql> INSTALL PLUGIN myplugin SONAME ‘somepluglib.so‘;
ERROR 1721 (HY000): Plugin ‘myplugin‘ is marked as not dynamically
installable. You have to stop the server to install it.

In this case, you must use --plugin-load, --plugin-load-add, or --early-plugin-load.

If a plugin is named both using a --plugin-load, --plugin-load-add, or --early-plugin-load option and (as a result of an earlier INSTALL PLUGIN statement) in the mysql.plugin table, the server starts but writes these messages to the error log:

[ERROR] Function ‘plugin_name‘ already exists
[Warning] Couldn‘t load plugin named ‘plugin_name‘
with soname ‘plugin_object_file‘.

Controlling Plugin Activation State

If the server knows about a plugin when it starts (for example, because the plugin is named using a --plugin-load option or is registered in the mysql.plugin table), the server loads and enables the plugin by default. It is possible to control activation state for such a plugin using a --plugin_name[=activation_state] startup option, where plugin_name is the name of the plugin to affect, such as innodb, csv, or validate_password. As with other options, dashes and underscores are interchangeable in option names. Also, activation state values are not case-sensitive. For example, --my_plugin=ON and --my-plugin=on are equivalent.

  • --plugin_name=OFF

    Tells the server to disable the plugin. This may not be possible for certain built-in plugins, such as mysql_native_password.

  • --plugin_name[=ON]

    Tells the server to enable the plugin. (Specifying the option as --plugin_name without a value has the same effect.) If the plugin fails to initialize, the server runs with the plugin disabled.

  • --plugin_name=FORCE

    Tells the server to enable the plugin, but if plugin initialization fails, the server does not start. In other words, this option forces the server to run with the plugin enabled or not at all.

  • --plugin_name=FORCE_PLUS_PERMANENT

    Like FORCE, but in addition prevents the plugin from being unloaded at runtime. If a user attempts to do so with UNINSTALL PLUGIN, an error occurs.

Plugin activation states are visible in the LOAD_OPTION column of the INFORMATION_SCHEMA.PLUGINS table.

Suppose that CSV, BLACKHOLE, and ARCHIVE are built-in pluggable storage engines and that you want the server to load them at startup, subject to these conditions: The server is permitted to run if CSV initialization fails, must require that BLACKHOLE initialization succeeds, and should disable ARCHIVE. To accomplish that, use these lines in an option file:

[mysqld]
csv=ON
blackhole=FORCE
archive=OFF

The --enable-plugin_name option format is a synonym for --plugin_name=ON. The --disable-plugin_name and --skip-plugin_name option formats are synonyms for --plugin_name=OFF.

If a plugin is disabled, either explicitly with OFF or implicitly because it was enabled with ON but fails to initialize, aspects of server operation that require the plugin will change. For example, if the plugin implements a storage engine, existing tables for the storage engine become inaccessible, and attempts to create new tables for the storage engine result in tables that use the default storage engine unless the NO_ENGINE_SUBSTITUTION SQL mode is enabled to cause an error to occur instead.

Disabling a plugin may require adjustment to other options. For example, if you start the server using --skip-innodb to disable InnoDB, other innodb_xxx options likely will need to be omitted at startup. In addition, because InnoDB is the default storage engine, it will not start unless you specify another available storage engine with --default_storage_engine. You must also set --default_tmp_storage_engine.

Uninstalling Plugins

At runtime, the UNINSTALL PLUGIN statement disables and uninstalls a plugin known to the server. The statement unloads the plugin and removes it from the mysql.plugin system table, if it is registered there. For this reason, UNINSTALL PLUGIN statement requires the DELETE privilege for the mysql.plugin table. With the plugin no longer registered in the table, the server does not load the plugin automatically for subsequent restarts.

UNINSTALL PLUGIN can unload a plugin regardless of whether it was loaded at runtime with INSTALL PLUGIN or at startup with a plugin-loading option, subject to these conditions:

  • It cannot unload plugins that are built in to the server. These can be identified as those that have a library name of NULL in the output from INFORMATION_SCHEMA.PLUGINS or SHOW PLUGINS.

  • It cannot unload plugins for which the server was started with --plugin_name=FORCE_PLUS_PERMANENT, which prevents plugin unloading at runtime. These can be identified from the LOAD_OPTION column of the INFORMATION_SCHEMA.PLUGINS table.

To uninstall a plugin that currently is loaded at server startup with a plugin-loading option, use this procedure.

  1. Remove any options related to the plugin from the my.cnf file.

  2. Restart the server.

  3. Plugins normally are installed using either a plugin-loading option at startup or with INSTALL PLUGIN at runtime, but not both. However, removing options for a plugin from the my.cnf file may not be sufficient to uninstall it if at some point INSTALL PLUGIN has also been used. If the plugin still appears in the output from INFORMATION_SCHEMA.PLUGINS or SHOW PLUGINS, use UNINSTALL PLUGIN to remove it from the mysql.plugin table. Then restart the server again.

5.6.2 Obtaining Server Plugin Information

There are several ways to determine which plugins are installed in the server:

  • The INFORMATION_SCHEMA.PLUGINS table contains a row for each loaded plugin. Any that have a PLUGIN_LIBRARY value of NULL are built in and cannot be unloaded.

    mysql> SELECT * FROM INFORMATION_SCHEMA.PLUGINS\G
    *************************** 1. row ***************************
               PLUGIN_NAME: binlog
            PLUGIN_VERSION: 1.0
             PLUGIN_STATUS: ACTIVE
               PLUGIN_TYPE: STORAGE ENGINE
       PLUGIN_TYPE_VERSION: 50158.0
            PLUGIN_LIBRARY: NULL
    PLUGIN_LIBRARY_VERSION: NULL
             PLUGIN_AUTHOR: Oracle Corporation
        PLUGIN_DESCRIPTION: This is a pseudo storage engine to represent the binlog in a transaction
            PLUGIN_LICENSE: GPL
               LOAD_OPTION: FORCE
    ...
    *************************** 10. row ***************************
               PLUGIN_NAME: InnoDB
            PLUGIN_VERSION: 1.0
             PLUGIN_STATUS: ACTIVE
               PLUGIN_TYPE: STORAGE ENGINE
       PLUGIN_TYPE_VERSION: 50158.0
            PLUGIN_LIBRARY: ha_innodb_plugin.so
    PLUGIN_LIBRARY_VERSION: 1.0
             PLUGIN_AUTHOR: Oracle Corporation
        PLUGIN_DESCRIPTION: Supports transactions, row-level locking,
                            and foreign keys
            PLUGIN_LICENSE: GPL
               LOAD_OPTION: ON
    ...
    
  • The SHOW PLUGINS statement displays a row for each loaded plugin. Any that have a Library value of NULL are built in and cannot be unloaded.

    mysql> SHOW PLUGINS\G
    *************************** 1. row ***************************
       Name: binlog
     Status: ACTIVE
       Type: STORAGE ENGINE
    Library: NULL
    License: GPL
    ...
    *************************** 10. row ***************************
       Name: InnoDB
     Status: ACTIVE
       Type: STORAGE ENGINE
    Library: ha_innodb_plugin.so
    License: GPL
    ...
    
  • The mysql.plugin table shows which plugins have been registered with INSTALL PLUGIN. The table contains only plugin names and library file names, so it does not provide as much information as the PLUGINS table or the SHOW PLUGINS statement.

5.6.3 MySQL Enterprise Thread Pool

Note

MySQL Enterprise Thread Pool is an extension included in MySQL Enterprise Edition, a commercial product. To learn more about commercial products, https://www.mysql.com/products/.

MySQL Enterprise Edition includes MySQL Enterprise Thread Pool, implemented using a server plugin. The default thread-handling model in MySQL Server executes statements using one thread per client connection. As more clients connect to the server and execute statements, overall performance degrades. The thread pool plugin provides an alternative thread-handling model designed to reduce overhead and improve performance. The plugin implements a thread pool that increases server performance by efficiently managing statement execution threads for large numbers of client connections.

The thread pool addresses several problems of the model that uses one thread per connection:

  • Too many thread stacks make CPU caches almost useless in highly parallel execution workloads. The thread pool promotes thread stack reuse to minimize the CPU cache footprint.

  • With too many threads executing in parallel, context switching overhead is high. This also presents a challenging task to the operating system scheduler. The thread pool controls the number of active threads to keep the parallelism within the MySQL server at a level that it can handle and that is appropriate for the server host on which MySQL is executing.

  • Too many transactions executing in parallel increases resource contention. In InnoDB, this increases the time spent holding central mutexes. The thread pool controls when transactions start to ensure that not too many execute in parallel.

Additional Resources

Section A.15, “MySQL 8.0 FAQ: MySQL Enterprise Thread Pool”

5.6.3.1 Thread Pool Components

The thread pool feature comprises these components:

  • A plugin library file implements a plugin for the thread pool code as well as several associated monitoring tables that provide information about thread pool operation:

    • As of MySQL 8.0.14, the monitoring tables are Performance Schema tables; see Section 26.12.15, “Performance Schema Thread Pool Tables”.

    • Prior to MySQL 8.0.14, the monitoring tables are INFORMATION_SCHEMA tables; see Section 25.46, “INFORMATION_SCHEMA Thread Pool Tables”.

      The INFORMATION_SCHEMA tables now are deprecated and will be removed in a future MySQL version. Applications should transition away from the old tables to the new tables. For example, if an application uses this query:

      SELECT * FROM INFORMATION_SCHEMA.TP_THREAD_STATE;

      The application should use this query instead:

      SELECT * FROM performance_schema.tp_thread_state;
    Note

    If you do not load all the monitoring tables, some or all MySQL Enterprise Monitor thread pool graphs will be empty.

    For a detailed description of how the thread pool works, see Section 5.6.3.3, “Thread Pool Operation”.

  • Several system variables are related to the thread pool. The thread_handling system variable has a value of loaded-dynamically when the server successfully loads the thread pool plugin.

    The other related variables are implemented by the thread pool plugin; they are not available unless it is enabled:

    If any variable implemented by the plugin is set to an illegal value at startup, plugin initialization fails and the plugin does not load.

    For information about setting thread pool parameters, see Section 5.6.3.4, “Thread Pool Tuning”.

  • The Performance Schema has instruments that expose information about the thread pool and may be used to investigate operational performance. To identify them, use this query:

    SELECT * FROM performance_schema.setup_instruments
    WHERE NAME LIKE ‘%thread_pool%‘;

    For more information, see Chapter 26, MySQL Performance Schema.

5.6.3.2 Thread Pool Installation

This section describes how to install MySQL Enterprise Thread Pool. For general information about installing plugins, see Section 5.6.1, “Installing and Uninstalling Plugins”.

To be usable by the server, the plugin library file must be located in the MySQL plugin directory (the directory named by the plugin_dir system variable). If necessary, configure the plugin directory location by setting the value of plugin_dir at server startup.

The plugin library file base name is thread_pool. The file name suffix differs per platform (for example, .so for Unix and Unix-like systems, .dll for Windows).

Thread Pool Installation as of MySQL 8.0.14

In MySQL 8.0.14 and higher, the thread pool monitoring tables are Performance Schema tables that are loaded and unloaded along with the thread pool plugin. The INFORMATION_SCHEMA versions of the tables are deprecated but still available; they are installed per the instructions in Thread Pool Installation Prior to MySQL 8.0.14.

To enable thread pool capability, load the plugin by starting the server with the --plugin-load-add option. To do this, put these lines in the server my.cnf file (adjust the .so suffix for your platform as necessary):

[mysqld]
plugin-load-add=thread_pool.so

To verify plugin installation, examine the INFORMATION_SCHEMA.PLUGINS table or use the SHOW PLUGINS statement (see Section 5.6.2, “Obtaining Server Plugin Information”). For example:

mysql> SELECT PLUGIN_NAME, PLUGIN_STATUS
       FROM INFORMATION_SCHEMA.PLUGINS
       WHERE PLUGIN_NAME LIKE ‘thread%‘;
+-----------------------+---------------+
| PLUGIN_NAME           | PLUGIN_STATUS |
+-----------------------+---------------+
| thread_pool           | ACTIVE        |
+-----------------------+---------------+

To verify that the Performance Schema monitoring tables are available, examine the INFORMATION_SCHEMA.TABLES table or use the SHOW TABLES statement. For example:

mysql> SELECT TABLE_NAME
       FROM INFORMATION_SCHEMA.TABLES
       WHERE TABLE_SCHEMA = ‘performance_schema‘
       AND TABLE_NAME LIKE ‘tp%‘;
+-----------------------+
| TABLE_NAME            |
+-----------------------+
| tp_thread_group_state |
| tp_thread_group_stats |
| tp_thread_state       |
+-----------------------+

If the server loads the thread pool plugin successfully, it sets the thread_handling system variable to loaded-dynamically.

If the plugin fails to initialize, check the server error log for diagnostic messages.

Thread Pool Installation Prior to MySQL 8.0.14

Prior to MySQL 8.0.14, the thread pool monitoring tables are plugins separate from the thread pool plugin and can be installed separately.

To enable thread pool capability, load the plugins to be used by starting the server with the --plugin-load-add option. For example, if you name only the plugin library file, the server loads all plugins that it contains (that is, the thread pool plugin and all the INFORMATION_SCHEMA tables). To do this, put these lines in the server my.cnf file (adjust the .so suffix for your platform as necessary):

[mysqld]
plugin-load-add=thread_pool.so

That is equivalent to loading all thread pool plugins by naming them individually:

[mysqld]
plugin-load-add=thread_pool=thread_pool.so
plugin-load-add=tp_thread_state=thread_pool.so
plugin-load-add=tp_thread_group_state=thread_pool.so
plugin-load-add=tp_thread_group_stats=thread_pool.so

If desired, you can load individual plugins from the library file. To load the thread pool plugin but not the INFORMATION_SCHEMA tables, use an option like this:

[mysqld]
plugin-load-add=thread_pool=thread_pool.so

To load the thread pool plugin and only the TP_THREAD_STATE INFORMATION_SCHEMA table, use options like this:

[mysqld]
plugin-load-add=thread_pool=thread_pool.so
plugin-load-add=tp_thread_state=thread_pool.so

To verify plugin installation, examine the INFORMATION_SCHEMA.PLUGINS table or use the SHOW PLUGINS statement (see Section 5.6.2, “Obtaining Server Plugin Information”). For example:

mysql> SELECT PLUGIN_NAME, PLUGIN_STATUS
       FROM INFORMATION_SCHEMA.PLUGINS
       WHERE PLUGIN_NAME LIKE ‘thread%‘ OR PLUGIN_NAME LIKE ‘tp%‘;
+-----------------------+---------------+
| PLUGIN_NAME           | PLUGIN_STATUS |
+-----------------------+---------------+
| thread_pool           | ACTIVE        |
| TP_THREAD_STATE       | ACTIVE        |
| TP_THREAD_GROUP_STATE | ACTIVE        |
| TP_THREAD_GROUP_STATS | ACTIVE        |
+-----------------------+---------------+

If the server loads the thread pool plugin successfully, it sets the thread_handling system variable to loaded-dynamically.

If a plugin fails to initialize, check the server error log for diagnostic messages.

5.6.3.3 Thread Pool Operation

The thread pool consists of a number of thread groups, each of which manages a set of client connections. As connections are established, the thread pool assigns them to thread groups in round-robin fashion.

The number of thread groups is configurable using the thread_pool_size system variable. The default number of groups is 16. For guidelines on setting this variable, see Section 5.6.3.4, “Thread Pool Tuning”.

The maximum number of threads per group is 4096 (or 4095 on some systems where one thread is used internally).

The thread pool separates connections and threads, so there is no fixed relationship between connections and the threads that execute statements received from those connections. This differs from the default thread-handling model that associates one thread with one connection such that a given thread executes all statements from its connection.

By default, the thread pool tries to ensure a maximum of one thread executing in each group at any time, but sometimes permits more threads to execute temporarily for best performance. The algorithm works in the following manner:

  • Each thread group has a listener thread that listens for incoming statements from the connections assigned to the group. When a statement arrives, the thread group either begins executing it immediately or queues it for later execution:

    • Immediate execution occurs if the statement is the only one received and no statements are queued or currently executing.

    • Queuing occurs if the statement cannot begin executing immediately.

  • If immediate execution occurs, execution is performed by the listener thread. (This means that temporarily no thread in the group is listening.) If the statement finishes quickly, the executing thread returns to listening for statements. Otherwise, the thread pool considers the statement stalled and starts another thread as a listener thread (creating it if necessary). To ensure that no thread group becomes blocked by stalled statements, the thread pool has a background thread that regularly monitors thread group states.

    By using the listening thread to execute a statement that can begin immediately, there is no need to create an additional thread if the statement finishes quickly. This ensures the most efficient execution possible in the case of a low number of concurrent threads.

    When the thread pool plugin starts, it creates one thread per group (the listener thread), plus the background thread. Additional threads are created as necessary to execute statements.

  • The value of the thread_pool_stall_limit system variable determines the meaning of finishes quickly” in the previous item. The default time before threads are considered stalled is 60ms but can be set to a maximum of 6s. This parameter is configurable to enable you to strike a balance appropriate for the server work load. Short wait values permit threads to start more quickly. Short values are also better for avoiding deadlock situations. Long wait values are useful for workloads that include long-running statements, to avoid starting too many new statements while the current ones execute.

  • If thread_pool_max_active_query_threads is 0, the default algorithm applies as just described for determining the maximum number of active threads per group. The default algorithm takes stalled threads into account and may temporarily permit more active threads. If thread_pool_max_active_query_threads is greater than 0, it places a limit on the number of active threads per group.

  • The thread pool focuses on limiting the number of concurrent short-running statements. Before an executing statement reaches the stall time, it prevents other statements from beginning to execute. If the statement executes past the stall time, it is permitted to continue but no longer prevents other statements from starting. In this way, the thread pool tries to ensure that in each thread group there is never more than one short-running statement, although there might be multiple long-running statements. It is undesirable to let long-running statements prevent other statements from executing because there is no limit on the amount of waiting that might be necessary. For example, on a replication master, a thread that is sending binary log events to a slave effectively runs forever.

  • A statement becomes blocked if it encounters a disk I/O operation or a user level lock (row lock or table lock). The block would cause the thread group to become unused, so there are callbacks to the thread pool to ensure that the thread pool can immediately start a new thread in this group to execute another statement. When a blocked thread returns, the thread pool permits it to restart immediately.

  • There are two queues, a high-priority queue and a low-priority queue. The first statement in a transaction goes to the low-priority queue. Any following statements for the transaction go to the high-priority queue if the transaction is ongoing (statements for it have begun executing), or to the low-priority queue otherwise. Queue assignment can be affected by enabling the thread_pool_high_priority_connection system variable, which causes all queued statements for a session to go into the high-priority queue.

    Statements for a nontransactional storage engine, or a transactional engine if autocommit is enabled, are treated as low-priority statements because in this case each statement is a transaction. Thus, given a mix of statements for InnoDB and MyISAM tables, the thread pool prioritizes those for InnoDB over those for MyISAM unless autocommit is enabled. With autocommit enabled, all statements will be low priority.

  • When the thread group selects a queued statement for execution, it first looks in the high-priority queue, then in the low-priority queue. If a statement is found, it is removed from its queue and begins to execute.

  • If a statement stays in the low-priority queue too long, the thread pool moves to the high-priority queue. The value of the thread_pool_prio_kickup_timer system variable controls the time before movement. For each thread group, a maximum of one statement per 10ms or 100 per second will be moved from the low-priority queue to the high-priority queue.

  • The thread pool reuses the most active threads to obtain a much better use of CPU caches. This is a small adjustment that has a great impact on performance.

  • While a thread executes a statement from a user connection, Performance Schema instrumentation accounts thread activity to the user connection. Otherwise, Performance Schema accounts activity to the thread pool.

Here are examples of conditions under which a thread group might have multiple threads started to execute statements:

  • One thread begins executing a statement, but runs long enough to be considered stalled. The thread group permits another thread to begin executing another statement even through the first thread is still executing.

  • One thread begins executing a statement, then becomes blocked and reports this back to the thread pool. The thread group permits another thread to begin executing another statement.

  • One thread begins executing a statement, becomes blocked, but does not report back that it is blocked because the block does not occur in code that has been instrumented with thread pool callbacks. In this case, the thread appears to the thread group to be still running. If the block lasts long enough for the statement to be considered stalled, the group permits another thread to begin executing another statement.

The thread pool is designed to be scalable across an increasing number of connections. It is also designed to avoid deadlocks that can arise from limiting the number of actively executing statements. It is important that threads that do not report back to the thread pool do not prevent other statements from executing and thus cause the thread pool to become deadlocked. Examples of such statements follow:

  • Long-running statements. These would lead to all resources used by only a few statements and they could prevent all others from accessing the server.

  • Binary log dump threads that read the binary log and send it to slaves. This is a kind of long-running statement” that runs for a very long time, and that should not prevent other statements from executing.

  • Statements blocked on a row lock, table lock, sleep, or any other blocking activity that has not been reported back to the thread pool by MySQL Server or a storage engine.

In each case, to prevent deadlock, the statement is moved to the stalled category when it does not complete quickly, so that the thread group can permit another statement to begin executing. With this design, when a thread executes or becomes blocked for an extended time, the thread pool moves the thread to the stalled category and for the rest of the statement‘s execution, it does not prevent other statements from executing.

The maximum number of threads that can occur is the sum of max_connections and thread_pool_size. This can happen in a situation where all connections are in execution mode and an extra thread is created per group to listen for more statements. This is not necessarily a state that happens often, but it is theoretically possible.

5.6.3.4 Thread Pool Tuning

This section provides guidelines on setting thread pool system variables for best performance, measured using a metric such as transactions per second.

thread_pool_size is the most important parameter controlling thread pool performance. It can be set only at server startup. Our experience in testing the thread pool indicates the following:

  • If the primary storage engine is InnoDB, the optimal thread_pool_size setting is likely to be between 16 and 36, with the most common optimal values tending to be from 24 to 36. We have not seen any situation where the setting has been optimal beyond 36. There may be special cases where a value smaller than 16 is optimal.

    For workloads such as DBT2 and Sysbench, the optimum for InnoDB seems to be usually around 36. For very write-intensive workloads, the optimal setting can sometimes be lower.

  • If the primary storage engine is MyISAM, the thread_pool_size setting should be fairly low. Optimal performance is often seen with values from 4 to 8. Higher values tend to have a slightly negative but not dramatic impact on performance.

Another system variable, thread_pool_stall_limit, is important for handling of blocked and long-running statements. If all calls that block the MySQL Server are reported to the thread pool, it would always know when execution threads are blocked. However, this may not always be true. For example, blocks could occur in code that has not been instrumented with thread pool callbacks. For such cases, the thread pool must be able to identify threads that appear to be blocked. This is done by means of a timeout, the length of which can be tuned using the thread_pool_stall_limit system variable. This parameter ensures that the server does not become completely blocked. The value of thread_pool_stall_limit has an upper limit of 6 seconds to prevent the risk of a deadlocked server.

thread_pool_stall_limit also enables the thread pool to handle long-running statements. If a long-running statement was permitted to block a thread group, all other connections assigned to the group would be blocked and unable to start execution until the long-running statement completed. In the worst case, this could take hours or even days.

The value of thread_pool_stall_limit should be chosen such that statements that execute longer than its value are considered stalled. Stalled statements generate a lot of extra overhead since they involve extra context switches and in some cases even extra thread creations. On the other hand, setting the thread_pool_stall_limit parameter too high means that long-running statements will block a number of short-running statements for longer than necessary. Short wait values permit threads to start more quickly. Short values are also better for avoiding deadlock situations. Long wait values are useful for workloads that include long-running statements, to avoid starting too many new statements while the current ones execute.

Suppose a server executes a workload where 99.9% of the statements complete within 100ms even when the server is loaded, and the remaining statements take between 100ms and 2 hours fairly evenly spread. In this case, it would make sense to set thread_pool_stall_limit to 10 (meaning 100ms). The default value of 60ms is suitable for servers that primarily execute very simple statements.

The thread_pool_stall_limit parameter can be changed at runtime to enable you to strike a balance appropriate for the server work load. Assuming that the tp_thread_group_stats table is enabled, you can use the following query to determine the fraction of executed statements that stalled:

SELECT SUM(STALLED_QUERIES_EXECUTED) / SUM(QUERIES_EXECUTED)
FROM performance_schema.tp_thread_group_stats;

This number should be as low as possible. To decrease the likelihood of statements stalling, increase the value of thread_pool_stall_limit.

When a statement arrives, what is the maximum time it can be delayed before it actually starts executing? Suppose that the following conditions apply:

In the worst case, the 10 high-priority statements represent 10 transactions that continue executing for a long time. Thus, in the worst case, no statements will be moved to the high-priority queue because it will always already contain statements awaiting execution. After 10 seconds, the new statement is eligible to be moved to the high-priority queue. However, before it can be moved, all the statements before it must be moved as well. This could take another 2 seconds because a maximum of 100 statements per second are moved to the high-priority queue. Now when the statement reaches the high-priority queue, there could potentially be many long-running statements ahead of it. In the worst case, every one of those will become stalled and it will take 1 second for each statement before the next statement is retrieved from the high-priority queue. Thus, in this scenario, it will take 222 seconds before the new statement starts executing.

This example shows a worst case for an application. How to handle it depends on the application. If the application has high requirements for the response time, it should most likely throttle users at a higher level itself. Otherwise, it can use the thread pool configuration parameters to set some kind of a maximum waiting time.

5.6.4 The Rewriter Query Rewrite Plugin

MySQL supports query rewrite plugins that can examine and possibly modify SQL statements received by the server before the server executes them. See Query Rewrite Plugins.

MySQL distributions include a postparse query rewrite plugin named Rewriter and scripts for installing the plugin and its associated components. These components work together to provide statement-rewriting capability:

  • A server-side plugin named Rewriter examines statements and may rewrite them, based on its in-memory cache of rewrite rules.

  • These statements are subject to rewriting:

    Standalone statements and prepared statements are subject to rewriting. Statements occurring within view definitions or stored programs are not subject to rewriting.

  • The Rewriter plugin uses a database named query_rewrite containing a table named rewrite_rules. The table provides persistent storage for the rules that the plugin uses to decide whether to rewrite statements. Users communicate with the plugin by modifying the set of rules stored in this table. The plugin communicates with users by setting the message column of table rows.

  • The query_rewrite database contains a stored procedure named flush_rewrite_rules() that loads the contents of the rules table into the plugin.

  • A user-defined function named load_rewrite_rules() is used by the flush_rewrite_rules() stored procedure.

  • The Rewriter plugin exposes system variables that enable plugin configuration and status variables that provide runtime operational information.

The following sections describe how to install and use the Rewriter plugin, and provide reference information for its associated components.

5.6.4.1 Installing or Uninstalling the Rewriter Query Rewrite Plugin

Note

If installed, the Rewriter plugin involves some overhead even when disabled. To avoid this overhead, do not install the plugin unless you plan to use it.

To install or uninstall the Rewriter query rewrite plugin, choose the appropriate script located in the share directory of your MySQL installation:

  • install_rewriter.sql: Choose this script to install the Rewriter plugin and its associated components.

  • uninstall_rewriter.sql: Choose this script to uninstall the Rewriter plugin and its associated components.

Run the chosen script as follows:

shell> mysql -u root -p < install_rewriter.sql
Enter password: (enter root password here)

The example here uses the install_rewriter.sql installation script. Substitute uninstall_rewriter.sql if you are uninstalling the plugin.

Running an installation script should install and enable the plugin. To verify that, connect to the server and execute this statement:

mysql> SHOW GLOBAL VARIABLES LIKE ‘rewriter_enabled‘;
+------------------+-------+
| Variable_name    | Value |
+------------------+-------+
| rewriter_enabled | ON    |
+------------------+-------+

For usage instructions, see Section 5.6.4.2, “Using the Rewriter Query Rewrite Plugin”. For reference information, see Section 5.6.4.3, “Rewriter Query Rewrite Plugin Reference”.

5.6.4.2 Using the Rewriter Query Rewrite Plugin

To enable or disable the plugin, enable or disable the rewriter_enabled system variable. By default, the Rewriter plugin is enabled when you install it (see Section 5.6.4.1, “Installing or Uninstalling the Rewriter Query Rewrite Plugin”). To set the initial plugin state explicitly, you can set the variable at server startup. For example, to enable the plugin in an option file, use these lines:

[mysqld]
rewriter_enabled=ON

It is also possible to enable or disable the plugin at runtime:

SET GLOBAL rewriter_enabled = ON;
SET GLOBAL rewriter_enabled = OFF;

Assuming that the Rewriter plugin is enabled, it examines and possibly modifies each rewritable statement received by the server. The plugin determines whether to rewrite statements based on its in-memory cache of rewriting rules, which are loaded from the rewrite_rules table in the query_rewrite database.

These statements are subject to rewriting:

Standalone statements and prepared statements are subject to rewriting. Statements occurring within view definitions or stored programs are not subject to rewriting.

Adding Rewrite Rules

To add rules for the Rewriter plugin, add rows to the rewrite_rules table, then invoke the flush_rewrite_rules() stored procedure to load the rules from the table into the plugin. The following example creates a simple rule to match statements that select a single literal value:

INSERT INTO query_rewrite.rewrite_rules (pattern, replacement)
VALUES(‘SELECT ?‘, ‘SELECT ? + 1‘);

The resulting table contents look like this:

mysql> SELECT * FROM query_rewrite.rewrite_rules\G
*************************** 1. row ***************************
                id: 1
           pattern: SELECT ?
  pattern_database: NULL
       replacement: SELECT ? + 1
           enabled: YES
           message: NULL
    pattern_digest: NULL
normalized_pattern: NULL

The rule specifies a pattern template indicating which SELECT statements to match, and a replacement template indicating how to rewrite matching statements. However, adding the rule to the rewrite_rules table is not sufficient to cause the Rewriter plugin to use the rule. You must invoke flush_rewrite_rules() to load the table contents into the plugin in-memory cache:

mysql> CALL query_rewrite.flush_rewrite_rules();
Tip

If your rewrite rules seem not to be working properly, make sure that you have reloaded the rules table by calling flush_rewrite_rules().

When the plugin reads each rule from the rules table, it computes a normalized (statement digest) form from the pattern and a digest hash value, and uses them to update the normalized_pattern and pattern_digest columns:

mysql> SELECT * FROM query_rewrite.rewrite_rules\G
*************************** 1. row ***************************
                id: 1
           pattern: SELECT ?
  pattern_database: NULL
       replacement: SELECT ? + 1
           enabled: YES
           message: NULL
    pattern_digest: d1b44b0c19af710b5a679907e284acd2ddc285201794bc69a2389d77baedddae
normalized_pattern: select ?

For information about statement digesting, normalized statements, and digest hash values, see Section 26.10, “Performance Schema Statement Digests and Sampling”.

If a rule cannot be loaded due to some error, calling flush_rewrite_rules() produces an error:

mysql> CALL query_rewrite.flush_rewrite_rules();
ERROR 1644 (45000): Loading of some rule(s) failed.

When this occurs, the plugin writes an error message to the message column of the rule row to communicate the problem. Check the rewrite_rules table for rows with non-NULL message column values to see what problems exist.

Patterns use the same syntax as prepared statements (see Section 13.5.1, “PREPARE Statement”). Within a pattern template, ? characters act as parameter markers that match data values. Parameter markers can be used only where data values should appear, not for SQL keywords, identifiers, and so forth. The ? characters should not be enclosed within quotation marks.

Like the pattern, the replacement can contain ? characters. For a statement that matches a pattern template, the plugin rewrites it, replacing ? parameter markers in the replacement using data values matched by the corresponding markers in the pattern. The result is a complete statement string. The plugin asks the server to parse it, and returns the result to the server as the representation of the rewritten statement.

After adding and loading the rule, check whether rewriting occurs according to whether statements match the rule pattern:

mysql> SELECT PI();
+----------+
| PI()     |
+----------+
| 3.141593 |
+----------+
1 row in set (0.01 sec)

mysql> SELECT 10;
+--------+
| 10 + 1 |
+--------+
|     11 |
+--------+
1 row in set, 1 warning (0.00 sec)

No rewriting occurs for the first SELECT statement, but does for the second. The second statement illustrates that when the Rewriter plugin rewrites a statement, it produces a warning message. To view the message, use SHOW WARNINGS:

mysql> SHOW WARNINGS\G
*************************** 1. row ***************************
  Level: Note
   Code: 1105
Message: Query ‘SELECT 10‘ rewritten to ‘SELECT 10 + 1‘ by a query rewrite plugin

A statement need not be rewritten to a statement of the same type. The following example loads a rule that rewrites DELETE statements to UPDATE statements:

INSERT INTO query_rewrite.rewrite_rules (pattern, replacement)
VALUES(‘DELETE FROM db1.t1 WHERE col = ?‘,
       ‘UPDATE db1.t1 SET col = NULL WHERE col = ?‘);
CALL query_rewrite.flush_rewrite_rules();

To enable or disable an existing rule, modify its enabled column and reload the table into the plugin. To disable rule 1:

UPDATE query_rewrite.rewrite_rules SET enabled = ‘NO‘ WHERE id = 1;
CALL query_rewrite.flush_rewrite_rules();

This enables you to deactivate a rule without removing it from the table.

To re-enable rule 1:

UPDATE query_rewrite.rewrite_rules SET enabled = ‘YES‘ WHERE id = 1;
CALL query_rewrite.flush_rewrite_rules();

The rewrite_rules table contains a pattern_database column that Rewriter uses for matching table names that are not qualified with a database name:

  • Qualified table names in statements match qualified names in the pattern if corresponding database and table names are identical.

  • Unqualified table names in statements match unqualified names in the pattern only if the default database is the same as pattern_database and the table names are identical.

Suppose that a table named appdb.users has a column named id and that applications are expected to select rows from the table using a query of one of these forms, where the second can be used only if appdb is the default database:

SELECT * FROM users WHERE appdb.id = id_value;
SELECT * FROM users WHERE id = id_value;

Suppose also that the id column is renamed to user_id (perhaps the table must be modified to add another type of ID and it is necessary to indicate more specifically what type of ID the id column represents).

The change means that applications must refer to user_id rather than id in the WHERE clause. But if there are old applications that cannot be written to change the SELECT queries they generate, they will no longer work properly. The Rewriter plugin can solve this problem. To match and rewrite statements whether or not they qualify the table name, add the following two rules and reload the rules table:

INSERT INTO query_rewrite.rewrite_rules
    (pattern, replacement) VALUES(
    ‘SELECT * FROM appdb.users WHERE id = ?‘,
    ‘SELECT * FROM appdb.users WHERE user_id = ?‘
    );
INSERT INTO query_rewrite.rewrite_rules
    (pattern, replacement, pattern_database) VALUES(
    ‘SELECT * FROM users WHERE id = ?‘,
    ‘SELECT * FROM users WHERE user_id = ?‘,
    ‘appdb‘
    );
CALL query_rewrite.flush_rewrite_rules();

Rewriter uses the first rule to match statements that use the qualified table name. It uses the second to match statements that used the unqualified name, but only if the default database is appdb (the value in pattern_database).

How Statement Matching Works

The Rewriter plugin uses statement digests and digest hash values to match incoming statements against rewrite rules in stages. The max_digest_length system variable determines the size of the buffer used for computing statement digests. Larger values enable computation of digests that distinguish longer statements. Smaller values use less memory but increase the likelihood of longer statements colliding with the same digest value.

The plugin matches each statement to the rewrite rules as follows:

  1. Compute the statement digest hash value and compare it to the rule digest hash values. This is subject to false positives, but serves as a quick rejection test.

  2. If the statement digest hash value matches any pattern digest hash values, match the normalized (statement digest) form of the statement to the normalized form of the matching rule patterns.

  3. If the normalized statement matches a rule, compare the literal values in the statement and the pattern. A ? character in the pattern matches any literal value in the statement. If the statement prepares a statement, ? in the pattern also matches ? in the statement. Otherwise, corresponding literals must be the same.

If multiple rules match a statement, it is nondeterministic which one the plugin uses to rewrite the statement.

If a pattern contains more markers than the replacement, the plugin discards excess data values. If a pattern contains fewer markers than the replacement, it is an error. The plugin notices this when the rules table is loaded, writes an error message to the message column of the rule row to communicate the problem, and sets the Rewriter_reload_error status variable to ON.

Rewriting Prepared Statements

Prepared statements are rewritten at parse time (that is, when they are prepared), not when they are executed later.

Prepared statements differ from nonprepared statements in that they may contain ? characters as parameter markers. To match a ? in a prepared statement, a Rewriter pattern must contain ? in the same location. Suppose that a rewrite rule has this pattern:

SELECT ?, 3

The following table shows several prepared SELECT statements and whether the rule pattern matches them.

Prepared StatementWhether Pattern Matches Statement
PREPARE s AS ‘SELECT 3, 3‘ Yes
PREPARE s AS ‘SELECT ?, 3‘ Yes
PREPARE s AS ‘SELECT 3, ?‘ No
PREPARE s AS ‘SELECT ?, ?‘ No
Rewriter Plugin Operational Information

The Rewriter plugin makes information available about its operation by means of several status variables:

mysql> SHOW GLOBAL STATUS LIKE ‘Rewriter%‘;
+-----------------------------------+-------+
| Variable_name                     | Value |
+-----------------------------------+-------+
| Rewriter_number_loaded_rules      | 1     |
| Rewriter_number_reloads           | 5     |
| Rewriter_number_rewritten_queries | 1     |
| Rewriter_reload_error             | ON    |
+-----------------------------------+-------+

For descriptions of these variables, see Section 5.6.4.3.4, “Rewriter Query Rewrite Plugin Status Variables”.

When you load the rules table by calling the flush_rewrite_rules() stored procedure, if an error occurs for some rule, the CALL statement produces an error, and the plugin sets the Rewriter_reload_error status variable to ON:

mysql> CALL query_rewrite.flush_rewrite_rules();
ERROR 1644 (45000): Loading of some rule(s) failed.

mysql> SHOW GLOBAL STATUS LIKE ‘Rewriter_reload_error‘;
+-----------------------+-------+
| Variable_name         | Value |
+-----------------------+-------+
| Rewriter_reload_error | ON    |
+-----------------------+-------+

In this case, check the rewrite_rules table for rows with non-NULL message column values to see what problems exist.

Rewriter Plugin Use of Character Sets

When the rewrite_rules table is loaded into the Rewriter plugin, the plugin interprets statements using the current global value of the character_set_client system variable. If the global character_set_client value is changed subsequently, the rules table must be reloaded.

A client must have a session character_set_client value identical to what the global value was when the rules table was loaded or rule matching will not work for that client.

5.6.4.3 Rewriter Query Rewrite Plugin Reference

The following discussion serves as a reference to these components associated with the Rewriter query rewrite plugin:

  • The Rewriter rules table in the query_rewrite database

  • Rewriter procedures and functions

  • Rewriter system and status variables

5.6.4.3.1 Rewriter Query Rewrite Plugin Rules Table

The rewrite_rules table in the query_rewrite database provides persistent storage for the rules that the Rewriter plugin uses to decide whether to rewrite statements.

Users communicate with the plugin by modifying the set of rules stored in this table. The plugin communicates information to users by setting the table‘s message column.

Note

The rules table is loaded into the plugin by the flush_rewrite_rules stored procedure. Unless that procedure has been called following the most recent table modification, the table contents do not necessarily correspond to the set of rules the plugin is using.

The rewrite_rules table has these columns:

  • id

    The rule ID. This column is the table primary key. You can use the ID to uniquely identify any rule.

  • pattern

    The template that indicates the pattern for statements that the rule matches. Use ? to represent parameter markers that match data values.

  • pattern_database

    The database used to match unqualified table names in statements. Qualified table names in statements match qualified names in the pattern if corresponding database and table names are identical. Unqualified table names in statements match unqualified names in the pattern only if the default database is the same as pattern_database and the table names are identical.

  • replacement

    The template that indicates how to rewrite statements matching the pattern column value. Use ? to represent parameter markers that match data values. In rewritten statements, the plugin replaces ? parameter markers in replacement using data values matched by the corresponding markers in pattern.

  • enabled

    Whether the rule is enabled. Load operations (performed by invoking the flush_rewrite_rules() stored procedure) load the rule from the table into the Rewriter in-memory cache only if this column is YES.

    This column makes it possible to deactivate a rule without removing it: Set the column to a value other than YES and reload the table into the plugin.

  • message

    The plugin uses this column for communicating with users. If no error occurs when the rules table is loaded into memory, the plugin sets the message column to NULL. A non-NULL value indicates an error and the column contents are the error message. Errors can occur under these circumstances:

    • Either the pattern or the replacement is an incorrect SQL statement that produces syntax errors.

    • The replacement contains more ? parameter markers than the pattern.

    If a load error occurs, the plugin also sets the Rewriter_reload_error status variable to ON.

  • pattern_digest

    This column is used for debugging and diagnostics. If the column exists when the rules table is loaded into memory, the plugin updates it with the pattern digest. This column may be useful if you are trying to determine why some statement fails to be rewritten.

  • normalized_pattern

    This column is used for debugging and diagnostics. If the column exists when the rules table is loaded into memory, the plugin updates it with the normalized form of the pattern. This column may be useful if you are trying to determine why some statement fails to be rewritten.

5.6.4.3.2 Rewriter Query Rewrite Plugin Procedures and Functions

Rewriter plugin operation uses a stored procedure that loads the rules table into its in-memory cache, and a helper user-defined function (UDF). Under normal operation, users invoke only the stored procedure. The UDF is intended to be invoked by the stored procedure, not directly by users.

  • flush_rewrite_rules()

    This stored procedure uses the load_rewrite_rules() UDF to load the contents of the rewrite_rules table into the Rewriter in-memory cache.

    Calling flush_rewrite_rules() implies COMMIT.

    Invoke this procedure after you modify the rules table to cause the plugin to update its cache from the new table contents. If any errors occur, the plugin sets the message column for the appropriate rule rows in the table and sets the Rewriter_reload_error status variable to ON.

  • load_rewrite_rules()

    This UDF is a helper routine used by the flush_rewrite_rules() stored procedure.

5.6.4.3.3 Rewriter Query Rewrite Plugin System Variables

The Rewriter query rewrite plugin supports the following system variables. These variables are available only if the plugin is installed (see Section 5.6.4.1, “Installing or Uninstalling the Rewriter Query Rewrite Plugin”).

5.6.4.3.4 Rewriter Query Rewrite Plugin Status Variables

The Rewriter query rewrite plugin supports the following status variables. These variables are available only if the plugin is installed (see Section 5.6.4.1, “Installing or Uninstalling the Rewriter Query Rewrite Plugin”).

  • Rewriter_number_loaded_rules

    The number of rewrite plugin rewrite rules successfully loaded from the rewrite_rules table into memory for use by the Rewriter plugin.

  • Rewriter_number_reloads

    The number of times the rewrite_rules table has been loaded into the in-memory cache used by the Rewriter plugin.

  • Rewriter_number_rewritten_queries

    The number of queries rewritten by the Rewriter query rewrite plugin since it was loaded.

  • Rewriter_reload_error

    Whether an error occurred the most recent time that the rewrite_rules table was loaded into the in-memory cache used by the Rewriter plugin. If the value is OFF, no error occurred. If the value is ON, an error occurred; check the message column of the rewriter_rules table for error messages.

5.6.5 The ddl_rewriter Plugin

MySQL 8.0.16 and higher includes a ddl_rewriter plugin that modifies CREATE TABLE statements received by the server before it parses and executes them. The plugin removes ENCRYPTION, DATA DIRECTORY, and INDEX DIRECTORY clauses, which may be helpful when restoring tables from SQL dump files created from databases that are encrypted or that have their tables stored outside the data directory. For example, the plugin may enable restoring such dump files into an unencrypted instance or in an environment where the paths outside the data directory are not accessible.

Before using the ddl_rewriter plugin, install it according to the instructions provided in Section 5.6.5.1, “Installing or Uninstalling ddl_rewriter”.

ddl_rewriter examines SQL statements received by the server prior to parsing, rewriting them according to these conditions:

  • ddl_rewriter considers only CREATE TABLE statements, and only if they are standalone statements that occur at the beginning of an input line or at the beginning of prepared statement text. ddl_rewriter does not consider CREATE TABLE statements within stored program definitions. Statements can extend over multiple lines.

  • Within statements considered for rewrite, instances of the following clauses are rewritten and each instance replaced by a single space:

    • ENCRYPTION

    • DATA DIRECTORY (at the table and partition levels)

    • INDEX DIRECTORY (at the table and partition levels)

  • Rewriting does not depend on lettercase.

If ddl_rewriter rewrites a statement, it generates a warning:

mysql> CREATE TABLE t (i INT) DATA DIRECTORY ‘/var/mysql/data‘;
Query OK, 0 rows affected, 1 warning (0.03 sec)

mysql> SHOW WARNINGS\G
*************************** 1. row ***************************
  Level: Note
   Code: 1105
Message: Query ‘CREATE TABLE t (i INT) DATA DIRECTORY ‘/var/mysql/data‘‘
         rewritten to ‘CREATE TABLE t (i INT) ‘ by a query rewrite plugin
1 row in set (0.00 sec)

If the general query log or binary log is enabled, the server writes to it statements as they appear after any rewriting by ddl_rewriter.

When installed, ddl_rewriter exposes the Performance Schema memory/rewriter/ddl_rewriter instrument for tracking plugin memory use. See Section 26.12.17.10, “Memory Summary Tables”

5.6.5.1 Installing or Uninstalling ddl_rewriter

This section describes how to install or uninstall the ddl_rewriter plugin. For general information about installing plugins, see Section 5.6.1, “Installing and Uninstalling Plugins”.

Note

If installed, the ddl_rewriter plugin involves some minimal overhead even when disabled. To avoid this overhead, install ddl_rewriter only for the period during which you intend to use it.

The primary use case is modification of statements restored from dump files, so the typical usage pattern is: 1) Install the plugin; 2) restore the dump file or files; 3) uninstall the plugin.

To be usable by the server, the plugin library file must be located in the MySQL plugin directory (the directory named by the plugin_dir system variable). If necessary, configure the plugin directory location by setting the value of plugin_dir at server startup.

The plugin library file base name is ddl_rewriter. The file name suffix differs per platform (for example, .so for Unix and Unix-like systems, .dll for Windows).

To install the ddl_rewriter plugin, use the INSTALL PLUGIN statement (adjust the .so suffix for your platform as necessary):

INSTALL PLUGIN ddl_rewriter SONAME ‘ddl_rewriter.so‘;

To verify plugin installation, examine the INFORMATION_SCHEMA.PLUGINS table or use the SHOW PLUGINS statement (see Section 5.6.2, “Obtaining Server Plugin Information”). For example:

mysql> SELECT PLUGIN_NAME, PLUGIN_STATUS, PLUGIN_TYPE
       FROM INFORMATION_SCHEMA.PLUGINS
       WHERE PLUGIN_NAME LIKE ‘ddl%‘;
+--------------+---------------+-------------+
| PLUGIN_NAME  | PLUGIN_STATUS | PLUGIN_TYPE |
+--------------+---------------+-------------+
| ddl_rewriter | ACTIVE        | AUDIT       |
+--------------+---------------+-------------+

As the preceding result shows, ddl_rewriter is implemented as an audit plugin.

If the plugin fails to initialize, check the server error log for diagnostic messages.

Once installed as just described, ddl_rewriter remains installed until uninstalled. To remove it, use UNINSTALL PLUGIN:

UNINSTALL PLUGIN ddl_rewriter;

If ddl_rewriter is installed, you can use the --ddl-rewriter option for subsequent server startups to control ddl_rewriter plugin activation. For example, to prevent the plugin from being enabled at runtime, use this option:

[mysqld]
ddl-rewriter=OFF

5.6.5.2 ddl_rewriter Plugin Options

This section describes the command options that control operation of the ddl_rewriter plugin. If values specified at startup time are incorrect, the ddl_rewriter plugin may fail to initialize properly and the server does not load it.

To control activation of the ddl_rewriter plugin, use this option:

5.6.6 Version Tokens

MySQL includes Version Tokens, a feature that enables creation of and synchronization around server tokens that applications can use to prevent accessing incorrect or out-of-date data.

The Version Tokens interface has these characteristics:

  • Version tokens are pairs consisting of a name that serves as a key or identifier, plus a value.

  • Version tokens can be locked. An application can use token locks to indicate to other cooperating applications that tokens are in use and should not be modified.

  • Version token lists are established per server (for example, to specify the server assignment or operational state). In addition, an application that communicates with a server can register its own list of tokens that indicate the state it requires the server to be in. An SQL statement sent by the application to a server not in the required state produces an error. This is a signal to the application that it should seek a different server in the required state to receive the SQL statement.

The following sections describe the components of Version Tokens, discuss how to install and use it, and provide reference information for its components.

5.6.6.1 Version Tokens Components

Version Tokens is based on a plugin library that implements these components:

  • A server-side plugin named version_tokens holds the list of version tokens associated with the server and subscribes to notifications for statement execution events. The version_tokens plugin uses the audit plugin API to monitor incoming statements from clients and matches each client‘s session-specific version token list against the server version token list. If there is a match, the plugin lets the statement through and the server continues to process it. Otherwise, the plugin returns an error to the client and the statement fails.

  • A set of user-defined functions (UDFs) provides an SQL-level API for manipulating and inspecting the list of server version tokens maintained by the plugin. The VERSION_TOKEN_ADMIN or SUPER privilege is required to call any of the Version Token UDFs.

  • When the version_tokens plugin loads, it defines the VERSION_TOKEN_ADMIN dynamic privilege. This privilege can be granted to users of the UDFs.

  • A system variable enables clients to specify the list of version tokens that register the required server state. If the server has a different state when a client sends a statement, the client receives an error.

5.6.6.2 Installing or Uninstalling Version Tokens

Note

If installed, Version Tokens involves some overhead. To avoid this overhead, do not install it unless you plan to use it.

This section describes how to install or uninstall Version Tokens, which is implemented in a plugin library file containing a plugin and user-defined functions (UDFs). For general information about installing or uninstalling plugins and UDFs, see Section 5.6.1, “Installing and Uninstalling Plugins”, and Section 5.7.1, “Installing and Uninstalling User-Defined Functions”.

To be usable by the server, the plugin library file must be located in the MySQL plugin directory (the directory named by the plugin_dir system variable). If necessary, configure the plugin directory location by setting the value of plugin_dir at server startup.

The plugin library file base name is version_tokens. The file name suffix differs per platform (for example, .so for Unix and Unix-like systems, .dll for Windows).

To install the Version Tokens plugin and UDFs, use the INSTALL PLUGIN and CREATE FUNCTION statements (adjust the .so suffix for your platform as necessary):

INSTALL PLUGIN version_tokens SONAME ‘version_token.so‘;
CREATE FUNCTION version_tokens_set RETURNS STRING
  SONAME ‘version_token.so‘;
CREATE FUNCTION version_tokens_show RETURNS STRING
  SONAME ‘version_token.so‘;
CREATE FUNCTION version_tokens_edit RETURNS STRING
  SONAME ‘version_token.so‘;
CREATE FUNCTION version_tokens_delete RETURNS STRING
  SONAME ‘version_token.so‘;
CREATE FUNCTION version_tokens_lock_shared RETURNS INT
  SONAME ‘version_token.so‘;
CREATE FUNCTION version_tokens_lock_exclusive RETURNS INT
  SONAME ‘version_token.so‘;
CREATE FUNCTION version_tokens_unlock RETURNS INT
  SONAME ‘version_token.so‘;

You must install the UDFs to manage the server‘s version token list, but you must also install the plugin because the UDFs will not work correctly without it.

If the plugin and UDFs are used on a master replication server, install them on all slave servers as well to avoid replication problems.

Once installed as just described, the plugin and UDFs remain installed until uninstalled. To remove them, use the UNINSTALL PLUGIN and DROP FUNCTION statements:

UNINSTALL PLUGIN version_tokens;
DROP FUNCTION version_tokens_set;
DROP FUNCTION version_tokens_show;
DROP FUNCTION version_tokens_edit;
DROP FUNCTION version_tokens_delete;
DROP FUNCTION version_tokens_lock_shared;
DROP FUNCTION version_tokens_lock_exclusive;
DROP FUNCTION version_tokens_unlock;

5.6.6.3 Using Version Tokens

Before using Version Tokens, install it according to the instructions provided at Section 5.6.6.2, “Installing or Uninstalling Version Tokens”.

A scenario in which Version Tokens can be useful is a system that accesses a collection of MySQL servers but needs to manage them for load balancing purposes by monitoring them and adjusting server assignments according to load changes. Such a system comprises these components:

  • The collection of MySQL servers to be managed.

  • An administrative or management application that communicates with the servers and organizes them into high-availability groups. Groups serve different purposes, and servers within each group may have different assignments. Assignment of a server within a certain group can change at any time.

  • Client applications that access the servers to retrieve and update data, choosing servers according to the purposes assigned them. For example, a client should not send an update to a read-only server.

Version Tokens permit server access to be managed according to assignment without requiring clients to repeatedly query the servers about their assignments:

  • The management application performs server assignments and establishes version tokens on each server to reflect its assignment. The application caches this information to provide a central access point to it.

    If at some point the management application needs to change a server assignment (for example, to change it from permitting writes to read only), it changes the server‘s version token list and updates its cache.

  • To improve performance, client applications obtain cache information from the management application, enabling them to avoid having to retrieve information about server assignments for each statement. Based on the type of statements it will issue (for example, reads versus writes), a client selects an appropriate server and connects to it.

  • In addition, the client sends to the server its own client-specific version tokens to register the assignment it requires of the server. For each statement sent by the client to the server, the server compares its own token list with the client token list. If the server token list contains all tokens present in the client token list with the same values, there is a match and the server executes the statement.

    On the other hand, perhaps the management application has changed the server assignment and its version token list. In this case, the new server assignment may now be incompatible with the client requirements. A token mismatch between the server and client token lists occurs and the server returns an error in reply to the statement. This is an indication to the client to refresh its version token information from the management application cache, and to select a new server to communicate with.

The client-side logic for detecting version token errors and selecting a new server can be implemented different ways:

  • The client can handle all version token registration, mismatch detection, and connection switching itself.

  • The logic for those actions can be implemented in a connector that manages connections between clients and MySQL servers. Such a connector might handle mismatch error detection and statement resending itself, or it might pass the error to the application and leave it to the application to resend the statement.

The following example illustrates the preceding discussion in more concrete form.

When Version Tokens initializes on a given server, the server‘s version token list is empty. Token list maintenance is performed by calling user-defined functions (UDFs). The VERSION_TOKEN_ADMIN or SUPER privilege is required to call any of the Version Token UDFs, so token list modification is expected to be done by a management or administrative application that has that privilege.

Suppose that a management application communicates with a set of servers that are queried by clients to access employee and product databases (named emp and prod, respectively). All servers are permitted to process data retrieval statements, but only some of them are permitted to make database updates. To handle this on a database-specific basis, the management application establishes a list of version tokens on each server. In the token list for a given server, token names represent database names and token values are read or write depending on whether the database must be used in read-only fashion or whether it can take reads and writes.

Client applications register a list of version tokens they require the server to match by setting a system variable. Variable setting occurs on a client-specific basis, so different clients can register different requirements. By default, the client token list is empty, which matches any server token list. When a client sets its token list to a nonempty value, matching may succeed or fail, depending on the server version token list.

To define the version token list for a server, the management application calls the version_tokens_set() UDF. (There are also UDFs for modifying and displaying the token list, described later.) For example, the application might send these statements to a group of three servers:

Server 1:

mysql> SELECT version_tokens_set(‘emp=read;prod=read‘);
+------------------------------------------+
| version_tokens_set(‘emp=read;prod=read‘) |
+------------------------------------------+
| 2 version tokens set.                    |
+------------------------------------------+

Server 2:

mysql> SELECT version_tokens_set(‘emp=write;prod=read‘);
+-------------------------------------------+
| version_tokens_set(‘emp=write;prod=read‘) |
+-------------------------------------------+
| 2 version tokens set.                     |
+-------------------------------------------+

Server 3:

mysql> SELECT version_tokens_set(‘emp=read;prod=write‘);
+-------------------------------------------+
| version_tokens_set(‘emp=read;prod=write‘) |
+-------------------------------------------+
| 2 version tokens set.                     |
+-------------------------------------------+

The token list in each case is specified as a semicolon-separated list of name=value pairs. The resulting token list values result in these server assingments:

  • Any server accepts reads for either database.

  • Only server 2 accepts updates for the emp database.

  • Only server 3 accepts updates for the prod database.

In addition to assigning each server a version token list, the management application also maintains a cache that reflects the server assignments.

Before communicating with the servers, a client application contacts the management application and retrieves information about server assignments. Then the client selects a server based on those assignments. Suppose that a client wants to perform both reads and writes on the emp database. Based on the preceding assignments, only server 2 qualifies. The client connects to server 2 and registers its server requirements there by setting its version_tokens_session system variable:

mysql> SET @@SESSION.version_tokens_session = ‘emp=write‘;

For subsequent statements sent by the client to server 2, the server compares its own version token list to the client list to check whether they match. If so, statements execute normally:

mysql> UPDATE emp.employee SET salary = salary * 1.1 WHERE id = 4981;
Query OK, 1 row affected (0.07 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> SELECT last_name, first_name FROM emp.employee WHERE id = 4981;
+-----------+------------+
| last_name | first_name |
+-----------+------------+
| Smith     | Abe        |
+-----------+------------+
1 row in set (0.01 sec)

Discrepancies between the server and client version token lists can occur two ways:

As long as the assignment of server 2 does not change, the client continues to use it for reads and writes. But suppose that the management application wants to change server assignments so that writes for the emp database must be sent to server 1 instead of server 2. To do this, it uses version_tokens_edit() to modify the emp token value on the two servers (and updates its cache of server assignments):

Server 1:

mysql> SELECT version_tokens_edit(‘emp=write‘);
+----------------------------------+
| version_tokens_edit(‘emp=write‘) |
+----------------------------------+
| 1 version tokens updated.        |
+----------------------------------+

Server 2:

mysql> SELECT version_tokens_edit(‘emp=read‘);
+---------------------------------+
| version_tokens_edit(‘emp=read‘) |
+---------------------------------+
| 1 version tokens updated.       |
+---------------------------------+

version_tokens_edit() modifies the named tokens in the server token list and leaves other tokens unchanged.

The next time the client sends a statement to server 2, its own token list no longer matches the server token list and an error occurs:

mysql> UPDATE emp.employee SET salary = salary * 1.1 WHERE id = 4982;
ERROR 3136 (42000): Version token mismatch for emp. Correct value read

In this case, the client should contact the management application to obtain updated information about server assignments, select a new server, and send the failed statement to the new server.

Note

Each client must cooperate with Version Tokens by sending only statements in accordance with the token list that it registers with a given server. For example, if a client registers a token list of ‘emp=read‘, there is nothing in Version Tokens to prevent the client from sending updates for the emp database. The client itself must refrain from doing so.

For each statement received from a client, the server implicitly uses locking, as follows:

  • Take a shared lock for each token named in the client token list (that is, in the version_tokens_session value)

  • Perform the comparison between the server and client token lists

  • Execute the statement or produce an error depending on the comparison result

  • Release the locks

The server uses shared locks so that comparisons for multiple sessions can occur without blocking, while preventing changes to the tokens for any session that attempts to acquire an exclusive lock before it manipulates tokens of the same names in the server token list.

The preceding example uses only a few of the user-defined included in the Version Tokens plugin library, but there are others. One set of UDFs permits the server‘s list of version tokens to be manipulated and inspected. Another set of UDFs permits version tokens to be locked and unlocked.

These UDFs permit the server‘s list of version tokens to be created, changed, removed, and inspected:

  • version_tokens_set() completely replaces the current list and assigns a new list. The argument is a semicolon-separated list of name=value pairs.

  • version_tokens_edit() enables partial modifications to the current list. It can add new tokens or change the values of existing tokens. The argument is a semicolon-separated list of name=value pairs.

  • version_tokens_delete() deletes tokens from the current list. The argument is a semicolon-separated list of token names.

  • version_tokens_show() displays the current token list. It takes no argument.

Each of those functions, if successful, returns a binary string indicating what action occurred. The following example establishes the server token list, modifies it by adding a new token, deletes some tokens, and displays the resulting token list:

mysql> SELECT version_tokens_set(‘tok1=a;tok2=b‘);
+-------------------------------------+
| version_tokens_set(‘tok1=a;tok2=b‘) |
+-------------------------------------+
| 2 version tokens set.               |
+-------------------------------------+
mysql> SELECT version_tokens_edit(‘tok3=c‘);
+-------------------------------+
| version_tokens_edit(‘tok3=c‘) |
+-------------------------------+
| 1 version tokens updated.     |
+-------------------------------+
mysql> SELECT version_tokens_delete(‘tok2;tok1‘);
+------------------------------------+
| version_tokens_delete(‘tok2;tok1‘) |
+------------------------------------+
| 2 version tokens deleted.          |
+------------------------------------+
mysql> SELECT version_tokens_show();
+-----------------------+
| version_tokens_show() |
+-----------------------+
| tok3=c;               |
+-----------------------+

Warnings occur if a token list is malformed:

mysql> SELECT version_tokens_set(‘tok1=a; =c‘);
+----------------------------------+
| version_tokens_set(‘tok1=a; =c‘) |
+----------------------------------+
| 1 version tokens set.            |
+----------------------------------+
1 row in set, 1 warning (0.00 sec)

mysql> SHOW WARNINGS\G
*************************** 1. row ***************************
  Level: Warning
   Code: 42000
Message: Invalid version token pair encountered. The list provided
         is only partially updated.
1 row in set (0.00 sec)

As mentioned previously, version tokens are defined using a semicolon-separated list of name=value pairs. Consider this invocation of version_tokens_set():

mysql> SELECT version_tokens_set(‘tok1=b;;; tok2= a = b ; tok1 = 1\‘2 3"4‘)
+---------------------------------------------------------------+
| version_tokens_set(‘tok1=b;;; tok2= a = b ; tok1 = 1\‘2 3"4‘) |
+---------------------------------------------------------------+
| 3 version tokens set.                                         |
+---------------------------------------------------------------+

Version Tokens interprets the argument as follows:

  • Whitespace around names and values is ignored. Whitespace within names and values is permitted. (For version_tokens_delete(), which takes a list of names without values, whitespace around names is ignored.)

  • There is no quoting mechanism.

  • Order of tokens is not significant except that if a token list contains multiple instances of a given token name, the last value takes precedence over earlier values.

Given those rules, the preceding version_tokens_set() call results in a token list with two tokens: tok1 has the value 1‘2 3"4, and tok2 has the value a = b. To verify this, call version_tokens_show():

mysql> SELECT version_tokens_show();
+--------------------------+
| version_tokens_show()    |
+--------------------------+
| tok2=a = b;tok1=1‘2 3"4; |
+--------------------------+

If the token list contains two tokens, why did version_tokens_set() return the value 3 version tokens set? That occurred because the original token list contained two definitions for tok1, and the second definition replaced the first.

The Version Tokens token-manipulation UDFs place these constraints on token names and values:

  • Token names cannot contain = or ; characters and have a maximum length of 64 characters.

  • Token values cannot contain ; characters. Length of values is constrained by the value of the max_allowed_packet system variable.

  • Version Tokens treats token names and values as binary strings, so comparisons are case-sensitive.

Version Tokens also includes a set of UDFs enabling tokens to be locked and unlocked:

Each locking function returns nonzero for success. Otherwise, an error occurs:

mysql> SELECT version_tokens_lock_shared(‘lock1‘, ‘lock2‘, 0);
+-------------------------------------------------+
| version_tokens_lock_shared(‘lock1‘, ‘lock2‘, 0) |
+-------------------------------------------------+
|                                               1 |
+-------------------------------------------------+

mysql> SELECT version_tokens_lock_shared(NULL, 0);
ERROR 3131 (42000): Incorrect locking service lock name ‘(null)‘.

Locking using Version Tokens locking functions is advisory; applications must agree to cooperate.

It is possible to lock nonexisting token names. This does not create the tokens.

Note

Version Tokens locking functions are based on the locking service described at Section 29.3.1, “The Locking Service”, and thus have the same semantics for shared and exclusive locks. (Version Tokens uses the locking service routines built into the server, not the locking service UDF interface, so those UDFs need not be installed to use Version Tokens.) Locks acquired by Version Tokens use a locking service namespace of version_token_locks. Locking service locks can be monitored using the Performance Schema, so this is also true for Version Tokens locks. For details, see Section 29.3.1.2.3, “Locking Service Monitoring”.

For the Version Tokens locking functions, token name arguments are used exactly as specified. Surrounding whitespace is not ignored and = and ; characters are permitted. This is because Version Tokens simply passes the token names to be locked as is to the locking service.

5.6.6.4 Version Tokens Reference

The following discussion serves as a reference to these Version Tokens components:

Version Tokens Functions

The Version Tokens plugin library includes several user-defined functions. One set of UDFs permits the server‘s list of version tokens to be manipulated and inspected. Another set of UDFs permits version tokens to be locked and unlocked. The VERSION_TOKEN_ADMIN or SUPER privilege is required to invoke any Version Tokens UDF.

The following UDFs permit the server‘s list of version tokens to be created, changed, removed, and inspected. Interpretation of name_list and token_list arguments (including whitespace handling) occurs as described in Section 5.6.6.3, “Using Version Tokens”, which provides details about the syntax for specifying tokens, as well as additional examples.

  • version_tokens_delete(name_list)

    Deletes tokens from the server‘s list of version tokens using the name_list argument and returns a binary string that indicates the outcome of the operation. name_list is a semicolon-separated list of version token names to delete.

    mysql> SELECT version_tokens_delete(‘tok1;tok3‘);
    +------------------------------------+
    | version_tokens_delete(‘tok1;tok3‘) |
    +------------------------------------+
    | 2 version tokens deleted.          |
    +------------------------------------+
    

    An argument of NULL is treated as an empty string, which has no effect on the token list.

    version_tokens_delete() deletes the tokens named in its argument, if they exist. (It is not an error to delete nonexisting tokens.) To clear the token list entirely without knowing which tokens are in the list, pass NULL or a string containing no tokens to version_tokens_set():

    mysql> SELECT version_tokens_set(NULL);
    +------------------------------+
    | version_tokens_set(NULL)     |
    +------------------------------+
    | Version tokens list cleared. |
    +------------------------------+
    mysql> SELECT version_tokens_set(‘‘);
    +------------------------------+
    | version_tokens_set(‘‘)       |
    +------------------------------+
    | Version tokens list cleared. |
    +------------------------------+
    
  • version_tokens_edit(token_list)

    Modifies the server‘s list of version tokens using the token_list argument and returns a binary string that indicates the outcome of the operation. token_list is a semicolon-separated list of name=value pairs specifying the name of each token to be defined and its value. If a token exists, its value is updated with the given value. If a token does not exist, it is created with the given value. If the argument is NULL or a string containing no tokens, the token list remains unchanged.

    mysql> SELECT version_tokens_set(‘tok1=value1;tok2=value2‘);
    +-----------------------------------------------+
    | version_tokens_set(‘tok1=value1;tok2=value2‘) |
    +-----------------------------------------------+
    | 2 version tokens set.                         |
    +-----------------------------------------------+
    mysql> SELECT version_tokens_edit(‘tok2=new_value2;tok3=new_value3‘);
    +--------------------------------------------------------+
    | version_tokens_edit(‘tok2=new_value2;tok3=new_value3‘) |
    +--------------------------------------------------------+
    | 2 version tokens updated.                              |
    +--------------------------------------------------------+
    
  • version_tokens_set(token_list)

    Replaces the server‘s list of version tokens with the tokens defined in the token_list argument and returns a binary string that indicates the outcome of the operation. token_list is a semicolon-separated list of name=value pairs specifying the name of each token to be defined and its value. If the argument is NULL or a string containing no tokens, the token list is cleared.

    mysql> SELECT version_tokens_set(‘tok1=value1;tok2=value2‘);
    +-----------------------------------------------+
    | version_tokens_set(‘tok1=value1;tok2=value2‘) |
    +-----------------------------------------------+
    | 2 version tokens set.                         |
    +-----------------------------------------------+
    
  • version_tokens_show()

    Returns the server‘s list of version tokens as a binary string containing a semicolon-separated list of name=value pairs.

    mysql> SELECT version_tokens_show();
    +--------------------------+
    | version_tokens_show()    |
    +--------------------------+
    | tok2=value2;tok1=value1; |
    +--------------------------+
    

The following UDFs permit version tokens to be locked and unlocked:

  • version_tokens_lock_exclusive(token_name[, token_name] ..., timeout)

    Acquires exclusive locks on one or more version tokens, specified by name as strings, timing out with an error if the locks are not acquired within the given timeout value.

    mysql> SELECT version_tokens_lock_exclusive(‘lock1‘, ‘lock2‘, 10);
    +-----------------------------------------------------+
    | version_tokens_lock_exclusive(‘lock1‘, ‘lock2‘, 10) |
    +-----------------------------------------------------+
    |                                                   1 |
    +-----------------------------------------------------+
    
  • version_tokens_lock_shared(token_name[, token_name] ..., timeout)

    Acquires shared locks on one or more version tokens, specified by name as strings, timing out with an error if the locks are not acquired within the given timeout value.

    mysql> SELECT version_tokens_lock_shared(‘lock1‘, ‘lock2‘, 10);
    +--------------------------------------------------+
    | version_tokens_lock_shared(‘lock1‘, ‘lock2‘, 10) |
    +--------------------------------------------------+
    |                                                1 |
    +--------------------------------------------------+
    
  • version_tokens_unlock()

    Releases all locks that were acquired within the current session using version_tokens_lock_exclusive() and version_tokens_lock_shared().

    mysql> SELECT version_tokens_unlock();
    +-------------------------+
    | version_tokens_unlock() |
    +-------------------------+
    |                       1 |
    +-------------------------+
    

The locking functions share these characteristics:

  • The return value is nonzero for success. Otherwise, an error occurs.

  • Token names are strings.

  • In contrast to argument handling for the UDFs that manipulate the server token list, whitespace surrounding token name arguments is not ignored and = and ; characters are permitted.

  • It is possible to lock nonexisting token names. This does not create the tokens.

  • Timeout values are nonnegative integers representing the time in seconds to wait to acquire locks before timing out with an error. If the timeout is 0, there is no waiting and the function produces an error if locks cannot be acquired immediately.

  • Version Tokens locking functions are based on the locking service described at Section 29.3.1, “The Locking Service”.

Version Tokens System Variables

Version Tokens supports the following system variables. These variables are unavailable unless the Version Tokens plugin is installed (see Section 5.6.6.2, “Installing or Uninstalling Version Tokens”).

System variables:

  • version_tokens_session

    PropertyValue
    Command-Line Format --version-tokens-session=value
    System Variable version_tokens_session
    Scope Global, Session
    Dynamic Yes
    SET_VAR Hint Applies No
    Type String
    Default Value NULL

    The session value of this variable specifies the client version token list and indicates the tokens that the client session requires the server version token list to have.

    If the version_tokens_session variable is NULL (the default) or has an empty value, any server version token list matches. (In effect, an empty value disables matching requirements.)

    If the version_tokens_session variable has a nonempty value, any mismatch between its value and the server version token list results in an error for any statement the session sends to the server. A mismatch occurs under these conditions:

    It is not a mismatch for the server version token list to include a token not named in the version_tokens_session value.

    Suppose that a management application has set the server token list as follows:

    mysql> SELECT version_tokens_set(‘tok1=a;tok2=b;tok3=c‘);
    +--------------------------------------------+
    | version_tokens_set(‘tok1=a;tok2=b;tok3=c‘) |
    +--------------------------------------------+
    | 3 version tokens set.                      |
    +--------------------------------------------+
    

    A client registers the tokens it requires the server to match by setting its version_tokens_session value. Then, for each subsequent statement sent by the client, the server checks its token list against the client version_tokens_session value and produces an error if there is a mismatch:

    mysql> SET @@SESSION.version_tokens_session = ‘tok1=a;tok2=b‘;
    mysql> SELECT 1;
    +---+
    | 1 |
    +---+
    | 1 |
    +---+
    
    mysql> SET @@SESSION.version_tokens_session = ‘tok1=b‘;
    mysql> SELECT 1;
    ERROR 3136 (42000): Version token mismatch for tok1. Correct value a
    

    The first SELECT succeeds because the client tokens tok1 and tok2 are present in the server token list and each token has the same value in the server list. The second SELECT fails because, although tok1 is present in the server token list, it has a different value than specified by the client.

    At this point, any statement sent by the client fails, unless the server token list changes such that it matches again. Suppose that the management application changes the server token list as follows:

    mysql> SELECT version_tokens_edit(‘tok1=b‘);
    +-------------------------------+
    | version_tokens_edit(‘tok1=b‘) |
    +-------------------------------+
    | 1 version tokens updated.     |
    +-------------------------------+
    mysql> SELECT version_tokens_show();
    +-----------------------+
    | version_tokens_show() |
    +-----------------------+
    | tok3=c;tok1=b;tok2=b; |
    +-----------------------+
    

    Now the client version_tokens_session value matches the server token list and the client can once again successfully execute statements:

    mysql> SELECT 1;
    +---+
    | 1 |
    +---+
    | 1 |
    +---+
    
  • version_tokens_session_number

    PropertyValue
    Command-Line Format --version-tokens-session-number=#
    System Variable version_tokens_session_number
    Scope Global, Session
    Dynamic No
    SET_VAR Hint Applies No
    Type Integer
    Default Value 0

    This variable is for internal use.

5.6.7 The Clone Plugin

The clone plugin permits cloning data locally or from a remote MySQL server instance. Cloned data is a physical snapshot of data stored in InnoDB that includes schemas, tables, tablespaces, and data dictionary metadata. The cloned data comprises a fully functional data directory, which permits using the clone plugin for MySQL server provisioning.

Figure 5.1 Local Cloning Operation

技术图片

A local cloning operation clones data from the MySQL server instance where the cloning operation is initiated to a directory on the same server or node where MySQL server instance runs.

Figure 5.2 Remote Cloning Operation

技术图片

A remote cloning operation involves a local MySQL server instance (the recipient”) where the cloning operation is initiated, and a remote MySQL server instance (the donor”) where the source data is located. When a remote cloning operation is initiated on the recipient, cloned data is transferred over the network from the donor to the recipient. By default, a remote cloning operation removes the data in the recipient data directory and replaces it with the cloned data. Optionally, you can clone data to a different directory on the recipient to avoid removing existing data.

There is no difference with respect to data that is cloned by a local cloning operation as compared to a remote cloning operation. Both operations clone the same data.

The clone plugin supports replication. In addition to cloning data, a cloning operation extracts and transfers replication coordinates from the donor and applies them on the recipient, which enables using the clone plugin for provisioning Group Replication members and replication slaves. Using the clone plugin for provisioning is considerably faster and more efficient than replicating a large number of transactions (see Section 5.6.7.6, “Cloning for Replication”). Group Replication members can also be configured to use the clone plugin as an alternative method of recovery, so that members automatically choose the most efficient way to retrieve group data from seed members. For more information, see Section 18.4.3.1, “Cloning for Distributed Recovery”.

The clone plugin supports cloning of encrypted and page-compressed data. See Section 5.6.7.4, “Cloning Encrypted Data”, and Section 5.6.7.5, “Cloning Compressed Data”.

The clone plugin must be installed before you can use it. For installation instructions, see Section 5.6.7.1, “Installing the Clone Plugin”. For cloning instructions, see Section 5.6.7.2, “Cloning Data Locally”, and Section 5.6.7.3, “Cloning Remote Data”.

Performance Schema tables and instrumentation are provided for monitoring cloning operations. See Section 5.6.7.9, “Monitoring Cloning Operations”.

5.6.7.1 Installing the Clone Plugin

This section describes how to install and configure the clone plugin. For remote cloning operations, the clone plugin must be installed on the donor and recipient MySQL server instances.

For general information about installing or uninstalling plugins, see Section 5.6.1, “Installing and Uninstalling Plugins”.

To be usable by the server, the plugin library file must be located in the MySQL plugin directory (the directory named by the plugin_dir system variable). If necessary, set the value of plugin_dir at server startup to tell the server the plugin directory location.

The plugin library file base name is mysql_clone.so. The file name suffix differs by platform (for example, .so for Unix and Unix-like systems, .dll for Windows).

To load the plugin at server startup, use the --plugin-load-add option to name the library file that contains it. With this plugin-loading method, the option must be given each time the server starts. For example, put these lines in your my.cnf file (adjust the .so suffix for your platform as necessary):

[mysqld]
plugin-load-add=mysql_clone.so

After modifying my.cnf, restart the server to cause the new settings to take effect.

Note

The --plugin-load-add option cannot be used to load the clone plugin when restarting the server during an upgrade from a previous MySQL version. For example, after upgrading binaries or packages from MySQL 5.7 to MySQL 8.0, attempting to restart the server with plugin-load-add=mysql_clone.so causes this error: [ERROR] [MY-013238] [Server] Error installing plugin ‘clone‘: Cannot install during upgrade. The workaround is to upgrade the server before attempting to start the server with plugin-load-add=mysql_clone.so.

Alternatively, to load the plugin at runtime, use this statement (adjust the .so suffix for your platform as necessary):

INSTALL PLUGIN clone SONAME ‘mysql_clone.so‘;

INSTALL PLUGIN loads the plugin, and also registers it in the mysql.plugins system table to cause the plugin to be loaded for each subsequent normal server startup without the need for --plugin-load-add.

To verify plugin installation, examine the INFORMATION_SCHEMA.PLUGINS table or use the SHOW PLUGINS statement (see Section 5.6.2, “Obtaining Server Plugin Information”). For example:

mysql> SELECT PLUGIN_NAME, PLUGIN_STATUS
       FROM INFORMATION_SCHEMA.PLUGINS
       WHERE PLUGIN_NAME = ‘clone‘;
+------------------------+---------------+
| PLUGIN_NAME            | PLUGIN_STATUS |
+------------------------+---------------+
| clone                  | ACTIVE        |
+------------------------+---------------+

If the plugin fails to initialize, check the server error log for clone or plugin-related diagnostic messages.

If the plugin has been previously registered with INSTALL PLUGIN or is loaded with --plugin-load-add, you can use the --clone option at server startup to control the plugin activation state. For example, to load the plugin at startup and prevent it from being removed at runtime, use these options:

[mysqld]
plugin-load-add=mysql_clone.so
clone=FORCE_PLUS_PERMANENT

If you want to prevent the server from running without the clone plugin, use --clone with a value of FORCE or FORCE_PLUS_PERMANENT to force server startup to fail if the plugin does not initialize successfully.

For more information about plugin activation states, see Controlling Plugin Activation State.

5.6.7.2 Cloning Data Locally

The clone plugin supports the following syntax for cloning data locally; that is, cloning data from the local MySQL data directory to another directory on the same server or node where the MySQL server instance runs:

CLONE LOCAL DATA DIRECTORY [=] ‘clone_dir‘;

To use CLONE syntax, the clone plugin must be installed. For installation instructions, see Section 5.6.7.1, “Installing the Clone Plugin”.

The BACKUP_ADMIN privilege is required to execute CLONE LOCAL DATA DIRECTORY statements.

mysql> GRANT BACKUP_ADMIN ON *.* TO ‘clone_user‘;

where clone_user is the MySQL user that will perform the cloning operation. The user you select to perform the cloning operation can be any MySQL user with the BACKUP_ADMIN privilege on *.*.

The following example demonstrates cloning data locally:

mysql> CLONE LOCAL DATA DIRECTORY = ‘/path/to/clone_dir‘;

where /path/to/clone_dir is the full path of the local directory that data is cloned to. An absolute path is required, and the specified directory (clone_dir”) must not exist, but the specified path must be an existent path. The MySQL server must have the necessary write access to create the directory.

Note

A local cloning operation does not support cloning of user-created tables or tablespaces that reside outside of the data directory. Attempting to clone such tables or tablespaces causes the following error: ERROR 1086 (HY000): File ‘/path/to/tablespace_name.ibd‘ already exists. Cloning a tablespace with the same path as the source tablespace would cause a conflict and is therefore prohibited.

All other user-created InnoDB tables and tablespaces, the InnoDB system tablespace, redo logs, and undo tablespaces are cloned to the specified directory.

If desired, you can start the MySQL server on the cloned directory after the cloning operation is complete.

shell> mysqld_safe --datadir=clone_dir

where clone_dir is the directory that data was cloned to.

For information about monitoring cloning operation status and progress, see Section 5.6.7.9, “Monitoring Cloning Operations”.

5.6.7.3 Cloning Remote Data

The clone plugin supports the following syntax for cloning remote data; that is, cloning data from a remote MySQL server instance (the donor) and transferring it to the MySQL instance where the cloning operation was initiated (the recipient).

CLONE INSTANCE FROM ‘user‘@‘host‘:port
IDENTIFIED BY ‘password‘
[DATA DIRECTORY [=] ‘clone_dir‘]
[REQUIRE [NO] SSL];

where:

  • user is the clone user on the donor MySQL server instance.

  • password is the user password.

  • host is the hostname address of the donor MySQL server instance. Internet Protocol version 6 (IPv6) address format is not supported. An alias to the IPv6 address can be used instead. An IPv4 address can be used as is.

  • port is the port number of the donor MySQL server instance. (The X Protocol port specified by mysqlx_port is not supported. Connecting to the donor MySQL server instance through MySQL Router is also not supported.)

  • DATA DIRECTORY [=] ‘clone_dir is an optional clause used to specify a directory on the recipient for the data you are cloning. Use this option if you do not want to remove existing data in the recipient data directory. An absolute path is required, and the directory must not exist. The MySQL server must have the necessary write access to create the directory.

    When the optional DATA DIRECTORY [=] ‘clone_dir clause is not used, a cloning operation removes existing data in the recipient data directory, replaces it with the cloned data, and automatically restarts the server afterward.

  • [REQUIRE [NO] SSL] explicitly specifies whether an encrypted connection is to be used or not when transferring cloned data over the network. An error is returned if the explicit specification cannot be satisfied. If an SSL clause is not specified, clone attempts to establish an encrypted connection by default, falling back to an insecure connection if the secure connection attempt fails. A secure connection is required when cloning encrypted data regardless of whether this clause is specified. For more information, see Configuring an Encrypted Connection for Cloning.

Note

By default, user-created InnoDB tables and tablespaces that reside in the data directory on the donor MySQL server instance are cloned to the data directory on the recipient MySQL server instance. If the DATA DIRECTORY [=] ‘clone_dir clause is specified, they are cloned to the specified directory.

User-created InnoDB tables and tablespaces that reside outside of the data directory on the donor MySQL server instance are cloned to the same path on the recipient MySQL server instance. An error is reported if a table or tablespace already exists.

By default, the InnoDB system tablespace, redo logs, and undo tablespaces are cloned to the same locations that are configured on the donor (as defined by innodb_data_home_dir and innodb_data_file_path, innodb_log_group_home_dir, and innodb_undo_directory, respectively). If the DATA DIRECTORY [=] ‘clone_dir clause is specified, those tablespaces and logs are cloned to the specified directory.

Remote Cloning Prerequisites

To perform a cloning operation, the clone plugin must be active on both the donor and recipient MySQL server instances. For installation instructions, see Section 5.6.7.1, “Installing the Clone Plugin”.

A MySQL user on the donor and recipient is required for executing the cloning operation (the clone user”).

  • On the donor, the clone user requires the BACKUP_ADMIN privilege for accessing and transferring data from the donor, and for blocking DDL during the cloning operation.

  • On the recipient, the clone user requires the CLONE_ADMIN privilege for replacing recipient data, blocking DDL during the cloning operation, and automatically restarting the server. The CLONE_ADMIN privilege includes BACKUP_ADMIN and SHUTDOWN privileges implicitly.

Instructions for creating the clone user and granting the required privileges are included in the remote cloning example that follows this prerequisite information.

The following prerequisites are checked when the CLONE INSTANCE statement is executed:

  • The donor and recipient must have the same MySQL server version. The clone plugin is supported in MYSQL 8.0.17 and higher.

    mysql> SHOW VARIABLES LIKE ‘version‘;
     +---------------+--------+
    | Variable_name | Value  |
    +---------------+--------+
    | version       | 8.0.17 |
    +---------------+--------+
  • The donor and recipient MySQL server instances must run on the same operating system and platform. For example, if the donor instance runs on a Linux 64-bit platform, the recipient instance must also run on that platform. Refer to your operating system documentation for information about how to determine your operating system platform.

  • The recipient must have enough disk space for the cloned data. By default, recipient data is removed prior to cloning the donor data, so you only require enough space for the donor data. If you clone to a named directory using the DATA DIRECTORY clause, you must have enough disk space for the existing recipient data and the cloned data. You can estimate the size of your data by checking the data directory size on your file system and the size of any tablespaces that reside outside of the data directory. When estimating data size on the donor, remember that only InnoDB data is cloned. If you store data in other storage engines, adjust your data size estimate accordingly.

  • InnoDB permits creating some tablespace types outside of the data directory. If the donor MySQL server instance has tablespaces that reside outside of the data directory, the cloning operation must be able access those tablespaces. You can query the INFORMATION_SCHEMA.FILES table to identify tablespaces that reside outside of the data directory. Files that reside outside of the data directory have a fully qualified path to a directory other than the data directory.

    mysql> SELECT FILE_NAME FROM INFORMATION_SCHEMA.FILES;
  • Plugins that are active on the donor, including any keyring plugin, must also be active on the recipient. You can identify active plugins by issuing a SHOW PLUGINS statement or by querying the INFORMATION_SCHEMA.PLUGINS table.

  • The donor and recipient must have the same MySQL server character set and collation. For information about MySQL server character set and collation configuration, see Section 10.15, “Character Set Configuration”.

  • The same innodb_page_size and innodb_data_file_path settings are required on the donor and recipient. The innodb_data_file_path setting on the donor and recipient must specify the same number of data files of an equivalent size. You can check variable settings using SHOW VARIABLES syntax.

    mysql> SHOW VARIABLES LIKE ‘innodb_page_size‘;
    mysql> SHOW VARIABLES LIKE ‘innodb_data_file_path‘;
  • If cloning encrypted or page-compressed data, the donor and recipient must have the same file system block size. For page-compressed data, the recipient file system must support sparse files and hole punching for hole punching to occur on the recipient. For information about these features and how to identify tables and tablespaces that use them, see Section 5.6.7.4, “Cloning Encrypted Data”, and Section 5.6.7.5, “Cloning Compressed Data”. To determine your file system block size, refer to your operating system documentation.

  • A secure connection is required if you are cloning encrypted data. See Configuring an Encrypted Connection for Cloning.

  • The clone_valid_donor_list setting on the recipient must include the host address of the donor MySQL server instance. You can only clone data from a host on the valid donor list. A MySQL user with the SYSTEM_VARIABLES_ADMIN privilege is required to configure this variable. Instructions for setting the clone_valid_donor_list variable are provided in the remote cloning example that follows this section. You can check the clone_valid_donor_list setting using SHOW VARIABLES syntax.

    mysql> SHOW VARIABLES LIKE ‘clone_valid_donor_list‘;
  • There must be no other cloning operation running. Only a single cloning operation is permitted at a time. To determine if a clone operation is running, query the clone_status table. See Monitoring Cloning Operations using Performance Schema Clone Tables.

  • The clone plugin transfers data in 1MB packets plus metadata. The minimum required max_allowed_packet value is therefore 2MB on the donor and the recipient MySQL server instances. A max_allowed_packet value less than 2MB results in an error. Use the following query to check your max_allowed_packet setting:

    mysql> SHOW VARIABLES LIKE ‘max_allowed_packet‘;

The following prerequisites also apply:

  • Undo tablespace file names on the donor must be unique. When data is cloned to the recipient, undo tablespaces, regardless of their location on the donor, are cloned to the innodb_undo_directory location on the recipient or to the directory specified by the DATA DIRECTORY [=] ‘clone_dir clause, if used. Duplicate undo tablespace file names on the donor are not permitted for this reason. As of MySQL 8.0.18, an error is reported if duplicate undo tablespace file names are encountered during a cloning operation. Prior to MySQL 8.0.18, cloning undo tablespaces with the same file name could result in undo tablespace files being overwritten on the recipient.

    To view undo tablespace file names on the donor to ensure that they are unique, query INFORMATION_SCHEMA.FILES:

    mysql> SELECT TABLESPACE_NAME, FILE_NAME FROM INFORMATION_SCHEMA.FILES 
           WHERE FILE_TYPE LIKE ‘UNDO LOG‘;

    For information about dropping and adding undo tablespace files, see Section 15.6.3.4, “Undo Tablespaces”.

  • By default, the recipient MySQL server instance is restarted (stopped and started) automatically after the data is cloned. For an automatic restart to occur, a monitoring process must be available on the recipient to detect server shutdowns. Otherwise, the cloning operation halts with the following error after the data is cloned, and the recipient MySQL server instance is shut down:

    ERROR 3707 (HY000): Restart server failed (mysqld is not managed by supervisor process).

    This error does not indicate a cloning failure. It means that the recipient MySQL server instance must be started again manually after the data is cloned. After starting the server manually, you can connect to the recipient MySQL server instance and check the Performance Schema clone tables to verify that the cloning operation completed successfully (see Monitoring Cloning Operations using Performance Schema Clone Tables.) The RESTART statement has the same monitoring process requirement. For more information, see Section 13.7.8.8, “RESTART Statement”. This requirement is not applicable if cloning to a named directory using the DATA DIRECTORY clause, as an automatic restart is not performed in this case.

  • Several variables control various aspects of a remote cloning operation. Before performing a remote cloning operation, review the variables and adjust settings as necessary to suit your computing environment. Clone variables are set on recipient MySQL server instance where the cloning operation is executed. See Section 5.6.7.12, “Clone System Variables”.

Cloning Remote Data

The following example demonstrates cloning remote data. By default, a remote cloning operation removes the data in the recipient data directory, replaces it with the cloned data, and restarts the MySQL server afterward.

The example assumes that remote cloning prerequisites are met. See Remote Cloning Prerequisites.

  1. Login to the donor MySQL server instance with an administrative user account.

    1. Create a clone user with the BACKUP_ADMIN privilege.

      mysql> CREATE USER ‘donor_clone_user‘@‘example.donor.host.com‘ IDENTIFIED BY ‘password‘;
      mysql> GRANT BACKUP_ADMIN on *.* to ‘donor_clone_user‘@‘example.donor.host.com‘;
      
    2. Install the clone plugin:

      mysql> INSTALL PLUGIN clone SONAME ‘mysql_clone.so‘;
  2. Login to the recipient MySQL server instance with an administrative user account.

    1. Create a clone user with the CLONE_ADMIN privilege.

      mysql> CREATE USER ‘recipient_clone_user‘@‘example.recipient.host.com‘ IDENTIFIED BY ‘password‘;
      mysql> GRANT CLONE_ADMIN on *.* to ‘recipient_clone_user‘@‘example.recipient.host.com‘;
      
    2. Install the clone plugin:

      mysql> INSTALL PLUGIN clone SONAME ‘mysql_clone.so‘;
    3. Add the host address of the donor MySQL server instance to the clone_valid_donor_list variable setting.

      mysql> SET GLOBAL clone_valid_donor_list = ‘example.donor.host.com:3306‘;
  3. Log on to the recipient MySQL server instance as the clone user you created previously (recipient_clone_user‘@‘example.recipient.host.com) and execute the CLONE INSTANCE statement.

    mysql> CLONE INSTANCE FROM ‘donor_clone_user‘@‘example.donor.host.com‘:3306
           IDENTIFIED BY ‘password‘;
    

    After the data is cloned, the MySQL server instance on the recipient is restarted automatically.

    For information about monitoring cloning operation status and progress, see Section 5.6.7.9, “Monitoring Cloning Operations”.

Cloning to a Named Directory

By default, a remote cloning operation removes the data in the recipient data directory and replaces it with the cloned data. By cloning to a named directory, you can avoid removing existing data from the recipient data directory.

The procedure for cloning to a named directory is the same procedure described in Cloning Remote Data with one exception: The CLONE INSTANCE statement must include the DATA DIRECTORY clause. For example:

mysql> CLONE INSTANCE FROM ‘user‘@‘example.donor.host.com‘:3306
       IDENTIFIED BY ‘password‘
       DATA DIRECTORY = ‘/path/to/clone_dir‘;

An absolute path is required, and the directory must not exist. The MySQL server must have the necessary write access to create the directory.

When cloning to a named directory, the recipient MySQL server instance is not restarted automatically after the data is cloned. If you want to restart the MySQL server on the named directory, you must do so manually:

shell> mysqld_safe --datadir=/path/to/clone_dir

where /path/to/clone_dir is the path to the named directory on the recipient.

Configuring an Encrypted Connection for Cloning

You can configure an encrypted connection for remote cloning operations to protect data as it is cloned over the network. An encrypted connection is required by default when cloning encrypted data. (see Section 5.6.7.4, “Cloning Encrypted Data”.)

The instructions that follow describe how to configure the recipient MySQL server instance to use an encrypted connection. It is assumed that the donor MySQL server instance is already configured to use encrypted connections. If not, refer to Section 6.3.1, “Configuring MySQL to Use Encrypted Connections” for server-side configuration instructions.

To configure the recipient MySQL server instance to use an encrypted connection:

  1. Make the client certificate and key files of the donor MySQL server instance available to the recipient host. Either distribute the files to the recipient host using a secure channel or place them on a mounted partition that is accessible to the recipient host. The client certificate and key files to make available include:

    • ca.pem

      The self-signed certificate authority (CA) file.

    • client-cert.pem

      The client public key certificate file.

    • client-key.pem

      The client private key file.

  2. Configure the following SSL options on the recipient MySQL server instance.

    • clone_ssl_ca

      Specifies the path to the self-signed certificate authority (CA) file.

    • clone_ssl_cert

      Specifies the path to the client public key certificate file.

    • clone_ssl_key

      Specifies the path to the client private key file.

    For example:

    clone_ssl_ca=/path/to/ca.pem   
    clone_ssl_cert=/path/to/client-cert.pem
    clone_ssl_key=/path/to/client-key.pem
    
  3. To require that an encrypted connection is used, include the REQUIRE SSL clause when issuing the CLONE statement on the recipient.

    mysql> CLONE INSTANCE FROM ‘user‘@‘example.donor.host.com‘:3306
           IDENTIFIED BY ‘password‘
           DATA DIRECTORY = ‘/path/to/clone_dir‘
           REQUIRE SSL;
    

    If an SSL clause is not specified, the clone plugin attempts to establish an encrypted connection by default, falling back to an unencrypted connection if the encrypted connection attempt fails.

    Note

    If you are cloning encrypted data, an encrypted connection is required by default regardless of whether the REQUIRE SSL clause is specified. Using REQUIRE NO SSL causes an error if you attempt to clone encrypted data.

5.6.7.4 Cloning Encrypted Data

Cloning of encrypted data is supported. The following requirements apply:

  • A secure connection is required when cloning remote data to ensure safe transfer of unencrypted tablespace keys over the network. Tablespace keys are decrypted at the donor before transport and re-encrypted at the recipient using the recipient master key. An error is reported if an encrypted connection is not available or the REQUIRE NO SSL clause is used in the CLONE INSTANCE statement. For information about configuring an encrypted connection for cloning, see Configuring an Encrypted Connection for Cloning.

  • When cloning data to a local data directory that uses a locally managed keyring, the same keyring must be used when starting the MySQL server on the clone directory.

  • When cloning data to a remote data directory (the recipient directory) that uses a locally managed keyring, the recipient keyring must be used when starting the MySQL sever on the cloned directory.

Note

The innodb_redo_log_encrypt and innodb_undo_log_encrypt variable settings cannot be modified while a cloning operation is in progress.

For information about the data encryption feature, see Section 15.13, “InnoDB Data-at-Rest Encryption”.

5.6.7.5 Cloning Compressed Data

Cloning of page-compressed data is supported. The following requirements apply when cloning remote data:

  • The recipient file system must support sparse files and hole punching for hole punching to occur on the recipient.

  • The donor and recipient file systems must have the same block size. If file system block sizes differ, an error similar to the following is reported: ERROR 3868 (HY000): Clone Configuration FS Block Size: Donor value: 114688 is different from Recipient value: 4096.

For information about the page compression feature, see Section 15.9.2, “InnoDB Page Compression”.

5.6.7.6 Cloning for Replication

The clone plugin supports replication. In addition to cloning data, a cloning operation extracts replication coordinates from the donor and transfers them to the recipient, which enables using the clone plugin for provisioning Group Replication members and replication slaves. Using the clone plugin for provisioning is considerably faster and more efficient than replicating a large number of transactions.

Group Replication members can also be configured to use the clone plugin as an option for distributed recovery, in which case joining members automatically choose the most efficient way to retrieve group data from existing group members. For more information, see Section 18.4.3.1, “Cloning for Distributed Recovery”.

During the cloning operation, both the binary log position (filename, offset) and the gtid_executed GTID set are extracted and transferred from the donor MySQL server instance to the recipient. This data permits initiating replication at a consistent position in the replication stream. The binary logs and relay logs, which are held in files, are not copied from the donor to the recipient. To initiate replication, the binary logs required for the recipient to catch up to the donor must not be purged between the time that the data is cloned and the time that replication is started. If the required binary logs are not available, a replication handshake error is reported. A cloned instance should therefore be added to a replication group without excessive delay to avoid required binary logs being purged or the new member lagging behind significantly, requiring more recovery time.

  • Issue this query on a cloned MySQL server instance to check the binary log position that was transferred to the recipient:

    mysql> SELECT BINLOG_FILE, BINLOG_POSITION FROM performance_schema.clone_status;
  • Issue this query on a cloned MySQL server instance to check the gtid_executed GTID set that was transferred to the recipient:

    mysql> SELECT @@GLOBAL.GTID_EXECUTED;

When master_info_repository=TABLE and relay_log_info_repository=TABLE are set on the recipient (which is the default in MySQL 8.0), the slave status logs are held in tables that are copied from the donor to the recipient during the cloning operation. The slave status logs hold replication-related configuration settings that can be used to resume replication correctly after the cloning operation.

  • In MySQL 8.0.17 and 8.0.18, only the table mysql.slave_master_info (the master info log) is copied.

  • From MySQL 8.0.19, the tables mysql.slave_relay_log_info (the relay log info log) and mysql.slave_worker_info (the slave worker log) are also copied.

For a list of what is included in each table, see Section 17.2.4.2, “Slave Status Logs”. Note that if master_info_repository=FILE and relay_log_info_repository=FILE are set on the server (which is not the default in MySQL 8.0 and is deprecated), the slave status logs are not cloned; they are only cloned if TABLE is set.

To clone for replication, perform the following steps:

  1. For a new group member for Group Replication, first configure the MySQL Server instance for Group Replication, following the instructions in Section 18.2.1.6, “Adding Instances to the Group”. Also set up the prerequisites for cloning described in Section 18.4.3.1, “Cloning for Distributed Recovery”. When you issue START GROUP_REPLICATION on the joining member, the cloning operation is managed automatically by Group Replication, so you do not need to carry out the operation manually, and you do not need to perform any further setup steps on the joining member.

  2. For a slave in a master/slave MySQL replication topology, first clone the data from the donor MySQL server instance to the recipient manually. The donor must be a master or slave in the replication topology. For cloning instructions, see Section 5.6.7.3, “Cloning Remote Data”.

  3. After the cloning operation completes successfully, if you want to use the same replication channels on the recipient MySQL server instance that were present on the donor, verify which of them can resume replication automatically in the master/slave MySQL replication topology, and which need to be set up manually.

    • For GTID-based replication, if the recipient is configured with gtid_mode=ON and has cloned from a donor with gtid_mode=ON, ON_PERMISSIVE, or OFF_PERMISSIVE, the gtid_executed GTID set from the donor is applied on the recipient. If the recipient is cloned from a slave already in the topology, replication channels on the recipient that use GTID auto-positioning (as specified by the MASTER_AUTO_POSITION option on the CHANGE MASTER TO statement) can resume replication automatically after the cloning operation when the channel is started. You do not need to perform any manual setup if you just want to use these same channels.

    • For binary log file position based replication, if the recipient is at MySQL 8.0.17 or 8.0.18, the binary log position from the donor is not applied on the recipient, only recorded in the Performance Schema clone_status table. Replication channels on the recipient that use binary log file position based replication must therefore be set up manually to resume replication after the cloning operation. Ensure that these channels are not configured to start replication automatically at server startup, as they will not have the binary log position and will attempt to start replication from the beginning.

    • For binary log file position based replication, if the recipient is at MySQL 8.0.19 or above, the binary log position from the donor is applied on the recipient. Replication channels on the recipient that use binary log file position based replication automatically attempt to carry out the relay log recovery process, using the cloned relay log information, before restarting replication. For a single-threaded slave (slave_parallel_workers is set to 0), relay log recovery should succeed in the absence of any other issues, enabling the channel to resume replication with no further setup. For a multithreaded slave (slave_parallel_workers is greater than 0), relay log recovery is likely to fail because it cannot usually be completed automatically. In this case, an error message is issued, and you must set the channel up manually.

  4. If you need to set up cloned replication channels manually, or want to use different replication channels on the recipient, the following instructions provide a summary and abbreviated examples for adding a recipient MySQL server instance to a replication topology. Also refer to the detailed instructions that apply to your replication setup.

    • To add a recipient MySQL server instance to a MySQL replication topology that uses GTID-based transactions as the replication data source, configure the instance as required, following the instructions in Section 17.1.3.4, “Setting Up Replication Using GTIDs”. Add replication channels for the instance as shown in the following abbreviated example. The CHANGE MASTER TO statement must define the host address and port number of the master, and the MASTER_AUTO_POSITION option should be enabled, as shown:

      mysql> CHANGE MASTER TO MASTER_HOST = ‘master_host_name‘, MASTER_PORT = master_port_num,
             ...
             MASTER_AUTO_POSITION = 1,
             FOR CHANNEL ‘setup_channel‘;
      mysql> START SLAVE USER = ‘user_name‘ PASSWORD = ‘password‘ FOR CHANNEL ‘setup_channel‘;  
      
    • To add a recipient MySQL server instance to a MySQL replication topology that uses binary log file position based replication, configure the instance as required, following the instructions in Section 17.1.2, “Setting Up Binary Log File Position Based Replication”. Add replication channels for the instance as shown in the following abbreviated example, using the binary log position that was transferred to the recipient during the cloning operation:

      mysql> SELECT BINLOG_FILE, BINLOG_POSITION FROM performance_schema.clone_status;  
      mysql> CHANGE MASTER TO MASTER_HOST = ‘master_host_name‘, MASTER_PORT = master_port_num,
             ...
             MASTER_LOG_FILE = ‘master_log_name‘,
             MASTER_LOG_POS = master_log_pos, 
             FOR CHANNEL ‘setup_channel‘;
      mysql> START SLAVE USER = ‘user_name‘ PASSWORD = ‘password‘ FOR CHANNEL ‘setup_channel‘;  
      

5.6.7.7 Directories and Files Created During a Cloning Operation

When data is cloned, the following directories and files are created for internal use. They should not be modified.

  • #clone: Contains internal clone files used by the cloning operation. Created in the directory that data is cloned to.

  • #ib_archive: Contains internally archived log files, archived on the donor during the cloning operation.

  • *.#clone files: Temporary data files created on the recipient while the existing data directory is replaced by a remote cloning operation.

5.6.7.8 Remote Cloning Operation Failure Handling

This section describes failure handing at different stages of a cloning operation.

  1. Prerequisites are checked (see Remote Cloning Prerequisites).

    • If a failure occurs during the prerequisite check, the CLONE INSTANCE operation reports an error.

  2. A backup lock is taken to block DDL operations.

    • If the cloning operation is unable to obtain a DDL lock within the time limit specified by the clone_ddl_timeout variable, an error is reported.

  3. User-created data (schemas, tables, tablespaces) and binary logs on the recipient are removed before data is cloned to the recipient data directory.

    • When user created data is removed from the recipient during a remote cloning operation, existing data in the recipient data directory is not saved and may be lost if a failure occurs. If the data to be replaced on the recipient is of importance, a backup should be taken before initiating a remote cloning operation.

      For informational purposes, warnings are printed to the server error log to specify when data removal starts and finishes:

      [Warning] [MY-013453] [InnoDB] Clone removing all user data for provisioning:
      Started...
      
      [Warning] [MY-013453] [InnoDB] Clone removing all user data for provisioning:
      Finished

      If a failure occurs while removing data, the recipient may be left with a partial set of schemas, tables, and tablespaces that existed before the cloning operation. Any time during the execution of a cloning operation or after a failure, the server is always in a consistent state.

  4. Data is cloned from the donor. User-created data, dictionary metadata, and other system data are cloned.

    • If a failure occurs while cloning data, the cloning operation is rolled back and all cloned data removed. At this stage, the previously existing data on the recipient has also been removed, which leaves the recipient with no user data.

      Should this scenario occur, you can either rectify the cause of the failure and re-execute the cloning operation, or forgo the cloning operation and restore the recipient data from a backup taken before the cloning operation.

  5. The server is restarted automatically (applies to remote cloning operations that do not clone to a named directory). During startup, typical server startup tasks are performed.

    • If the automatic server restart fails, you can restart the server manually to complete the cloning operation.

If a network error occurs during a cloning operation, the operation resumes if the error is resolved within five minutes. Otherwise, the operation aborts and returns an error.

5.6.7.9 Monitoring Cloning Operations

This section describes options for monitoring cloning operations.

Monitoring Cloning Operations using Performance Schema Clone Tables

A cloning operation may take some time to complete, depending on the amount of data and other factors related to data transfer. You can monitor the status and progress of a cloning operation on the recipient MySQL server instance using the clone_status and clone_progress Performance Schema tables.

Note

The clone_status and clone_progress Performance Schema tables can be used to monitor a cloning operation on the recipient MySQL server instance only. To monitor a cloning operation on the donor MySQL server instance, use the clone stage events, as described in Monitoring Cloning Operations Using Performance Schema Stage Events.

  • The clone_status table provides the state of the current or last executed cloning operation. A clone operation has four possible states: Not Started, In Progress, Completed, and Failed.

  • The clone_progress table provides progress information for the current or last executed clone operation, by stage. The stages of a cloning operation include DROP DATA, FILE COPY, PAGE_COPY, REDO_COPY, FILE_SYNC, RESTART, and RECOVERY.

The SELECT and EXECUTE privileges on the Performance Schema is required to access the Performance Schema clone tables.

To check the state of a cloning operation:

  1. Connect to the recipient MySQL server instance.

  2. Query the clone_status table:

    mysql> SELECT STATE FROM performance_schema.clone_status;
    +-----------+
    | STATE     |
    +-----------+
    | Completed |
    +-----------+
    

Should a failure occur during a cloning operation, you can query the clone_status table for error information:

mysql> SELECT STATE, ERROR_NO, ERROR_MESSAGE FROM performance_schema.clone_status;
+-----------+----------+---------------+
| STATE     | ERROR_NO | ERROR_MESSAGE |
+-----------+----------+---------------+
| Failed    |      xxx | "xxxxxxxxxxx" |
+-----------+----------+---------------+

To review the details of each stage of a cloning operation:

  1. Connect to the recipient MySQL server instance.

  2. Query the clone_progress table. For example, the following query provides state and end time data for each stage of the cloning operation:

    mysql> SELECT STAGE, STATE, END_TIME FROM performance_schema.clone_progress;
    +-----------+-----------+----------------------------+
    | stage     | state     | end_time                   |
    +-----------+-----------+----------------------------+
    | DROP DATA | Completed | 2019-01-27 22:45:43.141261 |
    | FILE COPY | Completed | 2019-01-27 22:45:44.457572 |
    | PAGE COPY | Completed | 2019-01-27 22:45:44.577330 |
    | REDO COPY | Completed | 2019-01-27 22:45:44.679570 |
    | FILE SYNC | Completed | 2019-01-27 22:45:44.918547 |
    | RESTART   | Completed | 2019-01-27 22:45:48.583565 |
    | RECOVERY  | Completed | 2019-01-27 22:45:49.626595 |
    +-----------+-----------+----------------------------+
    

    For other clone status and progress data points that you can monitor, refer to Section 26.12.16, “Performance Schema Clone Tables”.

Monitoring Cloning Operations Using Performance Schema Stage Events

A cloning operation may take some time to complete, depending on the amount of data and other factors related to data transfer. There are three stage events for monitoring the progress of a cloning operation. Each stage event reports WORK_COMPLETED and WORK_ESTIMATED values. Reported values are revised as the operation progresses.

This method of monitoring a cloning operation can be used on the donor or recipient MySQL server instance.

In order of occurrence, cloning operation stage events include:

  • stage/innodb/clone (file copy): Indicates progress of the file copy phase of the cloning operation. WORK_ESTIMATED and WORK_COMPLETED units are file chunks. The number of files to be transferred is known at the start of the file copy phase, and the number of chunks is estimated based on the number of files. WORK_ESTIMATED is set to the number of estimated file chunks. WORK_COMPLETED is updated after each chunk is sent.

  • stage/innodb/clone (page copy): Indicates progress of the page copy phase of cloning operation. WORK_ESTIMATED and WORK_COMPLETED units are pages. Once the file copy phase is completed, the number of pages to be transferred is known, and WORK_ESTIMATED is set to this value. WORK_COMPLETED is updated after each page is sent.

  • stage/innodb/clone (redo copy): Indicates progress of the redo copy phase of cloning operation. WORK_ESTIMATED and WORK_COMPLETED units are redo chunks. Once the page copy phase is completed, the number of redo chunks to be transferred is known, and WORK_ESTIMATED is set to this value. WORK_COMPLETED is updated after each chunk is sent.

The following example demonstrates how to enable stage/innodb/clone% event instruments and related consumer tables to monitor a cloning operation. For information about Performance Schema stage event instruments and related consumers, see Section 26.12.5, “Performance Schema Stage Event Tables”.

  1. Enable the stage/innodb/clone% instruments:

    mysql> UPDATE performance_schema.setup_instruments SET ENABLED = ‘YES‘
           WHERE NAME LIKE ‘stage/innodb/clone%‘;
    
  2. Enable the stage event consumer tables, which include events_stages_current, events_stages_history, and events_stages_history_long.

    mysql> UPDATE performance_schema.setup_consumers SET ENABLED = ‘YES‘
           WHERE NAME LIKE ‘%stages%‘;
    
  3. Run a cloning operation. In this example, a local data directory is cloned to a directory named cloned_dir.

    mysql> CLONE LOCAL DATA DIRECTORY = ‘/path/to/cloned_dir‘;
    
  4. Check the progress of the cloning operation by querying the Performance Schema events_stages_current table. The stage event shown differs depending on the cloning phase that is in progress. The WORK_COMPLETED column shows the work completed. The WORK_ESTIMATED column shows the work required in total.

    mysql> SELECT EVENT_NAME, WORK_COMPLETED, WORK_ESTIMATED FROM performance_schema.events_stages_current
           WHERE EVENT_NAME LIKE ‘stage/innodb/clone%‘;
    +--------------------------------+----------------+----------------+
    | EVENT_NAME                     | WORK_COMPLETED | WORK_ESTIMATED |
    +--------------------------------+----------------+----------------+
    | stage/innodb/clone (redo copy) |              1 |              1 |
    +--------------------------------+----------------+----------------+
    

    The events_stages_current table returns an empty set if the cloning operation has finished. In this case, you can check the events_stages_history table to view event data for the completed operation. For example:

    mysql> SELECT EVENT_NAME, WORK_COMPLETED, WORK_ESTIMATED FROM events_stages_history
           WHERE EVENT_NAME LIKE ‘stage/innodb/clone%‘;
    +--------------------------------+----------------+----------------+
    | EVENT_NAME                     | WORK_COMPLETED | WORK_ESTIMATED |
    +--------------------------------+----------------+----------------+
    | stage/innodb/clone (file copy) |            301 |            301 |
    | stage/innodb/clone (page copy) |              0 |              0 |
    | stage/innodb/clone (redo copy) |              1 |              1 |
    +--------------------------------+----------------+----------------+
    
Monitoring Cloning Operations Using Performance Schema Clone Instrumentation

Performance Schema provides instrumentation for advanced performance monitoring of clone operations. To view the available clone instrumentation, issue the following query:

mysql> SELECT * FROM performance_schema.setup_instruments
       WHERE NAME LIKE WHERE NAME LIKE ‘%clone%‘;
+----------------------------------------------+---------+
| NAME                                         | ENABLED |
+----------------------------------------------+---------+
| wait/synch/mutex/innodb/clone_snapshot_mutex | NO      |
| wait/synch/mutex/innodb/clone_sys_mutex      | NO      |
| wait/synch/mutex/innodb/clone_task_mutex     | NO      |
| wait/io/file/innodb/innodb_clone_file        | YES     |
| stage/innodb/clone (file copy)               | YES     |
| stage/innodb/clone (redo copy)               | YES     |
| stage/innodb/clone (page copy)               | YES     |
| statement/abstract/clone                     | YES     |
| statement/clone/local                        | YES     |
| statement/clone/client                       | YES     |
| statement/clone/server                       | YES     |
| memory/innodb/clone                          | YES     |
| memory/clone/data                            | YES     |
+----------------------------------------------+---------+
Wait Instruments

Performance schema wait instruments track events that take time. Clone wait event instruments include:

  • wait/synch/mutex/innodb/clone_snapshot_mutex: Tracks wait events for the clone snapshot mutex, which synchronizes access to the dynamic snapshot object (on the donor and recipient) between multiple clone threads.

  • wait/synch/mutex/innodb/clone_sys_mutex: Tracks wait events for the clone sys mutex. There is one clone system object in a MySQL server instance. This mutex synchronizes access to the clone system object on the donor and recipient. It is acquired by clone threads and other foreground and background threads.

  • wait/synch/mutex/innodb/clone_task_mutex: Tracks wait events for the clone task mutex, used for clone task management. The clone_task_mutex is acquired by clone threads.

  • wait/io/file/innodb/innodb_clone_file: Tracks all I/O wait operations for files that clone operates on.

For information about monitoring InnoDB mutex waits, see Section 15.16.2, “Monitoring InnoDB Mutex Waits Using Performance Schema”. For information about monitoring wait events in general, see Section 26.12.4, “Performance Schema Wait Event Tables”.

Stage Instruments

Performance Schema stage events track steps that occur during the statement-execution process. Clone stage event instruments include:

  • stage/innodb/clone (file copy): Indicates progress of the file copy phase of the cloning operation.

  • stage/innodb/clone (redo copy): Indicates progress of the redo copy phase of cloning operation.

  • stage/innodb/clone (page copy): Indicates progress of the page copy phase of cloning operation.

For information about monitoring cloning operations using stage events, see Monitoring Cloning Operations Using Performance Schema Stage Events. For general information about monitoring stage events, see Section 26.12.5, “Performance Schema Stage Event Tables”.

Statement Instruments

Performance Schema statement events track statement execution. When a clone operation is initiated, the different statement types tracked by clone statement instruments may be executed in parallel. You can observe these statement events in the Performance Schema statement event tables. The number of statements that execute depends on the clone_max_concurrency and clone_autotune_concurrency settings.

Clone statement event instruments include:

  • statement/abstract/clone: Tracks statement events for any clone operation before it is classified as a local, client, or server operation type.

  • statement/clone/local: Tracks clone statement events for local clone operations; generated when executing a CLONE LOCAL statement.

  • statement/clone/client: Tracks remote cloning statement events that occur on the recipient MySQL server instance; generated when executing a CLONE INSTANCE statement on the recipient.

  • statement/clone/server: Tracks remote cloning statement events that occur on the donor MySQL server instance; generated when executing a CLONE INSTANCE statement on the recipient.

For information about monitoring Performance Schema statement events, see Section 26.12.6, “Performance Schema Statement Event Tables”.

Memory Instruments

Performance Schema memory instruments track memory usage. Clone memory usage instruments include:

  • memory/innodb/clone: Tracks memory allocated by InnoDB for the dynamic snapshot.

  • memory/clone/data: Tracks memory allocated by the clone plugin during a clone operation.

For information about monitoring memory usage using Performance Schema, see Section 26.12.17.10, “Memory Summary Tables”.

The Com_clone Status Variable

The Com_clone status variable provides a count of CLONE statement executions.

For more information, refer to the discussion about Com_xxx statement counter variables in Section 5.1.10, “Server Status Variables”.

5.6.7.10 Stopping a Cloning Operation

If necessary, you can stop a cloning operation with a KILL QUERY processlist_id statement.

On the recipient MySQL server instance, you can retrieve the processlist identifier (PID) for a cloning operation from the PID column of the clone_status table.

mysql> SELECT * FROM performance_schema.clone_status\G
*************************** 1. row ***************************
             ID: 1
            PID: 8
          STATE: In Progress
     BEGIN_TIME: 2019-07-15 11:58:36.767
       END_TIME: NULL
         SOURCE: LOCAL INSTANCE
    DESTINATION: /path/to/clone_dir/
       ERROR_NO: 0
  ERROR_MESSAGE: 
    BINLOG_FILE: 
BINLOG_POSITION: 0
  GTID_EXECUTED: 

You can also retrieve the processlist identifier from the ID column of the INFORMATION_SCHEMA PROCESSLIST table, the Id column of SHOW PROCESSLIST output, or the PROCESSLIST_ID column of the Performance Schema threads table. These methods of obtaining the PID information can be used on the donor or recipient MySQL server instance.

5.6.7.11 Clone System Variable Reference

Table 5.5 Clone System Variable Reference

NameCmd-LineOption FileSystem VarStatus VarVar ScopeDynamic
clone_autotune_concurrency Yes Yes Yes   Global Yes
clone_buffer_size Yes Yes Yes   Global Yes
clone_ddl_timeout Yes Yes Yes   Global Yes
clone_enable_compression Yes Yes Yes   Global Yes
clone_max_concurrency Yes Yes Yes   Global Yes
clone_max_data_bandwidth Yes Yes Yes   Global Yes
clone_max_network_bandwidth Yes Yes Yes   Global Yes
clone_ssl_ca Yes Yes Yes   Global Yes
clone_ssl_cert Yes Yes Yes   Global Yes
clone_ssl_key Yes Yes Yes   Global Yes
clone_valid_donor_list Yes Yes Yes   Global Yes

5.6.7.12 Clone System Variables

This section describes the system variables that control operation of the clone plugin. If values specified at startup are incorrect, the clone plugin may fail to initialize properly and the server does not load it. In this case, the server may also produce error messages for other clone settings because it will not recognize them.

Each system variable has a default value. System variables can be set at server startup using options on the command line or in an option file. They can be changed dynamically at runtime using the SET statement, which enables you to modify operation of the server without having to stop and restart it.

Setting a global system variable runtime value normally requires the SYSTEM_VARIABLES_ADMIN or SUPER privilege. For more information, see Section 5.1.9.1, “System Variable Privileges”.

Clone variables are configured on the recipient MySQL server instance where the cloning operation is executed.

  • clone_autotune_concurrency

    PropertyValue
    Command-Line Format --clone-autotune-concurrency
    Introduced 8.0.17
    System Variable clone_autotune_concurrency
    Scope Global
    Dynamic Yes
    SET_VAR Hint Applies No
    Type Boolean
    Default Value ON

    Enables dynamic spawning of threads for remote cloning operations. The setting is applicable to recipient MySQL server instance only. The clone_max_concurrency variable defines the maximum number of threads that can be spawned.

    If clone_autotune_concurrency is disabled, clone_max_concurrency defines the actual number of threads spawned for a remote cloning operation.

  • clone_buffer_size

    PropertyValue
    Command-Line Format --clone-buffer-size
    Introduced 8.0.17
    System Variable clone_buffer_size
    Scope Global
    Dynamic Yes
    SET_VAR Hint Applies No
    Type Integer
    Default Value 4194304
    Minimum Value 1048576
    Maximum Value 268435456

    Defines the size of the intermediate buffer used when transferring data during a local cloning operation. This setting is not applicable to remote cloning operations. The default value is 4 mebibytes (MiB). A larger buffer size may permit I/O device drivers to fetch data in parallel, which can improve cloning performance.

  • clone_ddl_timeout

    PropertyValue
    Command-Line Format --clone-ddl-timeout
    Introduced 8.0.17
    System Variable clone_ddl_timeout
    Scope Global
    Dynamic Yes
    SET_VAR Hint Applies No
    Type Integer
    Default Value 300
    Minimum Value 0
    Maximum Value 2592000

    The time in seconds to wait for a backup lock when executing a cloning operation. This setting is applied on both the donor and recipient MySQL server instances. A cloning operation cannot run concurrently with DDL operations. A backup lock is required on the donor and recipient MySQL server instances. The cloning operation waits for current DDL operations to finish. Once backup locks are acquired, DDL operations must wait for the cloning operation to finish. A value of 0 means that no backup lock is to be taken for the cloning operation. In this case, the cloning operation fails with an error if a DDL operation is attempted concurrently.

  • clone_enable_compression

    PropertyValue
    Command-Line Format --clone-enable-compression
    Introduced 8.0.17
    System Variable clone_enable_compression
    Scope Global
    Dynamic Yes
    SET_VAR Hint Applies No
    Type Boolean
    Default Value OFF

    Enables compression of data at the network layer during a remote cloning operation. Compression saves network bandwidth at the cost of CPU. Enabling compression may improve the data transfer rate. This setting is only applied on the recipient MySQL server instance.

  • clone_max_concurrency

    PropertyValue
    Command-Line Format --clone-max-concurrency
    Introduced 8.0.17
    System Variable clone_max_concurrency
    Scope Global
    Dynamic Yes
    SET_VAR Hint Applies No
    Type Integer
    Default Value 16
    Minimum Value 1
    Maximum Value 128

    Defines the maximum number of concurrent threads for a remote cloning operation. The default value is 16. A greater number of threads can improve cloning performance but also reduces the number of permitted simultaneous client connections, which can affect the performance of existing client connections. This setting is only applied on the recipient MySQL server instance.

    If clone_autotune_concurrency is enabled (the default), clone_max_concurrency is the maximum number of threads that can be dynamically spawned for a remote cloning operation. If clone_autotune_concurrency is disabled, clone_max_concurrency defines the actual number of threads spawned for a remote cloning operation.

    A minimum data transfer rate of 1 mebibyte (MiB) per thread is recommended for remote cloning operations. The data transfer rate for a remote cloning operation is controlled by the clone_max_data_bandwidth variable.

  • clone_max_data_bandwidth

    PropertyValue
    Command-Line Format --clone-max-data-bandwidth
    Introduced 8.0.17
    System Variable clone_max_data_bandwidth
    Scope Global
    Dynamic Yes
    SET_VAR Hint Applies No
    Type Integer
    Default Value 0
    Minimum Value 0
    Maximum Value 1048576

    Defines the maximum data transfer rate in mebibytes (MiB) per second for a remote cloning operation. This variable helps manage the performance impact of a cloning operation. A limit should be set only when donor disk I/O bandwidth is saturated, affecting performance. A value of 0 means unlimited”, which permits cloning operations to run at the highest possible data transfer rate. This setting is only applicable to the recipient MySQL server instance.

    The minimum data transfer rate is 1 MiB per second, per thread. For example, if there are 8 threads, the minimum transfer rate is 8 MiB per second. The clone_max_concurrency variable controls the maximum number threads spawned for a remote cloning operation.

    The requested data transfer rate specified by clone_max_data_bandwidth may differ from the actual data transfer rate reported by the DATA_SPEED column in the performance_schema.clone_progress table. If your cloning operation is not achieving the desired data transfer rate and you have available bandwidth, check I/O usage on the recipient and donor. If there is underutilized bandwidth, I/O is the next mostly likely bottleneck.

  • clone_max_network_bandwidth

    PropertyValue
    Command-Line Format --clone-max-network-bandwidth
    Introduced 8.0.17
    System Variable clone_max_network_bandwidth
    Scope Global
    Dynamic Yes
    SET_VAR Hint Applies No
    Type Integer
    Default Value 0
    Minimum Value 0
    Maximum Value 1048576

    Specifies the maximum approximate network transfer rate in mebibytes (MiB) per second for a remote cloning operation. This variable can be used to manage the performance impact of a cloning operation on network bandwidth. It should be set only when network bandwidth is saturated, affecting performance on the donor instance. A value of 0 means unlimited”, which permits cloning at the highest possible data transfer rate over the network, providing the best performance. This setting is only applicable to the recipient MySQL server instance.

  • clone_ssl_ca

    PropertyValue
    Command-Line Format --clone-ssl-ca=file_name
    Introduced 8.0.14
    System Variable clone_ssl_ca
    Scope Global
    Dynamic Yes
    SET_VAR Hint Applies No
    Type File name
    Default Value empty string

    Specifies the path to the certificate authority (CA) file. Used to configure an encrypted connection for a remote cloning operation. This setting configured on the recipient and used when connecting to the donor.

  • clone_ssl_cert

    PropertyValue
    Command-Line Format --clone-ssl-cert=file_name
    Introduced 8.0.14
    System Variable clone_ssl_cert
    Scope Global
    Dynamic Yes
    SET_VAR Hint Applies No
    Type File name
    Default Value empty string

    Specifies the path to the public key certificate. Used to configure an encrypted connection for a remote cloning operation. This setting configured on the recipient and used when connecting to the donor.

  • clone_ssl_key

    PropertyValue
    Command-Line Format --clone-ssl-key=file_name
    Introduced 8.0.14
    System Variable clone_ssl_key
    Scope Global
    Dynamic Yes
    SET_VAR Hint Applies No
    Type File name
    Default Value empty string

    Specifies the path to the private key file. Used to configure an encrypted connection for a remote cloning operation. This setting configured on the recipient and used when connecting to the donor.

  • clone_valid_donor_list

    PropertyValue
    Command-Line Format --clone-valid-donor-list=value
    Introduced 8.0.17
    System Variable clone_valid_donor_list
    Scope Global
    Dynamic Yes
    SET_VAR Hint Applies No
    Type String
    Default Value NULL

    Defines valid donor host addresses for remote cloning operations. This setting is applied on the recipient MySQL server instance. A comma-separated list of values is permitted in the following format: HOST1:PORT1,HOST2:PORT2,HOST3:PORT3”. Spaces are not permitted.

    The clone_valid_donor_list variable adds a layer of security by providing control over the sources of cloned data. The privilege required to configure clone_valid_donor_list is different from the privilege required to execute remote cloning operations, which permits assigning those responsibilities to different roles. Configuring clone_valid_donor_list requires the SYSTEM_VARIABLES_ADMIN privilege, whereas executing a remote cloning operation requires the CLONE_ADMIN privilege.

    Internet Protocol version 6 (IPv6) address format is not supported. Internet Protocol version 6 (IPv6) address format is not supported. An alias to the IPv6 address can be used instead. An IPv4 address can be used as is.

5.6.7.13 Clone Plugin Limitations

The clone plugin is subject to these limitations:

  • DDL, including TRUNCATE TABLE, is not permitted during a cloning operation. This limitation should be considered when selecting data sources. A workaround is to use dedicated donor instances, which can accommodate DDL operations being blocked while data is cloned. Concurrent DML is permitted.

  • An instance cannot be cloned from a different MySQL server version. The donor and recipient must have the same MySQL server version. For example, you cannot clone between MySQL 5.7 and MySQL 8.0. The clone plugin is only supported in MySQL 8.0.17 and higher.

  • Only a single MySQL instance can be cloned at a time. Cloning multiple MySQL instances in a single cloning operation is not supported.

  • The X Protocol port specified by mysqlx_port is not supported for remote cloning operations (when specifying the port number of the donor MySQL server instance in a CLONE INSTANCE statement).

  • The clone plugin does not support cloning of MySQL server configurations. The recipient MySQL server instance retains its configuration, including persisted system variable settings (see Section 5.1.9.3, “Persisted System Variables”.)

  • The clone plugin does not support cloning of binary logs.

  • The clone plugin only clones data stored in InnoDB. Other storage engine data is not cloned. MyISAM and CSV tables stored in any schema including the sys schema are cloned as empty tables.

  • Connecting to the donor MySQL server instance through MySQL Router is not supported.

  • Local cloning operations do not support cloning of general tablespaces that were created with an absolute path. A cloned tablespace file with the same path as the source tablespace file would cause a conflict.

 

Mysql:5.5-5.6:Components & Plugins

标签:ttl   hang   instr   result   desire   deadlock   run   permanent   reject   

原文地址:https://www.cnblogs.com/jinzhenshui/p/12488602.html

(0)
(0)
   
举报
评论 一句话评论(0
登录后才能评论!
© 2014 mamicode.com 版权所有  联系我们:gaon5@hotmail.com
迷上了代码!