mysqldump?搭建復(fù)制報(bào)錯(cuò)原因解析
故障現(xiàn)象
某客戶反饋,使用 mysqldump 搭建從庫(kù),啟動(dòng)復(fù)制后,復(fù)制報(bào)錯(cuò):
Could not execute Write_rows event on table xxx; Duplicate entry 'xxx' for key 'PRIMARY'。
客戶使用的命令(看起來(lái)沒(méi)啥問(wèn)題)。
-- 主庫(kù)備份 shell> mysqldump -uroot -pxxx --master-data=2 --single-transaction -A --routines --events --triggers >/tmp/xxx.sql -- 從服務(wù)器還原備份并啟動(dòng)復(fù)制 mysql>reset master; mysql>reset slave all; mysql>source /tmp/xxx.sql ; mysql>change master to master_host='xxx',master_port=3306,master_user='xxx',master_password='xxx',master_auto_position=1; mysql>start slave;
問(wèn)題排查
查看復(fù)制報(bào)錯(cuò)表的表結(jié)構(gòu),發(fā)現(xiàn)表的存儲(chǔ)引擎為 MyISAM 引擎。
根據(jù)客戶反饋,表訪問(wèn)比較頻繁,mysqldump --single-transaction 選項(xiàng),只能保證 InnoDB 引擎表備份的一致性,無(wú)法保證 MyISAM 引擎表備份的一致性,問(wèn)題可能就出在這。
問(wèn)題解決
修改表的存儲(chǔ)引擎為 InnoDB 后,重新備份恢復(fù),可以正常搭建從庫(kù)。
問(wèn)題復(fù)現(xiàn)
下面我們來(lái)復(fù)現(xiàn)一下該問(wèn)題。
環(huán)境信息
| 操作系統(tǒng) | CentOS Linux release 7.5.1804 (Core) |
| 版本 | MySQL 5.7.25 |
| 主庫(kù) | 10.186.60.187 |
| 從庫(kù) | 10.186.60.37 |
| 主從 | 開(kāi)啟 GTID |
操作步驟
在主庫(kù),使用 Sysbench 造一張 1000w 數(shù)據(jù)的 InnoDB 引擎的表 testdb_innodb.sbtest1(造 1000w 數(shù)據(jù)主要目的是讓備份 InnoDB 引擎表的時(shí)間拉長(zhǎng))。
shell> sysbench /usr/share/sysbench/tests/include/oltp_legacy/oltp.lua \ --mysql-host=10.186.60.187 --mysql-port=3307 --mysql-user=root \ --mysql-password=1 --mysql-db=testdb_innodb --oltp-table-size=10000000 --oltp-tables-count=1 --threads=4 --report-interval=3 prepare -- 表結(jié)構(gòu)如下 mysql> show create table testdb_innodb.sbtest1; CREATE TABLE `sbtest1` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `k` int(10) unsigned NOT NULL DEFAULT '0', `c` char(120) COLLATE utf8mb4_bin NOT NULL DEFAULT '', `pad` char(60) COLLATE utf8mb4_bin NOT NULL DEFAULT '', PRIMARY KEY (`id`), KEY `k_1` (`k`) ) ENGINE=InnoDB AUTO_INCREMENT=10000001 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin -- 表總行數(shù)如下: mysql> select count(*) from testdb_innodb.sbtest1; +----------+ | count(*) | +----------+ | 10000000 | +----------+
在主庫(kù),造一張 MyISAM 引擎的表 testdb_myisam.sbtest2。
-- 表結(jié)構(gòu)如下: mysql> CREATE TABLE testdb_myisam.`sbtest2` ( `id` int(10) unsigned NOT NULL AUTO_INCREMENT, `k` int(10) unsigned NOT NULL DEFAULT '0', `c` char(120) COLLATE utf8mb4_bin NOT NULL DEFAULT '', `pad` char(60) COLLATE utf8mb4_bin NOT NULL DEFAULT '', PRIMARY KEY (`id`), KEY `k_1` (`k`) ) ENGINE=myisam AUTO_INCREMENT=1 DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin
在主庫(kù),開(kāi)始 mysqldump 邏輯備份,并在執(zhí)行備份 testdb_innodb.sbtest1 期間(備份的順序:先備份 testdb_innodb 庫(kù)),往 testdb_myisam.sbtest2 表插入一條數(shù)據(jù)。
-- 執(zhí)行 mysqldump 備份 shell> /data/mysql/base/5.7.25/bin/mysqldump -h10.186.60.187 -P3307 -uroot -p1 --master-data=2 --single-transaction -A --routines --events --triggers >/tmp/dump.sql -- 執(zhí)行備份 testdb_innodb.sbtest1 期間,往 testdb_myisam.sbtest2 表插入一條數(shù)據(jù) mysql> insert into testdb_myisam.`sbtest2`(k,c,pad) values(2,'myisam','myisam'); -- 通過(guò) MySQL general_log 觀察備份情況 2023-07-11T16:15:50.900581+08:00 2692 Connect root@10.186.60.187 on using TCP/IP 2023-07-11T16:15:50.901124+08:00 2692 Query /*!40100 SET @@SQL_MODE='' */ 2023-07-11T16:15:50.901529+08:00 2692 Query /*!40103 SET TIME_ZONE='+00:00' */ 2023-07-11T16:15:50.901743+08:00 2692 Query FLUSH /*!40101 LOCAL */ TABLES 2023-07-11T16:15:50.938083+08:00 2692 Query FLUSH TABLES WITH READ LOCK 2023-07-11T16:15:50.938281+08:00 2692 Query SET SESSION TRANSACTION ISOLATION LEVEL REPEATABLE READ 2023-07-11T16:15:50.938410+08:00 2692 Query START TRANSACTION /*!40100 WITH CONSISTENT SNAPSHOT */ 2023-07-11T16:15:50.938678+08:00 2692 Query SHOW VARIABLES LIKE 'gtid\_mode' 2023-07-11T16:15:50.980335+08:00 2692 Query SELECT @@GLOBAL.GTID_EXECUTED 2023-07-11T16:15:50.980566+08:00 2692 Query SHOW MASTER STATUS 2023-07-11T16:15:50.980758+08:00 2692 Query UNLOCK TABLES ...略 2023-07-11T16:15:51.541911+08:00 2692 Init DB testdb_innodb 2023-07-11T16:15:51.542012+08:00 2692 Query SHOW CREATE DATABASE IF NOT EXISTS `testdb_innodb` 2023-07-11T16:15:51.542139+08:00 2692 Query SAVEPOINT sp 2023-07-11T16:15:51.542224+08:00 2692 Query show tables 2023-07-11T16:15:51.542405+08:00 2692 Query show table status like 'sbtest1' 2023-07-11T16:15:51.543353+08:00 2692 Query SET SQL_QUOTE_SHOW_CREATE=1 2023-07-11T16:15:51.543467+08:00 2692 Query SET SESSION character_set_results = 'binary' 2023-07-11T16:15:51.543548+08:00 2692 Query show create table `sbtest1` 2023-07-11T16:15:51.543729+08:00 2692 Query SET SESSION character_set_results = 'utf8' 2023-07-11T16:15:51.543837+08:00 2692 Query show fields from `sbtest1` 2023-07-11T16:15:51.544172+08:00 2692 Query show fields from `sbtest1` 2023-07-11T16:15:51.544477+08:00 2692 Query SELECT /*!40001 SQL_NO_CACHE */ * FROM `sbtest1` 2023-07-11T16:15:57.603435+08:00 2683 Query insert into testdb_myisam.`sbtest2`(k,c,pad) values(2,'myisam','myisam') 2023-07-11T16:16:27.456357+08:00 2692 Query SET SESSION character_set_results = 'binary' 2023-07-11T16:16:27.471239+08:00 2692 Query use `testdb_innodb` 2023-07-11T16:16:27.471589+08:00 2692 Query select @@collation_database 2023-07-11T16:16:27.472065+08:00 2692 Query SHOW TRIGGERS LIKE 'sbtest1' 2023-07-11T16:16:27.506025+08:00 2692 Query SET SESSION character_set_results = 'utf8' 2023-07-11T16:16:27.506225+08:00 2692 Query ROLLBACK TO SAVEPOINT sp 2023-07-11T16:16:27.506383+08:00 2692 Query RELEASE SAVEPOINT sp 2023-07-11T16:16:27.506538+08:00 2692 Query show events 2023-07-11T16:16:27.507226+08:00 2692 Query use `testdb_innodb` 2023-07-11T16:16:27.507346+08:00 2692 Query select @@collation_database 2023-07-11T16:16:27.507457+08:00 2692 Query SET SESSION character_set_results = 'binary' 2023-07-11T16:16:27.507629+08:00 2692 Query SHOW FUNCTION STATUS WHERE Db = 'testdb_innodb' 2023-07-11T16:16:27.621194+08:00 2692 Query SHOW PROCEDURE STATUS WHERE Db = 'testdb_innodb' 2023-07-11T16:16:27.622726+08:00 2692 Query SET SESSION character_set_results = 'utf8' 2023-07-11T16:16:27.622900+08:00 2692 Init DB testdb_myisam 2023-07-11T16:16:27.623005+08:00 2692 Query SHOW CREATE DATABASE IF NOT EXISTS `testdb_myisam` 2023-07-11T16:16:27.623102+08:00 2692 Query SAVEPOINT sp 2023-07-11T16:16:27.623211+08:00 2692 Query show tables 2023-07-11T16:16:27.623566+08:00 2692 Query show table status like 'sbtest2' 2023-07-11T16:16:27.624197+08:00 2692 Query SET SQL_QUOTE_SHOW_CREATE=1 2023-07-11T16:16:27.624314+08:00 2692 Query SET SESSION character_set_results = 'binary' 2023-07-11T16:16:27.624401+08:00 2692 Query show create table `sbtest2` 2023-07-11T16:16:27.624518+08:00 2692 Query SET SESSION character_set_results = 'utf8' 2023-07-11T16:16:27.624605+08:00 2692 Query show fields from `sbtest2` 2023-07-11T16:16:27.625027+08:00 2692 Query show fields from `sbtest2` 2023-07-11T16:16:27.625391+08:00 2692 Query SELECT /*!40001 SQL_NO_CACHE */ * FROM `sbtest2` 2023-07-11T16:16:27.636073+08:00 2692 Query SET SESSION character_set_results = 'binary' 2023-07-11T16:16:27.636213+08:00 2692 Query use `testdb_myisam` 2023-07-11T16:16:27.636317+08:00 2692 Query select @@collation_database 2023-07-11T16:16:27.636429+08:00 2692 Query SHOW TRIGGERS LIKE 'sbtest2' 2023-07-11T16:16:27.636923+08:00 2692 Query SET SESSION character_set_results = 'utf8' 2023-07-11T16:16:27.637034+08:00 2692 Query ROLLBACK TO SAVEPOINT sp 2023-07-11T16:16:27.637116+08:00 2692 Query RELEASE SAVEPOINT sp 2023-07-11T16:16:27.637195+08:00 2692 Query show events 2023-07-11T16:16:27.637517+08:00 2692 Query use `testdb_myisam` 2023-07-11T16:16:27.637631+08:00 2692 Query select @@collation_database 2023-07-11T16:16:27.637741+08:00 2692 Query SET SESSION character_set_results = 'binary' 2023-07-11T16:16:27.637839+08:00 2692 Query SHOW FUNCTION STATUS WHERE Db = 'testdb_myisam' 2023-07-11T16:16:27.639206+08:00 2692 Query SHOW PROCEDURE STATUS WHERE Db = 'testdb_myisam' 2023-07-11T16:16:27.640377+08:00 2692 Query SET SESSION character_set_results = 'utf8' 2023-07-11T16:16:27.663274+08:00 2692 Quit
在從服務(wù)器,使用上述 mysqldump 邏輯備份文件執(zhí)行恢復(fù),搭建從庫(kù)。
-- 從庫(kù)查看數(shù)據(jù)庫(kù)
mysql> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mysql |
| performance_schema |
| sys |
+--------------------+
-- 清空從庫(kù) binlog 和 gtid 信息
mysql> reset master;
-- 查看確認(rèn)
mysql> show master status\G;
*************************** 1. row ***************************
File: mysql-bin.000001
Position: 154
Binlog_Do_DB:
Binlog_Ignore_DB:
Executed_Gtid_Set:
1 row in set (0.00 sec)
-- 執(zhí)行 mysqldump 邏輯備份文件恢復(fù)
mysql> source /tmp/dump.sql;
-- 建立復(fù)制,并啟動(dòng)復(fù)制
mysql> change master to MASTER_HOST='10.186.60.187',MASTER_PORT=3307,master_user='repl',master_password='1',MASTER_AUTO_POSITION=1;
mysql> start slave;
-- 查看復(fù)制狀態(tài)
mysql> show slave status\G;
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 10.186.60.187
Master_User: repl
Master_Port: 3307
Connect_Retry: 60
Master_Log_File: mysql-bin.000015
Read_Master_Log_Pos: 190088135
Relay_Log_File: mysql-relay.000002
Relay_Log_Pos: 414
Relay_Master_Log_File: mysql-bin.000015
Slave_IO_Running: Yes
Slave_SQL_Running: No
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno: 1062
Last_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 1 failed executing transaction '19112042-1f97-11ee-bf09-02000aba3cbb:3747' at master log mysql-bin.000015, end_log_pos 190087781. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
Skip_Counter: 0
Exec_Master_Log_Pos: 190087413
Relay_Log_Space: 1339
Until_Condition: None
Until_Log_File:
Until_Log_Pos: 0
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master: NULL
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno: 0
Last_IO_Error:
Last_SQL_Errno: 1062
Last_SQL_Error: Coordinator stopped because there were error(s) in the worker(s). The most recent failure being: Worker 1 failed executing transaction '19112042-1f97-11ee-bf09-02000aba3cbb:3747' at master log mysql-bin.000015, end_log_pos 190087781. See error log and/or performance_schema.replication_applier_status_by_worker table for more details about this failure or others, if any.
Replicate_Ignore_Server_Ids:
Master_Server_Id: 629181509
Master_UUID: 19112042-1f97-11ee-bf09-02000aba3cbb
Master_Info_File: mysql.slave_master_info
SQL_Delay: 0
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State:
Master_Retry_Count: 86400
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp: 230711 17:03:01
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set: 19112042-1f97-11ee-bf09-02000aba3cbb:3747-3748
Executed_Gtid_Set: 19112042-1f97-11ee-bf09-02000aba3cbb:1-3746
Auto_Position: 1
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
1 row in set (0.00 sec)
-- 查看復(fù)制具體報(bào)錯(cuò)內(nèi)容
mysql> select * from performance_schema.replication_applier_status_by_worker\G;
*************************** 1. row ***************************
CHANNEL_NAME:
WORKER_ID: 1
THREAD_ID: NULL
SERVICE_STATE: OFF
LAST_SEEN_TRANSACTION: 19112042-1f97-11ee-bf09-02000aba3cbb:3747
LAST_ERROR_NUMBER: 1062
LAST_ERROR_MESSAGE: Worker 1 failed executing transaction '19112042-1f97-11ee-bf09-02000aba3cbb:3747' at master log mysql-bin.000015, end_log_pos 190087781; Could not execute Write_rows event on table testdb_myisam.sbtest2; Duplicate entry '2' for key 'PRIMARY', Error_code: 1062; handler error HA_ERR_FOUND_DUPP_KEY; the event's master log FIRST, end_log_pos 190087781
LAST_ERROR_TIMESTAMP: 2023-07-11 17:03:01原理分析
- 當(dāng) mysqldump 開(kāi)始備份,并獲取一致性位點(diǎn)后,
UNLOCK TABLES前,記為 T1 時(shí)刻。 - 備份 InnoDB 表完成(假設(shè)先備份 InnoDB 表),記為 T2 時(shí)刻。
- 備份 MyISAM 引擎表完成,記為 T3 時(shí)刻。
- 在 T1 和 T2 之間,如果 MyISAM 引擎表有 INSERT 操作,會(huì)有 binlog 產(chǎn)生,mysqldump 也會(huì)把 T1 到 T2 之間對(duì) MyISAM 引擎表的 INSERT 數(shù)據(jù)備份下來(lái)。
- 這樣就產(chǎn)生了,啟動(dòng)復(fù)制后,由于 SQL 線程會(huì)回放 T1 到 T2 期間的 binlog,而這部分?jǐn)?shù)據(jù)已經(jīng)在備份文件里了,并恢復(fù)到從庫(kù)了,導(dǎo)致 SQL 線程回放報(bào)重復(fù)鍵的問(wèn)題。
- 使用該選項(xiàng)時(shí):
mysqldump --single-transaction獲取一致性備份只適用于 InnoDB 引擎,對(duì)于 InnoDB 引擎表的備份,獲取的是 T1 時(shí)刻的快照,對(duì)于非 InnoDB 引擎表的備份,獲取的是當(dāng)前最新數(shù)據(jù)。

改進(jìn)建議
- 把業(yè)務(wù)庫(kù)的非 InnoDB 引擎表,修改為 InnoDB,重新備份后搭建從庫(kù)( 修改表的存儲(chǔ)引擎開(kāi)銷較大,需要考慮改存儲(chǔ)引擎對(duì)在線業(yè)務(wù)的影響,適合表可以改為 InnoDB 引擎的情況)。
- 改用 Xtrabacku
p 備份工具。如果非 InnoDB 的表比較大,備份 MyISAM 引擎期間, 備份線程持有實(shí)例的全局讀鎖(FLUSH TABLES WITH READ LOCK)時(shí)間將增加,將影響數(shù)據(jù)庫(kù)可用性,選擇業(yè)務(wù)低峰時(shí)執(zhí)行。(適合短時(shí)間內(nèi)無(wú)法修改表存儲(chǔ)引擎的情況)。
更多技術(shù)文章,請(qǐng)?jiān)L問(wèn):http://m.fzitv.net/
關(guān)于 SQLE
愛(ài)可生開(kāi)源社區(qū)的 SQLE 是一款面向數(shù)據(jù)庫(kù)使用者和管理者,支持多場(chǎng)景審核,支持標(biāo)準(zhǔn)化上線流程,原生支持 MySQL 審核且數(shù)據(jù)庫(kù)類型可擴(kuò)展的 SQL 審核工具。
SQLE 獲取
| 類型 | 地址 |
|---|---|
| 版本庫(kù) | https://github.com/actiontech/sqle |
| 文檔 | https://actiontech.github.io/sqle-docs/ |
| 發(fā)布信息 | https://github.com/actiontech/sqle/releases |
| 數(shù)據(jù)審核插件開(kāi)發(fā)文檔 | https://actiontech.github.io/sqle-docs/docs/dev-manual/plugin... |
以上就是mysqldump 搭建復(fù)制報(bào)錯(cuò)原因解析的詳細(xì)內(nèi)容,更多關(guān)于mysqldump 搭建復(fù)制報(bào)錯(cuò)的資料請(qǐng)關(guān)注腳本之家其它相關(guān)文章!
- mysqldump進(jìn)行數(shù)據(jù)備份詳解
- 導(dǎo)致mysqld無(wú)法啟動(dòng)的一個(gè)錯(cuò)誤問(wèn)題及解決
- MySQL使用mysqldump實(shí)現(xiàn)數(shù)據(jù)完全備份
- mysql初始化命令mysqld?--initialize參數(shù)說(shuō)明小結(jié)
- MySQL啟動(dòng)失敗報(bào)錯(cuò):mysqld.service failed to run ‘start-pre‘ task的問(wèn)題分析與解決方案
- 使用mysqldump導(dǎo)出導(dǎo)入mysql表結(jié)構(gòu)或者數(shù)據(jù)
- mysqldump參數(shù)詳細(xì)說(shuō)明及用途
- docker安裝mysqld-exporter的實(shí)現(xiàn)
相關(guān)文章
MySQL服務(wù)器權(quán)限與對(duì)象權(quán)限詳解
這篇文章主要介紹了MySQL服務(wù)器權(quán)限與對(duì)象權(quán)限,具有很好的參考價(jià)值,希望對(duì)大家有所幫助,如有錯(cuò)誤或未考慮完全的地方,望不吝賜教2024-08-08
Oracle和MySQL中生成32位uuid的方法舉例(國(guó)產(chǎn)達(dá)夢(mèng)同Oracle)
近日遇到朋友問(wèn)及如何生成UUID,UUID是通用唯一識(shí)別碼(Universally Unique Identifier)方法,這里給大家總結(jié)下,這篇文章主要給大家介紹了關(guān)于Oracle和MySQL中生成32位uuid的方法,需要的朋友可以參考下2023-08-08
Mysql事務(wù)隔離級(jí)別原理實(shí)例解析
這篇文章主要介紹了Mysql事務(wù)隔離級(jí)別原理實(shí)例解析,文中通過(guò)示例代碼介紹的非常詳細(xì),對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值,需要的朋友可以參考下2020-03-03
mysql 8.0.20 winx64.zip壓縮版安裝配置方法圖文教程
這篇文章主要為大家詳細(xì)介紹了mysql 8.0.20 winx64.zip壓縮版安裝配置方法圖文教程,文中示例代碼介紹的非常詳細(xì),具有一定的參考價(jià)值,感興趣的小伙伴們可以參考一下2020-05-05

