0%

canal同步主库MySQL5.6至从库MySQL8.0

发表于 更新于 分类于 Disqus：
本文字数： 11k 阅读时长 ≈ 19 分钟

canal同步主库MySQL5.6至从库MySQL8.0

利用 docker 拉取对应 mysql 版本的镜像。

网易云 - Docker镜像

1
2
$ docker pull hub.c.163.com/library/mysql:5.6
$ docker pull hub.c.163.com/library/mysql:8.0
启动容器
1
2
$ docker run --name mysql-5.6 -e MYSQL_ROOT_PASSWORD=1234 -d hub.c.163.com/library/mysql:5.6
$ docker run --name mysql-8.0 -e MYSQL_ROOT_PASSWORD=1234 -d hub.c.163.com/library/mysql:8.0
查看各容器IP地址
1
$ ip addr
  • 172.17.0.2 mysql-5.6
  • 172.17.0.3 mysql-8.0
查看各容器 mysql 对应版本
1
2
3
# 172.17.0.2
root@7ae2a6435ca1:/# /usr/bin/mysql -V
/usr/bin/mysql  Ver 14.14 Distrib 5.6.36, for Linux (x86_64) using  EditLine wrapper
1
2
3
# 172.17.0.3
root@cc51ae9f3098:/# /usr/bin/mysql -V
/usr/bin/mysql  Ver 8.0.1-dmr for Linux on x86_64 (MySQL Community Server (GPL))
安装 vim 工具，用于编辑配置文件

由于仓库还是官方源，这里改成国内源，此处使用 163 源。

1
2
3
4
5
6
7
8
9
10
11
12
$ cat > /etc/apt/sources.list
deb http://mirrors.163.com/debian/ jessie main non-free contrib
deb http://mirrors.163.com/debian-archive/debian/ jessie-backports main non-free contrib
deb-src http://mirrors.163.com/debian/ jessie main non-free contrib
deb-src http://mirrors.163.com/debian-archive/debian/ jessie-backports main non-free contrib
deb http://mirrors.163.com/debian-security/ jessie/updates main non-free contrib
deb-src http://mirrors.163.com/debian-security/ jessie/updates main non-free contrib
# ctrl + c

# 由于 jessie 源在 163 上太久没有同步，忽略对文件有效期的校验。
# E: Release file for http://mirrors.163.com/debian-archive/debian/dists/jessie-backports/InRelease is expired (invalid since 632d12h 4min 50s). Updates for this repository will not be applied.
$ apt-get -o Acquire::Check-Valid-Until=false update
1
2
$ apt-get update
$ apt-get install -y vim
主库 172.17.0.2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
vim /etc/mysql/mysql.conf.d/mysqld.cnf
# Copyright (c) 2014, 2016, Oracle and/or its affiliates. All rights reserved.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA

#
# The MySQL  Server configuration file.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

[mysqld]
pid-file	= /var/run/mysqld/mysqld.pid
socket		= /var/run/mysqld/mysqld.sock
datadir		= /var/lib/mysql
log-error	= /var/log/mysql/error.log
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

server_id = 1
log-bin = mysql-bin
binlog-do-db = db
log-slave-updates
sync_binlog = 1
#auto_increment_offset = 1
#auto_increment_increment = 1
#expire_logs_days = 7  
#log_bin_trust_function_creators = 1

重启主库 mysqld 服务，进入 mysql 命令行模式。

1
2
3
4
5
6
7
8
9
# 查看主库当前binary log文件名和位置（position）
mysql> show master status \G;
*************************** 1. row ***************************
             File: mysql-bin.000001
         Position: 1152
     Binlog_Do_DB: db
 Binlog_Ignore_DB:
Executed_Gtid_Set:
1 row in set (0.01 sec)
从库 172.17.0.3
1
2
3
4
5
6
7
8
9
10
11
12
13
14
vim /etc/mysql/conf.d/docker.cnf

[mysqld]
skip-host-cache
skip-name-resolve

server_id = 2
log-bin = mysql-bin
log-slave-updates
sync_binlog = 0
innodb_flush_log_at_trx_commit = 0        
replicate-do-db = db         
slave-net-timeout = 60                    
log_bin_trust_function_creators = 1

重启从库 mysqld 服务，进入 mysql 命令行模式。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
# 设置主节点参数
mysql> change master to master_host='172.17.0.2',master_user='root',master_password='1234',master_log_file='mysql-bin.000001',master_log_pos=120;
# 开启主从同步
mysql> start slave;
# 查看主从同步状态
mysql> show slave status \G;
*************************** 1. row ***************************
               Slave_IO_State: Waiting for master to send event
                  Master_Host: 172.17.0.2
                  Master_User: root
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.000001
          Read_Master_Log_Pos: 120
               Relay_Log_File: cc51ae9f3098-relay-bin.000002
                Relay_Log_Pos: 317
        Relay_Master_Log_File: mysql-bin.000001
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: db
          Replicate_Ignore_DB:
           Replicate_Do_Table:
       Replicate_Ignore_Table:
      Replicate_Wild_Do_Table:
  Replicate_Wild_Ignore_Table:
                   Last_Errno: 0
                   Last_Error:
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 120
              Relay_Log_Space: 531
              Until_Condition: None
               Until_Log_File:
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File:
           Master_SSL_CA_Path:
              Master_SSL_Cert:
            Master_SSL_Cipher:
               Master_SSL_Key:
        Seconds_Behind_Master: 0
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error:
               Last_SQL_Errno: 0
               Last_SQL_Error:
  Replicate_Ignore_Server_Ids:
             Master_Server_Id: 1
                  Master_UUID: a514dd91-2587-11eb-a23f-0242ac110002
             Master_Info_File: /var/lib/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
           Master_Retry_Count: 86400
                  Master_Bind:
      Last_IO_Error_Timestamp:
     Last_SQL_Error_Timestamp:
               Master_SSL_Crl:
           Master_SSL_Crlpath:
           Retrieved_Gtid_Set:
            Executed_Gtid_Set:
                Auto_Position: 0
         Replicate_Rewrite_DB:
                 Channel_Name:
           Master_TLS_Version:
1 row in set (0.00 sec)
主从库数据同步测试
主库
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
# 1
mysql> CREATE TABLE `test` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `name` char(255) CHARACTER SET utf8mb4 DEFAULT '',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4;
# 2
mysql> insert into test(`name`) values("codezm"), ("codezmm");
# 3
mysql> update test set name = "codezm1" where id = 1;
# 4
mysql> delete from test where id = 2;
# 5
mysql> select version();
+------------+
| version()  |
+------------+
| 5.6.36-log |
+------------+
1 row in set (0.00 sec)
从库
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
# 1
mysql> show create table test;
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                                          |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| test  | CREATE TABLE `test` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `name` char(255) DEFAULT '',
  PRIMARY KEY (`id`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)
# 2
mysql> select * from test;
+----+---------+
| id | name    |
+----+---------+
|  1 | codezm1 |
|  2 | codezmm |
+----+---------+
2 rows in set (0.00 sec)
# 3
mysql> select * from test;
+----+---------+
| id | name    |
+----+---------+
|  1 | codezm1 |
|  2 | codezmm |
+----+---------+
1 row in set (0.00 sec)
# 4
mysql> select * from test;
+----+---------+
| id | name    |
+----+---------+
|  1 | codezm1 |
+----+---------+
2 rows in set (0.00 sec)
# 5
mysql> select version();
+---------------+
| version()     |
+---------------+
| 8.0.1-dmr-log |
+---------------+
1 row in set (0.00 sec)

canal 中间件

canal-server 配置
1
2
3
4
5
6
7
8
9
10
11
12
13
14
docker pull canal/canal-server:v1.1.4
docker run --name canal-server -d -p 11110-11112:11110-11112 -p 9100:9100 canal/canal-server:v1.1.4

# 无法指定 --ip 参数，待解决。。。
docker run --name canal-server-50.83 \
					 -e canal.instance.master.address=10.10.50.83:3306 \
					 -e canal.admin.manager=host.docker.internal:8089 \
					 -e canal.admin.register.auto=true \
					 -d --network lnmp_backend canal/canal-server:v1.1.4
					 
					 

# 配置主库信息
vi ./canal-server/conf/example/instance.properties
canal-admin 配置
1
2
3
4
5
6
7
docker pull canal/canal-admin:v1.1.4
docker run --name canal-admin -p 8089:8089 -d canal/canal-admin:v1.1.4


docker run --name canal-admin -p 8089:8089 -d --network lnmp_backend canal/canal-admin:v1.1.4

vi canal-admin/conf/application.yml
1
2
3
4
5
6
7
8
9
10
11
12
create database canal;

CREATE TABLE `user` (
  `id` bigint(20) unsigned NOT NULL AUTO_INCREMENT,
  `name` char(255) DEFAULT '',
  PRIMARY KEY (`id`)
) ENGINE=MYISAM DEFAULT CHARSET=utf8mb4;

use user;
insert into user(`name`) values("codezm"), ("codejm");
update user set name = "codezm1" where id = 1;
delete from user where id = 2;
主库
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
docker run --name mysql-5.6 -p 3307:3306 -d -e MYSQL_ROOT_PASSWORD=1234  hub.c.163.com/library/mysql:5.6

vim /etc/mysql/mysql.conf.d/mysqld.cnf
# Copyright (c) 2014, 2016, Oracle and/or its affiliates. All rights reserved.
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation; version 2 of the License.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA  02110-1301 USA

#
# The MySQL  Server configuration file.
#
# For explanations see
# http://dev.mysql.com/doc/mysql/en/server-system-variables.html

[mysqld]
pid-file        = /var/run/mysqld/mysqld.pid
socket          = /var/run/mysqld/mysqld.sock
datadir         = /var/lib/mysql
log-error       = /var/log/mysql/error.log
# Disabling symbolic-links is recommended to prevent assorted security risks
symbolic-links=0

server_id = 1
log-bin = mysql-bin
#binlog-format=ROW
binlog-format=mixed
从库
1
docker run --name mysql-8.0 -p 3308:3306 -e MYSQL_ROOT_PASSWORD=1234 -d hub.c.163.com/library/mysql:8.0
canal
1
2
3
4
5
6
2020-11-17 17:55:34.093 [destination = example , address = /192.168.65.2:3307 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - ---> begin to find start position, it will be long time for reset or first position
2020-11-17 17:55:34.093 [destination = example , address = /192.168.65.2:3307 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - prepare to find start position by switch ::1605598704000
2020-11-17 17:55:34.109 [destination = example , address = /192.168.65.2:3307 , EventParser] WARN  c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - Didn't find the corresponding binlog files from mysql-bin.000001 to mysql-bin.000001
2020-11-17 17:55:34.109 [destination = example , address = /192.168.65.2:3307 , EventParser] ERROR c.a.o.c.p.inbound.mysql.rds.RdsBinlogEventParserProxy - dump address /192.168.65.2:3307 has an error, retrying. caused by
com.alibaba.otter.canal.parse.exception.PositionNotFoundException: can't find start position for example
2020-11-17 17:55:34.110 [destination = example , address = /192.168.65.2:3307 , EventParser] ERROR com.alibaba.otter.canal.common.alarm.LogAlarmHandler - destination:example[com.alibaba.otter.canal.parse.exception.PositionNotFoundException: can't find start position for exampl

原因：meta.dat 中保存的位点信息和数据库的位点信息不一致；导致canal抓取不到数据库的动作；

解决方案：删除 meta.dat 文件，再重启 canal-server。
1
rm -f ./canal-server/conf/example/meta.dat

测试

10.10.50.83 news

配置 mysql

vi /data/mysql/my.cnf

1
2
3
server_id = 1
log-bin = mysql-bin
binlog-format=mixed

重启 mysql

1
2
3
4
# 关闭 mysql
/data/mysql/bin/mysqladmin -u root -p shutdown
# 启动 mysql
/bin/sh /data/mysql/bin/mysqld_safe --defaults-file=/data/mysql/my.cnf &

由于我先开启了 binlog，但生成的 binlog 没法使用。

历史数据没有 binlog。

重置binlog

1
2
3
4
5
show master status \G;
# 重置binlog，重置前需停止从库同步：slave stop
reset master;
# 确认重置
show master status \G;

测试数据库SQL

1
/data/mysql/bin/mysqldump -uroot -p news > /tmp/news.sql;

开启 canal dump 10.10.50.83 instance。

1
2
3
create database news;
use news;
source /tmp/news.sql;

启动客户端订阅并消费。

历史产生的 binlog 不消费

1
show binary logs;

image-20201119145217478

1
2

show binlog events in 'mysql-bin.000001' limit 2 \G;

image-20201119145145118

vi canal-server/conf/10.10.50.83/meta.dat

image-20201119145418590