OceanBase 4.x 版本支持2种部署模式：单机部署与分布式部署，同时支持从单机平滑扩展至分布式架构。这样，可以有效解决小型业务向大型业务转型时面临的扩展难题，降低了机器资源的成本。
以下将详述如何通过命令行，实现集群扩展的操作步骤。
| 注：以下操作基于OceanBase 社区版 4.2版本 另：如果集群使用OCP平台做了接管，可以直接在OCP上进行扩展操作，在集群中添加zone即可。 |
|---|
查看集群当前情况
查看当前zone情况
obclient [oceanbase]> select * from dba_ob_zones;
+-------+----------------------------+----------------------------+--------+-----+----------------+-----------+
| ZONE | CREATE_TIME | MODIFY_TIME | STATUS | IDC | REGION | TYPE |
+-------+----------------------------+----------------------------+--------+-----+----------------+-----------+
| zone1 | 2023-08-22 17:37:30.158883 | 2023-08-22 17:37:30.159941 | ACTIVE | | sys_region | ReadWrite |
+-------+----------------------------+----------------------------+--------+-----+----------------+-----------+
1 rows in set (0.021 sec)查看当前server情况
obclient [oceanbase]> select * from dba_ob_servers;
+---------------+----------+----+-------+----------+-----------------+--------+----------------------------+-----------+-----------------------+----------------------------+----------------------------+-------------------------------------------------------------------------------------------+-------------------+
| SVR_IP | SVR_PORT | ID | ZONE | SQL_PORT | WITH_ROOTSERVER | STATUS | START_SERVICE_TIME | STOP_TIME | BLOCK_MIGRATE_IN_TIME | CREATE_TIME | MODIFY_TIME | BUILD_VERSION | LAST_OFFLINE_TIME |
+---------------+----------+----+-------+----------+-----------------+--------+----------------------------+-----------+-----------------------+----------------------------+----------------------------+-------------------------------------------------------------------------------------------+-------------------+
| 172.24.255.17 | 2882 | 1 | zone1 | 2881 | YES | ACTIVE | 2023-08-22 17:37:40.814023 | NULL | NULL | 2023-08-22 17:37:30.449287 | 2023-08-22 17:37:42.220860 | 4.2.0.0_100010022023081817-0bdf1c0c5674e88c5ae9a8d0ae4f8077465d7fae(Aug 18 2023 17:32:49) | NULL |
+---------------+----------+----+-------+----------+-----------------+--------+----------------------------+-----------+-----------------------+----------------------------+----------------------------+-------------------------------------------------------------------------------------------+-------------------+
1 rows in set (0.001 sec)查看当前租户副本情况
obclient [oceanbase]> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS;
+-----------+-------------+-------------+--------------+---------------+
| TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE | LOCALITY |
+-----------+-------------+-------------+--------------+---------------+
| 1 | sys | SYS | RANDOM | FULL{1}@zone1 |
| 1001 | META$1002 | META | RANDOM | FULL{1}@zone1 |
| 1002 | ocp | USER | RANDOM | FULL{1}@zone1 |
+-----------+-------------+-------------+--------------+---------------+
3 rows in set (0.010 sec)新节点安装启动进程
服务器环境配置，保证和已安装节点环境配置相同
具体环境配置参考官方文档：部署前准备
检查安装包，建议使用admin用户，进程启动也是用admin用户
[admin@ob2 rpms]$ ll
-rw-r--r-- 1 admin admin 86234124 Aug 22 17:45 oceanbase-ce-4.2.0.0-100010022023081817.el7.x86_64.rpm
-rw-r--r-- 1 admin admin 158316 Aug 22 17:45 oceanbase-ce-libs-4.2.0.0-100010022023081817.el7.x86_64.rpm安装软件，先安装lib库
[root@ob2 rpms]# ll
-rw-r--r-- 1 admin admin 86234124 8月 22 17:45 oceanbase-ce-4.2.0.0-100010022023081817.el7.x86_64.rpm
-rw-r--r-- 1 admin admin 158316 8月 22 17:45 oceanbase-ce-libs-4.2.0.0-100010022023081817.el7.x86_64.rpm
[root@ob2 rpms]# rpm -ivh oceanbase-ce-libs-4.2.0.0-100010022023081817.el7.x86_64.rpm
准备中... ################################# [100%]
正在升级/安装...1:oceanbase-ce-libs-4.2.0.0-1000100################################# [100%]
[root@ob2 rpms]# rpm -ivh oceanbase-ce-4.2.0.0-100010022023081817.el7.x86_64.rpm
准备中... ################################# [100%]
正在升级/安装...1:oceanbase-ce-4.2.0.0-100010022023################################# [100%]创建目录，保持和已安装节点目录一致
[admin@OMS rpms]# sudo mkdir -p /obdata/{sstable,slog}
[admin@OMS rpms]# sudo mkdir -p /obredo/clog
[admin@OMS rpms]# sudo mkdir -p /home/admin/myoceanbase/oceanbase/{audit,etc2,etc3,log,run}
[admin@OMS rpms]# sudo chown -R admin:admin /obdata /obredo初始化目录，创建目录软连接
[admin@OMS rpms]# ln -s /obredo/clog /obdata/clog
[admin@OMS rpms]# ln -s /obdata /home/admin/myoceanbase/oceanbase/store启动节点，启动前配置下环境变量
export LD_LIBRARY_PATH=/home/admin/oceanbase/lib可以查看已启动节点进程启动命令，参考该命令，启动新的节点
[admin@ob2 ~]$ /home/admin/oceanbase/bin/observer -I 172.24.255.18 -r '172.24.255.18:2882:2881' -p 2881 -P 2882 -z zone2 -n myoceanbase -c 1 -d /obdata -o __min_full_resource_pool_memory=2147483648,enable_syslog_recycle=True,enable_syslog_wf=False,max_syslog_file_count=4,memory_limit=23G,datafile_size=209G,system_memory=5G,log_disk_size=60G,cpu_count=16
/home/admin/oceanbase/bin/observer -I 172.24.255.18 -r 172.24.255.18:2882:2881 -p 2881 -P 2882 -z zone3 -n myoceanbase -c 1 -d /obdata -o __min_full_resource_pool_memory=2147483648,enable_syslog_recycle=True,enable_syslog_wf=False,max_syslog_file_count=4,memory_limit=23G,datafile_size=209G,system_memory=5G,log_disk_size=60G,cpu_count=16
local_ip: 172.24.255.18
rs list: 172.24.255.18:2882:2881
mysql port: 2881
rpc port: 2882
zone: zone3
appname: myoceanbase
cluster id: 1
data_dir: /obdata
optstr: __min_full_resource_pool_memory=2147483648,enable_syslog_recycle=True,enable_syslog_wf=False,max_syslog_file_count=4,memory_limit=23G,datafile_size=209G,system_memory=5G,log_disk_size=60G,cpu_count=16参数解释：
| 参数 | 说明 |
| -I | -i | -I：指定待启动的节点 IP。在多机部署场景下，不能指定 127.0.0.1 作为目标 IP。 -i：指定网卡名，可通过 ifconfig 命令查看。 |
| -p | 指定服务端口号，一般指定为 2881。 |
| -P | 指定 RPC 端口号，一般指定为 2882。 |
| -n | 指定集群名称。可自定义，不同集群名称不要重复即可。 |
| -z | 指定启动的 observer 进程所属的 Zone。 |
| -d | 指定集群主目录，初始化目录时创建的目录。除集群名字 $cluster_name 外，其他不要变动。 |
| -c | 指定集群 ID。为一组数字，可以自定义，不同集群不要重复即可。 |
| -l | 指定日志级别。 |
| -r | 指定 RS 列表，格式是 $ip:2882:2881，分号分割，表示 Root Service 信息。 |
| -o | 指定集群启动参数，需要根据实际情况设置。 system_memory：指定 OceanBase 内部保留内存，默认是 30G ，机器内存比较少的情况下把这个调小，影响就是可能在性能测试时有内存不足问题。 datafile_size：指定 OceanBase 数据文件 sstable 的大小（一次性初始化），根据 /data/1/ 可用空间评估，建议不少于 100G，同时又保留一些剩余空间。 |
添加zone
回到已安装节点并登陆
执行添加zone
obclient [oceanbase]> alter system add zone zone2;
Query OK, 0 rows affected (0.003 sec)执行激活zone
obclient [oceanbase]> alter system start zone zone2;
Query OK, 0 rows affected (0.003 sec)obclient [oceanbase]> select * from dba_ob_zones;
+-------+----------------------------+----------------------------+--------+-----+----------------+-----------+
| ZONE | CREATE_TIME | MODIFY_TIME | STATUS | IDC | REGION | TYPE |
+-------+----------------------------+----------------------------+--------+-----+----------------+-----------+
| zone1 | 2023-08-22 17:37:30.158883 | 2023-08-22 17:37:30.159941 | ACTIVE | | sys_region | ReadWrite |
| zone2 | 2023-08-22 17:50:09.193881 | 2023-08-22 17:50:20.712790 | ACTIVE | | default_region | ReadWrite |
+-------+----------------------------+----------------------------+--------+-----+----------------+-----------+
2 rows in set (0.001 sec)添加server
向zone中添加server
obclient [oceanbase]> ALTER SYSTEM ADD SERVER '172.24.255.18:2882' ZONE 'zone2';
Query OK, 0 rows affected (0.015 sec)obclient [oceanbase]> select * from dba_ob_servers;
+---------------+----------+----+-------+----------+-----------------+--------+----------------------------+-----------+-----------------------+----------------------------+----------------------------+-------------------------------------------------------------------------------------------+-------------------+
| SVR_IP | SVR_PORT | ID | ZONE | SQL_PORT | WITH_ROOTSERVER | STATUS | START_SERVICE_TIME | STOP_TIME | BLOCK_MIGRATE_IN_TIME | CREATE_TIME | MODIFY_TIME | BUILD_VERSION | LAST_OFFLINE_TIME |
+---------------+----------+----+-------+----------+-----------------+--------+----------------------------+-----------+-----------------------+----------------------------+----------------------------+-------------------------------------------------------------------------------------------+-------------------+
| 172.24.255.17 | 2882 | 1 | zone1 | 2881 | YES | ACTIVE | 2023-08-22 17:37:40.814023 | NULL | NULL | 2023-08-22 17:37:30.449287 | 2023-08-22 17:37:42.220860 | 4.2.0.0_100010022023081817-0bdf1c0c5674e88c5ae9a8d0ae4f8077465d7fae(Aug 18 2023 17:32:49) | NULL |
| 172.24.255.18 | 2882 | 2 | zone2 | 2881 | NO | ACTIVE | 2023-08-22 19:00:25.594024 | NULL | NULL | 2023-08-22 19:00:16.206194 | 2023-08-22 19:00:27.409670 | 4.2.0.0_100010022023081817-0bdf1c0c5674e88c5ae9a8d0ae4f8077465d7fae(Aug 18 2023 17:32:49) | NULL |
+---------------+----------+----+-------+----------+-----------------+--------+----------------------------+-----------+-----------------------+----------------------------+----------------------------+-------------------------------------------------------------------------------------------+-------------------+
2 rows in set (0.000 sec)添加租户副本
此时，租户的副本还只有一个，需要再手动添加副本。这里以ocp租户为例，将ocp租户进行扩容
obclient [oceanbase]> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS where tenant_name="ocp";
+-----------+-------------+-------------+--------------+---------------+
| TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE | LOCALITY |
+-----------+-------------+-------------+--------------+---------------+
| 1002 | ocp | USER | RANDOM | FULL{1}@zone1 |
+-----------+-------------+-------------+--------------+---------------+
1 row in set (0.017 sec)添加副本前，需要给新增的节点上添加resource_pool，添加resource pool有两种方式，一种是新建resource pool，一种是将现有租户的resource pool扩展到新的节点上
方式一：新增resource pool
添加unit模版（可使用已有unit模版）-> 新增resource_pool -> 为租户添加resource_pool
这里直接使用已有的unit模版，尽量保证每个副本的资源配置相同，查看ocp租户的unit模版名
obclient [oceanbase]> select * from DBA_OB_UNIT_CONFIGS;
+----------------+---------------------------+----------------------------+----------------------------+---------+---------+-------------+---------------+---------------------+---------------------+-------------+
| UNIT_CONFIG_ID | NAME | CREATE_TIME | MODIFY_TIME | MAX_CPU | MIN_CPU | MEMORY_SIZE | LOG_DISK_SIZE | MAX_IOPS | MIN_IOPS | IOPS_WEIGHT |
+----------------+---------------------------+----------------------------+----------------------------+---------+---------+-------------+---------------+---------------------+---------------------+-------------+
| 1 | sys_unit_config | 2023-08-22 17:37:30.017366 | 2023-08-22 17:37:30.017366 | 1 | 1 | 2147483648 | 2147483648 | 9223372036854775807 | 9223372036854775807 | 1 |
| 1003 | ocp_unit | 2023-08-22 17:43:41.451573 | 2023-08-22 17:43:41.451573 | 1 | 1 | 2147483648 | 6442450944 | 9223372036854775807 | 9223372036854775807 | 1 |
+----------------+---------------------------+----------------------------+----------------------------+---------+---------+-------------+---------------+---------------------+---------------------+-------------+
2 rows in set (0.001 sec)或者使用如下方式查看租户的unit
SELECT NAME FROM DBA_OB_UNIT_CONFIGS WHERE UNIT_CONFIG_ID
IN
(SELECT UNIT_CONFIG_ID FROM DBA_OB_UNITS WHERE TENANT_ID
IN
(SELECT TENANT_ID FROM DBA_OB_TENANTS WHERE TENANT_NAME='ocp'));可以看到ocp租户的unit模版为ocp_unit，因此再新建一个resource pool，使用ocp_unit 模版
查看当前ocp租户的resource_pool
obclient [oceanbase]> select * from DBA_OB_RESOURCE_POOLS;
+------------------+----------+-----------+----------------------------+----------------------------+------------+----------------+-----------+--------------+
| RESOURCE_POOL_ID | NAME | TENANT_ID | CREATE_TIME | MODIFY_TIME | UNIT_COUNT | UNIT_CONFIG_ID | ZONE_LIST | REPLICA_TYPE |
+------------------+----------+-----------+----------------------------+----------------------------+------------+----------------+-----------+--------------+
| 1 | sys_pool | 1 | 2023-08-22 17:37:30.020615 | 2023-08-22 17:37:30.027149 | 1 | 1 | zone1 | FULL |
| 1001 | ocp_pool | 1002 | 2023-08-22 17:37:43.620077 | 2023-08-22 17:43:41.475641 | 1 | 1003 | zone1 | FULL |
+------------------+----------+-----------+----------------------------+----------------------------+------------+----------------+-----------+--------------+
2 rows in set (0.003 sec)添加resource_pool
root@oceanbase03:43:05>create resource pool ocp_pool2 unit="ocp_unit", unit_num=1,zone_list=('zone2');
Query OK, 0 rows affected (0.02 sec)obclient [oceanbase]> select * from DBA_OB_RESOURCE_POOLS;
+------------------+-----------+-----------+----------------------------+----------------------------+------------+----------------+-----------+--------------+
| RESOURCE_POOL_ID | NAME | TENANT_ID | CREATE_TIME | MODIFY_TIME | UNIT_COUNT | UNIT_CONFIG_ID | ZONE_LIST | REPLICA_TYPE |
+------------------+-----------+-----------+----------------------------+----------------------------+------------+----------------+-----------+--------------+
| 1 | sys_pool | 1 | 2023-08-22 17:37:30.020615 | 2023-08-22 17:37:30.027149 | 1 | 1 | zone1 | FULL |
| 1001 | ocp_pool | 1002 | 2023-08-22 17:37:43.620077 | 2023-08-22 17:43:41.475641 | 1 | 1003 | zone1 | FULL |
| 1002 | ocp_pool2 | NULL | 2023-08-22 18:50:46.640215 | 2023-08-22 18:50:46.640215 | 1 | 1004 | zone2 | FULL |
+------------------+-----------+-----------+----------------------------+----------------------------+------------+----------------+-----------+--------------+
2 rows in set (0.003 sec)resource_pool的tenant_id为NULL，为ocp租户添加新建的resource_pool
root@oceanbase03:55:12> alter tenant ocp resource_pool_list=('ocp_pool','ocp_pool2');
Query OK, 0 rows affected (0.79 sec)obclient [oceanbase]> select * from DBA_OB_RESOURCE_POOLS;
+------------------+-----------+-----------+----------------------------+----------------------------+------------+----------------+-----------+--------------+
| RESOURCE_POOL_ID | NAME | TENANT_ID | CREATE_TIME | MODIFY_TIME | UNIT_COUNT | UNIT_CONFIG_ID | ZONE_LIST | REPLICA_TYPE |
+------------------+-----------+-----------+----------------------------+----------------------------+------------+----------------+-----------+--------------+
| 1 | sys_pool | 1 | 2023-08-22 17:37:30.020615 | 2023-08-22 17:37:30.027149 | 1 | 1 | zone1 | FULL |
| 1001 | ocp_pool | 1002 | 2023-08-22 17:37:43.620077 | 2023-08-22 17:43:41.475641 | 1 | 1003 | zone1 | FULL |
| 1002 | ocp_pool2 | 1002 | 2023-08-22 18:50:46.640215 | 2023-08-22 18:50:46.640215 | 1 | 1004 | zone2 | FULL |
+------------------+-----------+-----------+----------------------------+----------------------------+------------+----------------+-----------+--------------+
2 rows in set (0.003 sec)
方式二：扩展已有resource pool
或者直接修改现有租户的resource pool
root@oceanbase03:55:12> ALTER RESOURCE POOL ocp_pool ZONE_LIST=('zone1','zone2');修改租户的locality
obclient [oceanbase]> ALTER TENANT ocp locality="F@zone1, F@zone2";
Query OK, 0 rows affected (0.139 sec)
注：这里有个前提条件，
- 添加副本时，primary zone所在的region内至少要有2个全功能型副本，否则就会添加失败，报错“primary zone F type replica not enough in its region not allowed”；
- primary region不能多于1个，否则会报“ERROR 1235 (0A000): tenant primary zone span regions not supported”
这些限制目的主要是防止多数派能跨region，造成延迟过大
obclient [oceanbase]> ALTER TENANT ocp locality="F@zone1, F@zone2";
ERROR 4179 (HY000): primary zone F type replica not enough in its region not allowedobclient [oceanbase]> ALTER TENANT ocp locality="F@zone1, F@zone2, F@zone3";
ERROR 1235 (0A000): tenant primary zone span regions not supported查看副本添加任务情况
obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE = 'ALTER_TENANT_LOCALITY';
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+----------------------------------------------+---------------+---------------+-------------+
| JOB_ID | JOB_TYPE | JOB_STATUS | RESULT_CODE | PROGRESS | START_TIME | MODIFY_TIME | TENANT_ID | SQL_TEXT | EXTRA_INFO | RS_SVR_IP | RS_SVR_PORT |
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+----------------------------------------------+---------------+---------------+-------------+
| 1 | ALTER_TENANT_LOCALITY | SUCCESS | 0 | 100 | 2023-08-23 12:00:33.923511 | 2023-08-23 12:10:58.213438 | 1002 | ALTER TENANT ocp locality="F@zone1, F@zone2" | FULL{1}@zone1 | 172.24.255.17 | 2882 |
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+----------------------------------------------+---------------+---------------+-------------+
1 rows in set (0.017 sec)当 JOB_STATUS 为 SUCCESS，则表示添加成功
再次查看租户的副本情况
obclient [oceanbase]> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS where tenant_name="ocp";
+-----------+-------------+-------------+--------------+------------------------------+
| TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE | LOCALITY |
+-----------+-------------+-------------+--------------+------------------------------+
| 1002 | ocp | USER | zone1;zone2 | FULL{1}@zone1, FULL{1}@zone2 |
+-----------+-------------+-------------+--------------+------------------------------+
1 row in set (0.004 sec)再添加新的节点，使用同样方式即可。
添加zone3及租户副本
obclient [oceanbase]> ALTER TENANT ocp locality="F@zone1, F@zone2, F@zone3";
Query OK, 0 rows affected (0.121 sec)添加成功
obclient [oceanbase]> SELECT * FROM oceanbase.DBA_OB_TENANT_JOBS WHERE JOB_TYPE = 'ALTER_TENANT_LOCALITY';
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+-------------------------------------------------------+------------------------------+---------------+-------------+
| JOB_ID | JOB_TYPE | JOB_STATUS | RESULT_CODE | PROGRESS | START_TIME | MODIFY_TIME | TENANT_ID | SQL_TEXT | EXTRA_INFO | RS_SVR_IP | RS_SVR_PORT |
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+-------------------------------------------------------+------------------------------+---------------+-------------+
| 1 | ALTER_TENANT_LOCALITY | SUCCESS | 0 | 100 | 2023-08-23 12:00:33.923511 | 2023-08-23 12:10:58.213438 | 1002 | ALTER TENANT ocp locality="F@zone1, F@zone2" | FULL{1}@zone1 | 172.24.255.17 | 2882 |
| 2 | ALTER_TENANT_LOCALITY | SUCCESS | 0 | 100 | 2023-08-23 13:47:51.957014 | 2023-08-23 14:00:14.633674 | 1002 | ALTER TENANT ocp locality="F@zone1, F@zone2, F@zone3" | FULL{1}@zone1, FULL{1}@zone2 | 172.24.255.17 | 2882 |
+--------+-----------------------+------------+-------------+----------+----------------------------+----------------------------+-----------+-------------------------------------------------------+------------------------------+---------------+-------------+
2 rows in set (0.003 sec)查看租户副本情况
obclient [oceanbase]> SELECT TENANT_ID,TENANT_NAME,TENANT_TYPE,PRIMARY_ZONE,LOCALITY FROM oceanbase.DBA_OB_TENANTS where tenant_name="ocp";
+-----------+-------------+-------------+--------------+---------------------------------------------+
| TENANT_ID | TENANT_NAME | TENANT_TYPE | PRIMARY_ZONE | LOCALITY |
+-----------+-------------+-------------+--------------+---------------------------------------------+
| 1002 | ocp | USER | zone1;zone2 | FULL{1}@zone1, FULL{1}@zone2, FULL{1}@zone3 |
+-----------+-------------+-------------+--------------+---------------------------------------------+
1 row in set (0.006 sec)高可用测试
创建一张表并插入一条数据
obclient [test]> create table t1(id int);
Query OK, 0 rows affected (0.194 sec)obclient [test]> insert into t1 values(111);
Query OK, 1 row affected (0.008 sec)obclient [test]> select * from t1;
+------+
| id |
+------+
| 111 |
+------+
1 row in set (0.002 sec)查看表的leader所在节点
表leader在 172.24.255.17 上，通过shell写个脚本，循环每隔一秒查询下t1表
#!/bin/bashfor ((i=1;i<1000; i++));
do
time=$(date "+%Y-%m-%d %H:%M:%S")
obclient -h172.24.255.17 -P2883 -u'root@obtest#myoceanbase' -p'xxxx' -Dtest -A -e "select * from t1;" > /dev/null;
if [ $? -eq 0 ]; thenecho $time: "SUCCESS"
elseecho $time: "FAILED"
fi
sleep 1;
done然后把 172.24.255.17 机器上的observer进程kill掉，看下是否能自动实现切换
从下面的执行结果来看，SQL在卡了5s左右之后，继续输出了结果，并且因为Paxos协议，这个切换过程中，可以保证数据没有丢失。
再次查看表LEADER情况，可以看到进程杀死之后，LEADER切换到了172.24.255.18上
