【测试类型：故障注入】【测试版本：7.0.0】存在业务压力情况下，cm_ctl switchover前端默认超时时长2min后，实际后台仍在进行switchover后续流程情况下，cm会进行重新仲裁，restart原主。

【标题描述】: 存在业务压力情况下，cm_ctl switchover前端默认超时时长2min后，实际后台仍在进行switchover后续流程情况下，cm会进行重新仲裁，restart原主。

【测试类型：故障注入】【测试版本：7.0.0】 存在业务压力情况下，cm_ctl switchover前端默认超时时长2min后，实际后台仍在进行switchover后续流程情况下，cm会进行重新仲裁，restart原主。

【操作系统和硬件信息】(查询命令: cat /etc/system-release, uname -a):
Kylin Linux Advanced Server release V10 (Sword)

【测试环境】(单机/1主x备x级联备):
一主两备CM集群

【被测功能】:
业务压力下cm_ctl switchover功能

【测试类型】:
可靠性测试

【数据库版本】(查询命令: gaussdb -V):
gaussdb (openGauss 7.0.0-RC1 build 30d8cc0f) compiled at 2024-11-27 03:03:47 commit 0 last mr

【预置条件】:
一主两备CM集群功能正常
设置synchronous_commit=remote_receive,设置enable_incremental_checkpoint = off，加大主机flush差距及switchover时checkpoint时长，构造默认switchover超时场景；

【操作步骤】(请填写详细的操作步骤):
1. TPCC30仓100并发执行20min以上
2. 步骤1执行过程中，备机2进行cm_ctl switchover
cm_ctl switchover -n 3 -D /data/relia_app/peilq_cm/cluster/dn1

【预期输出】:
1. tpcc在步骤2执行后，异常终止
2. switchover前端可能超时，后台继续进行switchover，直至成功

【实际输出】:
1. tpcc在步骤2执行后，异常终止
2. switchover前端超时报错后，降备主机进行了KEY_EVENT_RESTART，备1备被仲裁成新主
前端超时：
![输入图片说明](https://foruda.gitee.com/images/1733908209949439991/2e891d26_8568116.png "屏幕截图")
最终集群状态：备1升主
![输入图片说明](https://foruda.gitee.com/images/1733908282869735779/db076a9b_8568116.png "屏幕截图")

【原因分析】:
1. 这个问题的根因
2. 问题推断过程
3. 还有哪些原因可能造成类似现象
4. 该问题是否有临时规避措施
5. 问题解决方案
6. 预计修复问题时间

【日志信息】(请附上日志文件、截图、coredump信息):
 _备2与主机时间差为5s_

```

============switchover后，TPCC执行异常退出
2024-12-09-19:54:22,146 [Thread-195] ERROR  jTPCCTData : FATAL: terminating connection due to administrator command

============switchover pg日志
备机2进行switchover
2024-12-09 19:54:16.078 6756d8c7.1 [unknown] 281473538072592 [unknown] 0 dn_6001_6002_6003 00000  0 [BACKEND] LOG:  to do switchover
2024-12-09 19:54:16.079 6756d8d6.6138 [unknown] 281463032164224 dn_6003 0 dn_6001_6002_6003 00000  0 [BACKEND] LOG:  send fast switchover request to primary

主机接收switchover
2024-12-09 19:54:21.794 6756d8dc.5065 [unknown] 281463168479104 dn_6003 0 dn_6001_6002_6003 00000  0 [BACKEND] LOG:  received fast switchover request from standby

======cm_agent 升主备2日志打印
2024-12-09 19:54:15.670 tid=3584858 ProcessCmsMsg LOG: switchover msg from cm_server, data_dir :/data/relia_app/peilq_cm/cluster/dn1  nodeType is 2
2024-12-09 19:54:16.776 tid=3584834  ERROR: datanode_wal_send_state_string_to_int failed, input string dbstate is: (Wait promoting)
2024-12-09 19:54:17.791 tid=3584834  ERROR: datanode_wal_send_state_string_to_int failed, input string dbstate is: (Wait promoting)
………重复日志打印…………………
2024-12-09 19:56:13.555 tid=3584834  ERROR: datanode_wal_send_state_string_to_int failed, input string dbstate is: (Wait promoting)
2024-12-09 19:56:14.570 tid=3584834  ERROR: datanode_wal_send_state_string_to_int failed, input string dbstate is: (Wait promoting)
2024-12-09 19:56:14.748 tid=3584858 ProcessCmsMsg ERROR: instId(6003) process_lock_no_primary_command(select * from pg_catalog.disable_conn('prohibit_connection', '', 0);) fail return FAIL!
2024-12-09 19:56:14.748 tid=3584858 ProcessCmsMsg ERROR: set lock1 to instance 6003 failed.

======cm_server key_event日志
2024-12-09 19:54:21.374 tid=3318360 CTL_WORKER: [KeyEvent: KEY_EVENT_SWITCHOVER] [Instance: 6003] [Details: send switchover message, node=3, instance=6003]
2024-12-09 19:56:20.294 tid=3318361 CTL_WORKER: [KeyEvent: KEY_EVENT_RESTART] [Instance: 6001] [Details: send restart message, node=1, instanceId=6001]
2024-12-09 19:56:22.867 tid=3318365 AGENT_WORKER: [KeyEvent: KEY_EVENT_FAILOVER] [Instance: 6002] [Details: Failover message has sent to instance 6002, term 706, sendFailoverTimes is 0.]
2024-12-09 19:56:30.298 tid=3318364 AGENT_WORKER: [KeyEvent: KEY_EVENT_NOTIFY_STANDBY] [Instance: 6001] [Details: send notify standby to instance(6001)]

======cm_server 日志，cm_ctl switchover超时时间后，降备节点假死检测判断为假死
2024-12-09 19:54:21.375 tid=3318357 IO_WORKER LOG: cmserver send msg to node 3, msgtype: MSG_CM_AGENT_SWITCHOVER

2024-12-09 19:56:20.294 tid=3318358 IO_WORKER LOG: cmserver send msg to node 1, msgtype: MSG_CM_AGENT_RESTART
2024-12-09 19:56:20.294 tid=3318361 CTL_WORKER LOG: phony dead times(12:-1) already exceeded, will restart(6001)
2024-12-09 19:56:20.294 tid=3318361 CTL_WORKER LOG: [KeyEvent: KEY_EVENT_RESTART] [Instance: 6001] [Details: send restart message, node=1, instanceId=6001]

========降备主节点pg日志打印线程堆栈
2024-12-09 19:55:22.435 6756daaa.5068 postgres 281463566413696 stack_perf_main 0 dn_6001_6002_6003 00000  0 [BACKEND] LOG:  Print all thread stack
        Thread 0 tid<281473577541648> lwtid<3318195>
        comm_poll(pollfd*, unsigned long, int) + 0xa4
        0xaaaaac5c273c
        PostmasterMain(int, char**) + 0x28dc
        main + 0x3e4
        __libc_start_main + 0xe0
        0xaaaaabb71408

Thread 1 tid<281464625737600> lwtid<3318332>
        WaitLatchOrSocket(Latch volatile*, int, int, long) + 0x1ec
        SysLoggerMain(int) + 0x720
        int GaussDbThreadMain<(knl_thread_role)17>(knl_thread_arg*) + 0x308
        0xaaaaac5aed10
        0xffffacbc57ac
        0xffffacb0d0fc

Thread 2 tid<281464608956288> lwtid<3318333>
        WaitLatchOrSocket(Latch volatile*, int, int, long) + 0x1ec
        PgAuditorMain() + 0x444
        int GaussDbThreadMain<(knl_thread_role)15>(knl_thread_arg*) + 0x324
        0xaaaaac5aed10
        0xffffacbc57ac
        0xffffacb0d0fc

Thread 3 tid<281464578809728> lwtid<3318335>
        WaitLatchOrSocket(Latch volatile*, int, int, long) + 0x1ec
        AlarmCheckerMain() + 0x2ac
        int GaussDbThreadMain<(knl_thread_role)20>(knl_thread_arg*) + 0x304
        0xaaaaac5aed10
        0xffffacbc57ac
        0xffffacb0d0fc

Thread 4 tid<281464542048128> lwtid<3318337>
        WaitLatchOrSocket(Latch volatile*, int, int, long) + 0x1ec
        ReaperBackendMain() + 0x1a4
        int GaussDbThreadMain<(knl_thread_role)21>(knl_thread_arg*) + 0x2cc
        0xaaaaac5aed10
        0xffffacbc57ac
        0xffffacb0d0fc

Thread 5 tid<281463566413696> lwtid<3332136>
        gs_signal_send(unsigned long, int, int) + 0x210
        signal_child(unsigned long, int, int) + 0x28
        get_stack_according_to_tid(unsigned long, StringInfoData*) + 0x130
        get_stack_according_to_lwtid(int, StringInfoData*) + 0xf4
        print_all_stack() + 0x9c
        stack_perf_main() + 0x1dc
        int GaussDbThreadMain<(knl_thread_role)68>(knl_thread_arg*) + 0x33c
        0xaaaaac5aed10
        0xffffacbc57ac
        0xffffacb0d0fc

Thread 6 tid<281463721078656> lwtid<3318410>
        pg_usleep(long) + 0x68
        0xaaaaacc597e8
        CheckPointBuffers(int, bool) + 0x80
        0xaaaaaca2b130
        CreateCheckPoint(int) + 0x67c
        CheckpointerMain() + 0x62c
        int GaussDbThreadMain<(knl_thread_role)38>(knl_thread_arg*) + 0x3ec
        0xaaaaac5aed10
        0xffffacbc57ac
        0xffffacb0d0fc

Thread 7 tid<281463652921216> lwtid<3318413>
        WaitLatchOrSocket(Latch volatile*, int, int, long) + 0x1ec
        PgstatCollectorMain() + 0x3b0
        int GaussDbThreadMain<(knl_thread_role)16>(knl_thread_arg*) + 0x324
        0xaaaaac5aed10
        0xffffacbc57ac
        0xffffacb0d0fc

Thread 8 tid<281464113246080> lwtid<3318930>
        WaitLatchOrSocket(Latch volatile*, int, int, long) + 0x1ec
        0xaaaaacbe62b4
        WalSenderMain() + 0xb20
        PostgresMain(int, char**, char const*, char const*) + 0x48c
        0xaaaaac5aec7c
        int GaussDbThreadMain<(knl_thread_role)1>(knl_thread_arg*) + 0x380
        0xaaaaac5aed10
        0xffffacbc57ac
        0xffffacb0d0fc

Thread 9 tid<281463202033536> lwtid<3318887>
        0xaaaaacc48664
        heartbeat_main() + 0x324
        int GaussDbThreadMain<(knl_thread_role)48>(knl_thread_arg*) + 0x3ec
        0xaaaaac5aed10
        0xffffacbc57ac
        0xffffacb0d0fc

Thread 10 tid<281463168479104> lwtid<3319212>
        WaitLatchOrSocket(Latch volatile*, int, int, long) + 0x1ec
        0xaaaaacbe62b4
        WalSenderMain() + 0xb20
        PostgresMain(int, char**, char const*, char const*) + 0x48c
        0xaaaaac5aec7c
        int GaussDbThreadMain<(knl_thread_role)1>(knl_thread_arg*) + 0x380
        0xaaaaac5aed10
        0xffffacbc57ac
        0xffffacb0d0fc
```

【测试代码】:

openGauss/CM

内容风险标识

评论 (2)

openGauss/CM .gitee-modal { width: 500px !important; }

内容风险标识