Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Replication is broken: Slave_IO_Running: No, Slave_SQL_Running: No #983

Open
artem-dronov opened this issue Jun 2, 2021 · 5 comments
Open

Replication is broken: Slave_IO_Running: No, Slave_SQL_Running: No #983

artem-dronov opened this issue Jun 2, 2021 · 5 comments

Comments

@artem-dronov
Copy link

@artem-dronov artem-dronov commented Jun 2, 2021

Good day! gh-ost cannot find the replication master, previously we promoted the slave to the master, that is, if we SHOW SLAVE STATUS on the master, it will show Slave_IO_Running: No, Slave_SQL_Running: No, what should we do in this case?

./gh-ost --host=master_host --user=test_user --password=????? \ --database=test_db --table=test --assume-rbr --allow-on-master \ --verbose --alter="add test SMALLINT NULL" --chunk-size=3000 --assume-rbr \ --max-load=Threads_connected=20 > gh.log &

2021-06-02 09:08:53 INFO starting gh-ost 1.1.1
2021-06-02 09:08:53 INFO Migrating test_db.test
2021-06-02 09:08:53 INFO inspector connection validated on master_host:3306
2021-06-02 09:08:53 INFO User has REPLICATION CLIENT, REPLICATION SLAVE privileges, and has ALL privileges on test_db.*
2021-06-02 09:08:53 INFO binary logs validated on master_host:3306
2021-06-02 09:08:53 INFO Inspector initiated on slave_host:3306, version 5.7.32-log
2021-06-02 09:08:53 INFO Table found. Engine=InnoDB
2021-06-02 09:08:53 INFO Estimated number of rows via EXPLAIN: 75277111
2021-06-02 09:08:53 INFO Recursively searching for replication master
2021-06-02 09:08:53 INFO Tearing down inspector
2021-06-02 09:08:53 FATAL Replication on master_host:3306 is broken: Slave_IO_Running: No, Slave_SQL_Running: No. Please make sure replication runs before using gh-ost.
@shlomi-noach
Copy link
Contributor

@shlomi-noach shlomi-noach commented Jun 2, 2021

You should run RESET SLAVE ALL on your current primary. This is assuming there's nothing in the relay logs that you need. that's the normal thing to do after promoting a new primary -- but that's just my suggestion, make sure you understand what you are doing.

@artem-dronov
Copy link
Author

@artem-dronov artem-dronov commented Jun 2, 2021

You should run RESET SLAVE ALL on your current primary. This is assuming there's nothing in the relay logs that you need. that's the normal thing to do after promoting a new primary -- but that's just my suggestion, make sure you understand what you are doing.

Does this action not affect the health of the primary node?

@shlomi-noach
Copy link
Contributor

@shlomi-noach shlomi-noach commented Jun 2, 2021

right now your primary node thinks it's a replica. But if it's a primary, it probably shouldn't think so, it's risky. Because it may try to reconnect to whatever master_host is listed right now.

Define "affect the health"? RESET SLAVE ALL does not affect production traffic and is a non-blocking operation.

@artem-dronov
Copy link
Author

@artem-dronov artem-dronov commented Jun 2, 2021

I'm sorry, I haven't done this before. I'm more worried about whether the execution of this command on the master node will affect the synchronization of other slave nodes connected to the master.

@shlomi-noach
Copy link
Contributor

@shlomi-noach shlomi-noach commented Jun 2, 2021

Running this command on the primary node (master) does not affect the synchronization of the replicas connected to the primary node.

Obviously do not run it on the replicas.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@shlomi-noach @artem-dronov