Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added a new GHE_REMOTE_GIT_FSCK option #225

Closed
wants to merge 10 commits into from
Closed

Conversation

@rubiojr
Copy link
Member

@rubiojr rubiojr commented Jun 10, 2016

Run git fsck remotely after a restore and report the results.

Looks something like:

Restoring MySQL database ...
 --> Importing MySQL data...
Restoring Redis database ...
Restoring Git repositories into cluster ...
Restoring Gists into cluster ...
Checking remote git repositories ...
Running git fsck on ghe-test-cluster-migration-node1 ...
* Repos verified: 4, Errors: 1, Took: 5s
Running git fsck on ghe-test-cluster-migration-node2 ...
* Repos verified: 4, Errors: 0, Took: 5s
Running git fsck on ghe-test-cluster-migration-node3 ...
* Repos verified: 4, Errors: 0, Took: 4s
Running git fsck on ghe-test-cluster-migration-node4 ...
* Repos verified: 3, Errors: 0, Took: 4s

When the verbose mode is enabled:

Restoring MySQL database ...
 --> Importing MySQL data...
Restoring Redis database ...
Restoring Git repositories into cluster ...
Restoring Gists into cluster ...
Checking remote git repositories ...
Running git fsck on ghe-test-cluster-migration-node1 ...
ERROR /data/user/repositories/c/nw/c8/1e/72/2/2.git ghe-admin/test2 1s
error: HEAD: invalid sha1 pointer 8dc523028e592fb72020f6f1901ac033e326eebb
error: refs/heads/master: invalid sha1 pointer 8dc523028e592fb72020f6f1901ac033e326eebb
error: refs/heads/master: invalid reflog entry 8dc523028e592fb72020f6f1901ac033e326eebb
notice: No default references
* Repos verified: 4, Errors: 1, Took: 5s
Running git fsck on ghe-test-cluster-migration-node2 ...
* Repos verified: 4, Errors: 0, Took: 5s
Running git fsck on ghe-test-cluster-migration-node3 ...
* Repos verified: 4, Errors: 0, Took: 4s
Running git fsck on ghe-test-cluster-migration-node4 ...
* Repos verified: 3, Errors: 0, Took: 4s

It will exit 1 if the remote fsck fails and GHE_REMOTE_GIT_FSCK_ABORT_ON_ERROR=yes,
warn otherwise.

Note: git fsck is quite expensive and slow on large installs and for now it's only
useful for debugging/troubleshooting so the new option is disabled by default.

/cc @github/backup-utils

Run git fsck remotely after a restore and report the results.

Looks something like:

```
Restoring MySQL database ...
 --> Importing MySQL data...
Restoring Redis database ...
Restoring Git repositories into cluster ...
Restoring Gists into cluster ...
Checking remote git repositories ...
Running git fsck on ghe-test-cluster-migration-node1 ...
* Repos verified: 4, Errors: 1, Took: 5s
Running git fsck on ghe-test-cluster-migration-node2 ...
* Repos verified: 4, Errors: 0, Took: 5s
Running git fsck on ghe-test-cluster-migration-node3 ...
* Repos verified: 4, Errors: 0, Took: 4s
Running git fsck on ghe-test-cluster-migration-node4 ...
* Repos verified: 3, Errors: 0, Took: 4s
```

When the verbose mode is enabled:

```
Restoring MySQL database ...
 --> Importing MySQL data...
Restoring Redis database ...
Restoring Git repositories into cluster ...
Restoring Gists into cluster ...
Checking remote git repositories ...
Running git fsck on ghe-test-cluster-migration-node1 ...
ERROR /data/user/repositories/c/nw/c8/1e/72/2/2.git ghe-admin/test2 1s
error: HEAD: invalid sha1 pointer 8dc523028e592fb72020f6f1901ac033e326eebb
error: refs/heads/master: invalid sha1 pointer 8dc523028e592fb72020f6f1901ac033e326eebb
error: refs/heads/master: invalid reflog entry 8dc523028e592fb72020f6f1901ac033e326eebb
notice: No default references
* Repos verified: 4, Errors: 1, Took: 5s
Running git fsck on ghe-test-cluster-migration-node2 ...
* Repos verified: 4, Errors: 0, Took: 5s
Running git fsck on ghe-test-cluster-migration-node3 ...
* Repos verified: 4, Errors: 0, Took: 4s
Running git fsck on ghe-test-cluster-migration-node4 ...
* Repos verified: 3, Errors: 0, Took: 4s
```

It will exit 1 if the remote fsck fails and GHE_REMOTE_GIT_FSCK_ABORT_ON_ERROR=yes,
warn otherwise.

Note: git fsck is quite expensive and slow on large installs and for now it's only
useful for debugging/troubleshooting so the new option is disabled by default.
@djdefi
Copy link
Member

@djdefi djdefi commented Apr 5, 2017

💖 This seems like it would be useful for automating restores to a DR or staging environment regularly. We could likely catch some issues before they become critical.

…nflicts
@lildude
Copy link
Member

@lildude lildude commented Jan 10, 2018

Do you still want this in backup-utils @rubiojr?

@rubiojr
Copy link
Member Author

@rubiojr rubiojr commented Jan 11, 2018

@lildude I think it'd be useful for customers as @djdefi suggested. Thoughts?

@lildude
Copy link
Member

@lildude lildude commented Jan 11, 2018

Thoughts?

None 😁 . Was really just seeing if we want to still go ahead with this and merge this PR.

@rubiojr
Copy link
Member Author

@rubiojr rubiojr commented Jan 26, 2018

None 😁 . Was really just seeing if we want to still go ahead with this and merge this PR.

Would be worth double checking if this still works before merging, it's been a while.

lildude added 7 commits Feb 3, 2018
It's already defined and test friendly.
It's risky and not cross-platform
@github github deleted a comment Aug 5, 2019
@rubiojr rubiojr closed this Feb 12, 2020
@rubiojr rubiojr deleted the rubiojr/ghe-remote-git-fsck branch Feb 12, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

3 participants
You can’t perform that action at this time.