Skip to content

Allow injection of TUFMetadataDir in tests #10478

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Feb 21, 2025

Conversation

williammartin
Copy link
Member

Description

Fixes #10390 (hopefully)

As described in #10390 (comment) the fundamental issue here is that there is a third party client (TUF) that is usually instantiated once on a gh attestation invocation but gets instantiated once per test. It downloads and stores json metadata in a shared location. It does this by writing a temp file and then using os.Rename to move it to the right place…godoc says:

Even within the same directory, on non-Unix platforms Rename is not an atomic operation.

This PR allows for injection of this location.

Reviewer Notes

I'm not really sure whether TUF is something that is supposed to leak out of the verification package, so this may be leaky. I chose to use an Option type to signify the possible absence of this because the alternative without larger refactoring is to pass around bare strings and rely on the zero value, which is pretty uninformative to the reader.

An alternative might be to have a struct that can be passed to the DefaultOptionsWithCacheSetting and GitHubTUFOptions functions but that seems overblown right now.

I'm not wedded to either of these things, just wanted to get something out that fixed the flaky tests, and it seems to have:

PS C:\Users\williammartin\workspace\cli> 1..10 | % { go test -tags=integration ./pkg/cmd/attestation/verification ./pkg/cmd/attestation/verify -count 2 -failfast }
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.849s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        22.185s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.424s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        20.310s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.837s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        20.663s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.917s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        21.998s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  5.919s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        20.572s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.875s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        22.144s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  5.236s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        21.297s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.907s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        21.929s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.854s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        20.692s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verification  4.439s
ok      github.com/cli/cli/v2/pkg/cmd/attestation/verify        21.337s

This avoids multiple tests using the same dir for metadata, which was causing flakes
Copy link
Contributor

@malancas malancas left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@williammartin williammartin merged commit 7fa4882 into trunk Feb 21, 2025
17 checks passed
@williammartin williammartin deleted the wm/fix-flaky-tuf-client-race branch February 21, 2025 10:58
tmeijn pushed a commit to tmeijn/dotfiles that referenced this pull request Mar 6, 2025
This MR contains the following updates:

| Package | Update | Change |
|---|---|---|
| [cli/cli](https://github.com/cli/cli) | minor | `v2.67.0` -> `v2.68.0` |

MR created with the help of [el-capitano/tools/renovate-bot](https://gitlab.com/el-capitano/tools/renovate-bot).

**Proposed changes to behavior should be submitted there as MRs.**

---

### Release Notes

<details>
<summary>cli/cli (cli/cli)</summary>

### [`v2.68.0`](https://github.com/cli/cli/releases/tag/v2.68.0): GitHub CLI 2.68.0

[Compare Source](cli/cli@v2.67.0...v2.68.0)

#### What's Changed

##### ✨ Features

-   \[gh repo view] Improve error message for forked repo by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10334
-   Add signer-digest, source-ref, and source-digest options for `gh attestation verify` by [@&#8203;malancas](https://github.com/malancas) in cli/cli#10308
-   \[gh pr checkout] Add --no-tags option to git fetch commands in checkout by [@&#8203;latzskim](https://github.com/latzskim) in cli/cli#10479
-   \[`gh issue/pr comment`] Add `--create-if-none` and prompts to create a comment if no comment already exists  by [@&#8203;latzskim](https://github.com/latzskim) in cli/cli#10427
-   \[gh cache delete --all] Add `--succeed-on-no-caches` flag to return exit code 0 by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10327
-   \[gh release create] Fail when there are no new commits since the last release by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10398
-   update default upstream when forking repo during MR creation by [@&#8203;daviddl9](https://github.com/daviddl9) in cli/cli#10458

##### 🐛 Fixes

-   Refactor `GetLocalAttestations` and clean up custom registry transport by [@&#8203;malancas](https://github.com/malancas) in cli/cli#10382
-   Check `GH_REPO` too in addition to `--repo` for disambiguation by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#10539
    -   (Fixes `gh secret` subcommands not working outside of a repository)
-   Fix unhandled panic in FindWorkflow and add tests by [@&#8203;jtmcg](https://github.com/jtmcg) in cli/cli#10521
-   Fix checkout when URL arg is from fork and cwd is upstream by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#10512
-   \[gh api] Escape package name (URL encoding) for packages endpoint by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10384
-   Fix `remoteResolver` caching issue by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10456
-   Fix gh project item-edit to allow --number 0 as a valid value by [@&#8203;aryanbhosale](https://github.com/aryanbhosale) in cli/cli#10417
-   Add mutex to fix race in attestation test client by [@&#8203;codysoyland](https://github.com/codysoyland) in cli/cli#10439
-   Base64 decode GPG passphrase in deployment workflow by [@&#8203;BagToad](https://github.com/BagToad) in cli/cli#10546

##### 📚 Docs & Chores

-   Deep Dive Document Release Process by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#10503
-   Inconsistent format of examples in help text by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10508
-   Inconsistent format of description of flags (starting with lowercase letter) by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10507
-   Update Go version to 1.23 in CONTRIBUTING.md by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#10504
-   Fix minor auth login help typo by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#10501
-   docs: document how to revoke `gh` OAuth tokens in `auth logout`'s help by [@&#8203;BagToad](https://github.com/BagToad) in cli/cli#10490
-   chore: update codespaces Go version by [@&#8203;BagToad](https://github.com/BagToad) in cli/cli#10491
-   Allow injection of TUFMetadataDir in tests by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#10478
-   refactor: use a more straightforward return value by [@&#8203;beforetech](https://github.com/beforetech) in cli/cli#10489
-   Use subtests in attestation verification integration tests by [@&#8203;williammartin](https://github.com/williammartin) in cli/cli#10463
-   Fix typo in README by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10445
-   Update usage to lower-kebab-case by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10447
-   Standardize URLs by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10429
-   Remove trailing whitespace by [@&#8203;iamazeem](https://github.com/iamazeem) in cli/cli#10430

##### :dependabot: Dependencies

-   Bump actions/attest-build-provenance from 2.2.0 to 2.2.2 by [@&#8203;dependabot](https://github.com/dependabot) in cli/cli#10518
-   Bump github.com/go-jose/go-jose/v4 from 4.0.2 to 4.0.5 by [@&#8203;dependabot](https://github.com/dependabot) in cli/cli#10499
-   Bump github.com/spf13/pflag from 1.0.5 to 1.0.6 by [@&#8203;dependabot](https://github.com/dependabot) in cli/cli#10338

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied.

♻ **Rebasing**: Whenever MR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this MR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this MR, check this box

---

This MR has been generated by [Renovate Bot](https://github.com/renovatebot/renovate).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xODYuMCIsInVwZGF0ZWRJblZlciI6IjM5LjE4Ni4wIiwidGFyZ2V0QnJhbmNoIjoibWFpbiIsImxhYmVscyI6WyJSZW5vdmF0ZSBCb3QiXX0=-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

TestLiveSigstoreVerifier/with_2/3_verified_attestations is flaky
2 participants