I have this errors multiple times by day...
my first thought was the cache size that was around 1.5gb... but the problem still happend even with a cache of about 150mb

@kevinlul @fl250144 Please provide full logs with step debugging enabled so that we can debug this further. Thank you.

Hello, unfortunately the bug was not reproducible, as it cleared up when we cancelled the workflow and reran it. I think the network issues that may cause this bug (e.g. GitHub API hanging) would be difficult for us to reproduce.

@kevinlul Please keep the step debugging on and provide the logs if it reoccurs anytime soon.

37.zip
@vsvipul here it is
this only one example, i have more if you need.

We have experienced this multiple times as well.

We have this issue as well if we use self hosted runners hosted in europe.
At the same time usiing the Github provided runners I don't get this :/

Anybody else who's getting this using own runners?

I encountered this behavior now as well multiple times (on github provided runners). The first time the cache download was stuck for 6 hours. As it happens only rarely non-deterministic I can't provide any debug logs nether the link to the workflow as it's a private repo.

For me this occurred during using the cache feature of actions/setup-node which internally uses actions/cache.

Run actions/setup-node@v3
with:
  node-version-file: .nvmrc
  cache: yarn
  always-auth: false
  check-latest: false
  token: ***
Resolved .nvmrc as 16
Found in cache @ /opt/hostedtoolcache/node/16.15.0/x64
/usr/local/bin/yarn --version
1.22.18
/usr/local/bin/yarn cache dir
/home/runner/.cache/yarn/v6
Received 0 of 1585939639 (0.0%), 0.0 MBs/sec
Received 83886080 of 1585939639 (5.3%), 40.0 MBs/sec
Received 222298112 of 1585939639 (14.0%), 70.7 MBs/sec
Received 348127232 of 1585939639 (22.0%), 83.0 MBs/sec
Received 494927872 of 1585939639 (31.2%), 94.3 MBs/sec
Received 612368384 of 1585939639 (38.6%), 97.2 MBs/sec
Received 713031680 of 1585939639 (45.0%), 97.0 MBs/sec
Received 817889280 of 1585939639 (51.6%), 97.4 MBs/sec
Received 910163968 of 1585939639 (57.4%), 96.3 MBs/sec
Received 1002438656 of 1585939639 (63.2%), 95.5 MBs/sec
Received 1107296256 of 1585939639 (69.8%), 95.9 MBs/sec
Received 1199570944 of 1585939639 (75.6%), 95.2 MBs/sec
Received 1296039936 of 1585939639 (81.7%), 95.0 MBs/sec
Received 1396703232 of 1585939639 (88.1%), 95.0 MBs/sec
Received 1497366528 of 1585939639 (94.4%), 95.1 MBs/sec
Received 1577551031 of 1585939639 (99.5%), 93.9 MBs/sec
Received 1577551031 of 1585939639 (99.5%), 88.4 MBs/sec
Received 1577551031 of 1585939639 (99.5%), 83.5 MBs/sec
Received 1577551031 of 1585939639 (99.5%), 79.1 MBs/sec
Received 1577551031 of 1585939639 (99.5%), 75.1 MBs/sec
Received 1577551031 of 1585939639 (99.5%), 71.6 MBs/sec
Received 1577551031 of 1585939639 (99.5%), 68.3 MBs/sec
Received 1577551031 of 1585939639 (99.5%), 65.3 MBs/sec
Received 1577551031 of 1585939639 (99.5%), 62.6 MBs/sec
Received 1577551031 of 1585939639 (99.5%), 60.1 MBs/sec
[.... truncated, thousand lines and several hours later...] 
Received 1577551031 of 1585939639 (99.5%), 0.1 MBs/sec
Received 1577551031 of 1585939639 (99.5%), 0.1 MBs/sec
Received 1577551031 of 1585939639 (99.5%), 0.1 MBs/sec
Received 1577551031 of 1585939639 (99.5%), 0.1 MBs/sec
Received 1577551031 of 1585939639 (99.5%), 0.1 MBs/sec
Error: The operation was canceled.

As a workaround I added timeout-minutes to the steps so the workflow will at least get canceled automatically.

We suspect an issue with the Azure SDK and we've opened an issue for the same here -
Azure/azure-sdk-for-js#22321

This happened to my repo as well. 😞

same happening to our repo as well,
self-hosted macOS arm64 ,
Cache Size: ~786 MB (823760266 B),
action repository 'actions/cache@v3' (SHA:c3f1317a9e7b1ef106c153ac8c0f00fed3ddbc0d)

Same problem here - gets stuck at 99.9% randomly.

Received 3103784959 of 3165194974 (98.1%), 28.0 MBs/sec
Received 3145727999 of 3165194974 (99.4%), 28.1 MBs/sec
Received 3161000670 of 3165194974 (99.9%), 27.9 MBs/sec
(...more lines with speed going down to 0.1...)
Received 3161000670 of 3165194974 (99.9%), 0.1 MBs/sec
Received 3161000670 of 3165194974 (99.9%), 0.1 MBs/sec
Error: The operation was canceled.

Hi all,

We have released a new actions/cache version v3.0.7 that doesn't run the stuck runner for 6 hours, instead aborts the download and gets out of the cache restore step in case the download doesn't complete within an hour. Users using actions/cache@v3 need not do any change and should see this behaviour in their action runs starting today.

This has been done as an intermittent fix as we are still waiting for the azure team to release the fix for the same. Azure/azure-sdk-for-js#22321

This is a big improvement; thank you! Is there any way to configure the timeout? 1 hour still hurts when our workflow typically runs in ~2.5 minutes.

Hi @Kurt-von-Laven 👋🏽
For now we've set it to one hour as we didn't want any customer to face issue in case their self hosted runner runs on a low network bandwidth. As an improvement on top of this, we're looking for options to make this better and configuring timeout is one of the options.

We'll get back regarding this soon. Thanks for your patience. 😊

How is the progress on this issue going? We are seeing this multiple times a day, even with a retry step based on initial failure from SEGMENT_DOWNLOAD_TIMEOUT_MINS

I know they recently lowered the default timeout to 10 minutes, so I assume the root cause upstream remains unfixed.

I know they recently lowered the default timeout to 10 minutes, so I assume the root cause upstream remains unfixed.

The 10 minutes is for a segment. What happened to use yesterday was that it had been running for 47 minutes before I manually cancelled it. Perhaps in our case, it constantly but very slowly was moving a head with segments in under 10 minutes but ultimately would have taken the full 60 minutes eventually.

@kotewar in #810 (comment), you wrote:

Apologies, the environment variable had a typo in the issue as well as documentation. That's why it was not working as expected.

SEGMENT_DOWNLOAD_TIMEOUT_MINS with S at the end should be used.

Could you please edit all of the other comments where you wrote SEGMENT_DOWNLOAD_TIMEOUT_MIN?

• Action can wait for at least 27 minutes when no progress is being made on the download #810 (comment)
• Action can wait for at least 27 minutes when no progress is being made on the download #810 (comment)
• Action can wait for at least 27 minutes when no progress is being made on the download #810 (comment) (this will require a comment clarifying things)

We tripped on this today after it took an hour to retrieve a cache.

@jsoref, sorry. Updated the comments.

Hey everyone,

I'm Adam, co-founder for BuildJet for GitHub Actions. Our customers often complain about reliability and speed issues like the one reported here. Today, we launched BuildJet Cache, a GitHub Cache alternative thats reliable, fast and free.

It’s fully compatible with GitHub’s Cache, all you need to do is replace action/cache with buildjet/cache. It works on any runner - be it official, self-hosted, or BuildJet. Whichever runner you choose, BuildJet Cache doubles your free storage, offering a generous 20 GB/repo/week storage space at no cost.

Head over to our launch post for more details: https://buildjet.com/for-github-actions/blog/launch-buildjet-cache

This issue is the leading cause of failures in our CI workflows. I'm seeing this on almost every run, and it's causing huge problems. Is there anything that we as users can do to show the Azure team that this issue needs attention? The last comment I saw from that team said that they'd place it on their backlog, so I've got zero confidence this issue is going to be fixed anytime soon.

My general approach is to open tickets to support. I can't say it'll help.

If it were happening to me regularly, I'd try the third party drop-in replacement. We've only tripped on it occasionally, so I'm mostly frustrated with the lousy messaging which I poked them to improve.

We have merged a fix for this (see actions/toolkit#1484). Please @-mention me if you observe failures so that I can investigate.