Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(ngcc): pause async processing if another process has the lockfile #35131

Conversation

petebacondarwin
Copy link
Member

@petebacondarwin petebacondarwin commented Feb 3, 2020

PR Checklist

Please check if your PR fulfills the following requirements:

PR Type

What kind of change does this PR introduce?

  • Bugfix
  • Feature
  • Code style update (formatting, local variables)
  • Refactoring (no functional changes, no api changes)
  • Build related changes
  • CI related changes
  • Documentation content changes
  • angular.io application / infrastructure changes
  • Other... Please describe:

What is the current behavior?

Issue Number: N/A

What is the new behavior?

Does this PR introduce a breaking change?

  • Yes
  • No

Other information

@petebacondarwin petebacondarwin changed the title feat(ngcc): WIP feat(ngcc): pause async processing if another process has the lockfile Feb 3, 2020
@pullapprove pullapprove bot requested a review from alxhub Feb 3, 2020
@petebacondarwin petebacondarwin force-pushed the ngcc-async-lockfile-pause branch 2 times, most recently from 4e8cf5b to 9210143 Compare Feb 4, 2020
@ngbot ngbot bot added this to the needsTriage milestone Feb 4, 2020
@ngbot ngbot bot added this to the needsTriage milestone Feb 4, 2020
@ngbot ngbot bot added this to the needsTriage milestone Feb 4, 2020
@petebacondarwin petebacondarwin marked this pull request as ready for review Feb 5, 2020
@petebacondarwin petebacondarwin force-pushed the ngcc-async-lockfile-pause branch 2 times, most recently from fb27dc1 to 15e611e Compare Feb 6, 2020
Copy link
Member

@gkalpak gkalpak left a comment

LGTM, except I think we should reset the waiting period when the pid in the lockfile changes.
Really excited to incorporate this into the CLI workflow \o/

This is needed by ngcc when reading volatile files that may
be changed by an external process (e.g. the lockfile).
…ckfile

ngcc uses a lockfile to prevent two ngcc instances from executing at the
same time. Previously, if a lockfile was found the current process would
error and exit.

Now, when in async mode, the current process is able to wait for the previous
process to release the lockfile before continuing itself.
@ngbot
Copy link

@ngbot ngbot bot commented Feb 10, 2020

I see that you just added the PR action: merge label, but the following checks are still failing:
    failure status "pullapprove" is failing
    pending forbidden labels detected: PR action: review
    pending status "google3" is pending
    pending 3 pending code reviews

If you want your PR to be merged, it has to pass all the CI checks.

If you can't get the PR to a green state due to flakes or broken master, please try rebasing to master and/or restarting the CI job. If that fails and you believe that the issue is not due to your change, please contact the caretaker and ask for help.

]]);
// We need to write to the rawFs to ensure that we don't update the cache at this point
rawFs.writeFile(lockFile.lockFilePath, '444');
await new Promise(resolve => setTimeout(resolve, 250));
Copy link
Member

@gkalpak gkalpak Feb 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks like a potential source of flakes 😁
Could we use fakeAsync() and tick() to make this test deterministic?

Copy link
Member Author

@petebacondarwin petebacondarwin Feb 10, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, I was a little bit worried about this. We cannot use the Angular fakeAsync stuff, since that is driven via Zone.js and we definitely not going to bring that in here just for testing...

I tried to make the timeouts as reliable as possible so that it would be extremely unlikely for it to flake:

  • having proven that the class has already checked the lockfile (and read its value) we are immediately writing the new one synchronously so there is no chance for the setTimeout inside LockFile to be triggered.
  • we then set a timeout for 250ms before the next check but the class has been configured to check the contents of the lockfile every 100ms. So there should be at least 2 opportunities for a setTimeout to be triggered before this testing one is triggered.
  • The class is setup not to fail until 10 attempts have occurred, which is a full second, so there is very little chance that the 250ms timeout is not triggered before we run out of attempts.

Do you have any better ideas?

Copy link
Member

@gkalpak gkalpak Feb 11, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough. I am fine leaving this as is and address it in the future if we start seeing flakes.

For future reference, one way to make this deterministic would be using Jasmine's Clock. There is one twist, because we need to also flush microtasks after each retryDelay ms, so it would be something like:

const clock = jasmine.clock();
const moveToNextIteration = () => {
  clock().tick(100);     // Or whatever `retryDelay` we have specified.
  return Promise.resolve();  // To wait for microtask queue to be flushed.
};

clock.withMock(async () => {
  const rawFs = getFileSystem();
  const fs = new CachedFileSystem(rawFs);
  const lockFile = new LockFileUnderTest(fs);

  fs.writeFile(lockFile.lockFilePath, '188');
  const promise = lockFile.lock(async () => lockFile.log.push('fn()'));
  expect(lockFile.log).toEqual(['create()']);
  expect(lockFile.getLogger().logs.info).toEqual([...]);

  rawFs.writeFile(lockFile.lockFilePath, '444');
  await moveToNextIteration();
  await moveToNextIteration();
  expect(lockFile.getLogger().logs.info).toEqual([...]]);

  ...
});

@kara
Copy link
Contributor

@kara kara commented Feb 11, 2020

@petebacondarwin Are you still waiting on a review from @alxhub or is this ready to presubmit/merge?

@petebacondarwin
Copy link
Member Author

@petebacondarwin petebacondarwin commented Feb 11, 2020

Just a presubmit please @kara

@petebacondarwin petebacondarwin removed the request for review from alxhub Feb 11, 2020
alxhub added a commit that referenced this issue Feb 19, 2020
…ckfile (#35131)

ngcc uses a lockfile to prevent two ngcc instances from executing at the
same time. Previously, if a lockfile was found the current process would
error and exit.

Now, when in async mode, the current process is able to wait for the previous
process to release the lockfile before continuing itself.

PR Close #35131
alxhub added a commit that referenced this issue Feb 19, 2020
…35131)

This is needed by ngcc when reading volatile files that may
be changed by an external process (e.g. the lockfile).

PR Close #35131
alxhub added a commit that referenced this issue Feb 19, 2020
…ckfile (#35131)

ngcc uses a lockfile to prevent two ngcc instances from executing at the
same time. Previously, if a lockfile was found the current process would
error and exit.

Now, when in async mode, the current process is able to wait for the previous
process to release the lockfile before continuing itself.

PR Close #35131
alxhub added a commit that referenced this issue Feb 19, 2020
…s the lockfile (#35131)"

This reverts commit b970028.

This is a feature commit and was improperly merged to the patch branch.
@petebacondarwin petebacondarwin deleted the ngcc-async-lockfile-pause branch Feb 23, 2020
@angular-automatic-lock-bot
Copy link

@angular-automatic-lock-bot angular-automatic-lock-bot bot commented Mar 25, 2020

This issue has been automatically locked due to inactivity.
Please file a new issue if you are encountering a similar or related problem.

Read more about our automatic conversation locking policy.

This action has been performed automatically by a bot.

@angular-automatic-lock-bot angular-automatic-lock-bot bot locked and limited conversation to collaborators Mar 25, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

None yet

5 participants