Fantastic to see progress on this!

With regards to the questions:

Should I add Windows support?

I'd say that's a requirement, yes.

Should I transfer the Corepack repository to the Node org? If so, how?

To do so, we'd need an issue opened in the nodejs/admin repo proposing the change. Assuming there are no objections after a few days, it's considered accepted and we just move it in. There's no real fanfare here.

What testing strategy do you recommend? Corepack itself is tested on its repository: here

Take a look at the test/parallel/test-npm-* tests and follow a similar pattern. The testing does not have to be comprehensive but should be enough to be reasonably certain that changes in core haven't broken something in corepack.

There still seem to be some lingering TODOs from the issues linked in the original point before this should land:

• what are the criteria for adding package managers to corepack
• what are the criteria for removing package mangers to corepack
• what are the criteria for evolutions of package managers (like Berry) being added to corepack
• do all the current package managers meet all those criteria
• are there any package managers that are left out despite meeting that criteria
• have we measured the impact on our consumers, like FaaS platforms or end-user developers
• what is that impact on our users, beyond speculation
• can users turn this off

I'd also like to ask:

• are we comfortable adding something this fundamental that has a single maintainer
• are we as the Node.js willing and 100% able to commit, without question, to supporting this for the next decade
• if we land this, shouldn't it be behind a flag

Additionally, I think it's worth being upfront about our intents here: do we plan to eventually try to install npm through this? This should be probably be discussed since it could be impactful in deciding how to approach landing this / if it should be landed. Others shared their views in some of the aforementioned issues, and I have my own opinions on this that I'd be more than happy to share if they are potentially relevant.

Should I add Windows support?

All tier 1 and tier 2 platforms listed on https://github.com/nodejs/node/blob/master/BUILDING.md#platform-list must be supported before this moves out of experimental (imo it's fine to land without initial support for some platforms if this feature is experimental and opt-in)

To some of @bnb's questions:

what are the criteria for adding package managers to corepack
what are the criteria for removing package mangers to corepack
what are the criteria for evolutions of package managers (like Berry) being added to corepack
do all the current package managers meet all those criteria
are there any package managers that are left out despite meeting that criteria

I do not believe that these need to be answered before this lands and while it is still marked experimental. I would agree that these are exit criteria to graduate from experimental, however.

have we measured the impact on our consumers, like FaaS platforms or end-user developers
what is that impact on our users, beyond speculation

We really won't know until it's in user's hands, for that, we should land it as opt-in experimental.

can users turn this off

They should be able to, yes.

are we comfortable adding something this fundamental that has a single maintainer
are we as the Node.js willing and 100% able to commit, without question, to supporting this for the next decade

We haven't applied that strict of reasoning to other things and we shouldn't now. There's no guarantee that anything we add to Node.js is going to have people willing and able "without question" to support it for the next decade. That said, this is the whole reason for wanting to bring the project into Node.js and encourage additional maintainers. That said, this thing is really small and shouldn't need much effort to maintain.

if we land this, shouldn't it be behind a flag

Given that it's an install-time thing and not a Node.js runtime thing, the "flag" would be an option in the interactive installer to omit it, which absolutely should be the case.

are we comfortable adding something this fundamental that has a single maintainer

I think there will be no lack of maintainers. This feature would be really useful for pnpm users, so I'll help out if there'll be a need. I believe other pnpm users/contributors will be willing to help as well.

I have some concerns.

1. What's the story for self-updating? Ie, if someone runs npm install -g npm@latest, and "npm" is a shell script that calls pmm npm "$@", then self-updating will be broken without specifying --force. If that is not addressed, it presents a significant security concern, as we can no longer ship urgent patch upgrades to package managers (or even new features) without a new node release or requiring that users manually remove the pmm shell shim.

2. What's the story for updating the version that pmm has in its "known good" set? Surely, pmm's knowledge of npm/yarn/pnpm version goodness will be less than that of the maintainers of these projects who triage and investigate bugs on a daily basis. If self-updating is blocked (or even just changed from the currently documented flows), then this will become a huge hassle almost immediately.

3. How is this better than just shipping all three package managers, and providing an option in the msi/pkg installer to choose them as optional components? That seems much simpler, a less complicated user experience, an equivalently level playing field, and neatly addresses the two concerns above. If the objection is "smaller core", well, that ship has sailed. The optimal solution there would seem to be "no package manager", which no one wants. Shipping a new shell script shim which installs any of 3 package managers, and can't be easily updated, feels like we're making core reduced on bytes, but vastly bloated on complexity.

What's the story for self-updating

Well, to be clear, corepack does not currently replace the npm binary. So there's no immediate risk there. That said, the corepack prepare and corepack hydrate commands cover updating the package manager versions. Newer versions can also be pinned in the package.json for a project.

What's the story for updating the version that pmm has in its "known good" set?

Pull request... No different really than updating the npm client version in core.

How is this better than just shipping all three package managers,

People who do not want yarn installed should not ever need to have it. Shipping all three would bloat the installer image needlessly, and providing an option to download on install is really no different. This gives the additional benefit of allowing different projects to select which package manager they require, which an install time only option would not provide.

and can't be easily updated

Not sure what you're basing this on. The design documentation for corepack goes into detail on how both corepack and the package manager would be updated and the flow is not much more complicated than what we have currently.

@jasnell which yarn exiting zero is "having it".

I don't think anyone suggested how to make this opt in for v15 yet, and imo if it's opt out it's definitely a semver major change (especially with @ljharb point on which). My suggestion is to initially ship it with Node.js without creating the scripts for each package manager, and then users who want to opt in the behavior can run pmm init/pmm opt-in/node --experimental-pmm-opt-in (those are some suggestions but you get the gist).

What's the story for self-updating? Ie, if someone runs npm install -g npm@latest, and "npm" is a shell script that calls pmm npm "$@", then self-updating will be broken without specifying --force

As @jasnell mentioned the npm binary is untouched by this PR, and I personally don't intend to change that, as I don't feel it's my place to make any critical change about a competing project (plus it comes with its own set of considerations unique to its situation). My only goal here is to ensure that my users' interests are accounted for - nothing more, nothing less 🙂

What's the story for updating the version that pmm has in its "known good" set?

The LKG releases bundled with Node will be updated by pull request (just like npm is updated by pull request). The LKG releases that apply on an individual user's machine (typically what people would change by running npm install -g) can be updated by this user using --activate.

Note that npm install -g will keep working, as it will overwrite the Corepack symlink with the regular one (meaning that it won't break the workflow for people running this command right after installing Node). @isaacs If you're open to collaboration please refer to this topic, as we can probably slightly improve the npm install -g DX together (right now it works because Yarn workarounds a small issue with npm).

My suggestion is to initially ship it with Node.js without creating the scripts for each package manager, and then users who want to opt in the behavior can run pmm init/pmm opt-in/node --experimental-pmm-opt-in (those are some suggestions but you get the gist).

(note: pmm is now Corepack, since pmm was already taken on the npm registry)

My main worry with making it opt-in is classic: if there's a problem, we will be less likely to know about it - meaning that once we reach the end of the experiment, even if nothing bad happened, we still won't have enough confidence that the workflow will be stable enough for an even release (especially since Node wouldn't monitor how many people opted-in, so we would have to assume that it would be a very low amount).

By contrast, seeing real-life feedback for an opt-out would guarantee that nothing sneaked by us, allowing us to safely make informed decision later down the road. And of course that wouldn't prevent us from having corepack disable, just in case.

People who do not want yarn installed should not ever need to have it. Shipping all three would bloat the installer image needlessly, and providing an option to download on install is really no different. This gives the additional benefit of allowing different projects to select which package manager they require, which an install time only option would not provide.

On this point there is a difference between shipping the package managers in the installer image vs option to download on install in that the former allows installation on a machine without an internet connection.

@arcanis, based on the feedback, the concrete changes I'd like to see before moving this forward:

1. Initially, let's land it as opt-in... that is, require a flag for installer.py to enable it. This will give folks an opportunity to experiment hands on and identify any potential issues. The idea would be to switch that to opt-out after a couple of minor release cycles once it has been demonstrated that folks fears of potential issues have been relieved.

2. Let's implement the command to remove/disable a particular jumper. Something like a corepack disable ${name} should do the trick. In addition, let's make it possible to specify these in the installer.py command line arguments so that a particular jumper can be disabled immediately on installation. Similarly, a corepack enable ${name} should reenable a particular jumper.

Given that this does not impact npm at all, there really shouldn't be anything further to discuss there unless the npm team decides to adopt it.

also corepack disable to disable everything

The idea would be to switch that to opt-out after a couple of minor release cycles once it has been demonstrated that folks fears of potential issues have been relieved

I still don't think we can switch from opt-in to opt-out in a semver minor (not as it is today at least). IMO this is not something we should rush as it has the potential of breaking many automated workflows via scenarios we didn't consider so far.

For give me if I am wrong, just trying to understand things, first I love the idea of a level playing ground for all package managers, but why not just have --with-npm or --with-yarn as flags on install and let the user decide at install what package manager they want as their default and then go from there?

If I use npm for everything on homebrew. Ican just do brew install node --with-npm and then now I have npm, or brew install node --with-yarn and now npm doesn't exist, but yarn does.

I am worried about the complexity and UX of having all these extra commands and things going around. I know myself using this would be trivial, but my co-worker who updates our gatsby site and only knows what commands we teach her and often messages me because npm start failed because she forgot to cd into the right directory would add TONS of extra confusion for her.

That would more or less be something that brew does, not that we do. Our official distributions, aside from the windows installers, are just files that you copy, and you're free to not copy the npm-related files.

This message might answer your question: #15244 (comment)

Which package manager to use is not up to the user but rather up to the project. All developers that work on a given project should use the same package manager.

Ok, so if corepack doesn't install a npm "jump script" as the proposal suggests, then you're saying that someone installs node, and runs npm and it's not found?

I'm not sure how this needle is being threaded here. Either there's an executable in the PATH called npm, or there isn't.

Unfortunately approach doesn't fit any Linux installed that I'm aware of (and we don't maintain the brew recipe for Node.js either). It also doesn't fix the problem that "I want to use npm but this project I'm working on only works with yarn".

I am worried about the complexity and UX of having all these extra commands and things going around

The idea is that no extra commands will be required (you need to run yarn? You just run yarn, if it's not installed it will install it for you). There was a demonstration of this tool on the Aug 27 TSC meeting for anyone who wants to see how it works.

npm is still packaged in its entirety, not part of corepack.

@isaacs there will be no changes to npm at this time. If a user runs npm, npm will run, that's it.

Ok, so the idea is that node ships npm and corepack, with a yarn jump binary that installs yarn the first time you run it?

What happens when someone runs npm install yarn --global? (If that jump binary is at /usr/local/bin/yarn, and it is not a symlink to the bin in ../node_modules/yarn it'll refuse to clobber it. I believe yarn has the same protection in place.)

If that's the case, then npm can self-update and yarn can't (and can't be installed by npm either). Not exactly a level playing field.

Note that npm install -g will keep working, as it will overwrite the Corepack symlink with the regular one (meaning that it won't break the workflow for people running this command right after installing Node). @isaacs If you're open to collaboration please refer to this topic, as we can probably slightly improve the npm install -g DX together (right now it works because Yarn workarounds a small issue with npm).

@jasnell

and can't be easily updated

Not sure what you're basing this on. The design documentation for corepack goes into detail on how both corepack and the package manager would be updated and the flow is not much more complicated than what we have currently.

I'm basing it on the fact that npm i -g npm would not update itself, because it'd refuse to clobber an executable outside of its control. If npm isn't included in corepack, and node continues to ship the real npm, then that's not an issue for npm. But it is still an issue for yarn and pnpm, and all of the extant documentation on how to get yarn and pnpm (ie, by installing it with npm i -g.)

How is this better than just shipping all three package managers,

People who do not want yarn installed should not ever need to have it.

If you are shipping a binary named yarn that automatically installs yarn from the registry the first time it's run, then they have it. You've just traded bytes in the installer with bytes over the wire later.

Shipping all three would bloat the installer image needlessly, and providing an option to download on install is really no different.

It would increase bytes, but reduce complexity. I believe the tradeoff is better. Download time is less valuable than debugging time.

This gives the additional benefit of allowing different projects to select which package manager they require, which an install time only option would not provide.

Should this just be a thing npm can do? It wouldn't be hard at all to add an npm command that will run a command with whichever package manager is declared in the project, and/or raise warnings/errors if you've specified a package manager other than npm, with instructions on how to install the package manager specified.

I've added a documentation page:

@nodejs/tsc do you think this should be considered semver major? It's an experimental feature and using yarn or pnpm is opt in, so shouldn't break any existing users. The only binary added to PATH is corepack. If it is semver-major we'll either need to rush to get it o n v15, or we'll need to wait until v16, which is not ideal as this is a feature that should be tried on an odd version first. Please chime in over the next few days.

I believe my objection is addressed, but I would like to try the PR out locally before dismissing it. I'll try to do so over the weekend. If I don't comment on the issue before Monday morning (PST) feel free to dismiss my objection.

I'll try and find time to do a more in depth review tomorrow and Monday to ensure nothing appears Semver-Major... but I do like the idea of us agreeing this can land as a Semver-Minor to give us more flexibility towards landing the change.

Can someone clarify this for me:

With this change, a package chooses its package manager, not the end user? A package manager (at some version) becomes essentially a peer dependency of a package?

Is there precedent for this in other ecosystems?

With this change, a package chooses its package manager, not the end user? A package manager (at some version) becomes essentially a peer dependency of a package?

Packages don't choose their package managers, no. Once published, they are expected to work across all package managers. Nothing changes in this regard (in fact, it'll be even more important if Yarn and pnpm become as "legitimate" as npm is).

Projects (which may be published as packages or not) do choose their package managers. For instance, if you wish to contribute to the Babel codebase, you should use Yarn, because that's what their repository is configured for. Because of this, if you wish to contribute to the Babel codebase, you better have Yarn in your environment. This is what we're aiming to improve.

(In other words: only the top-level packageManager field matters; the transitive ones from your dependencies are ignored)

so the "requiring a specific version of package manager x" applies only to a dev environment...kind of like a .npmrc or package-lock.json. ok.

How adding a manager of package managers to Node.js really makes things easier for us?

Doesn't it make things even more convoluted and distorted for Node.js users? It feels very confusing to me - another uncommon and counterintuitive tool to learn about.

Shouldn't we see a Node.js as package manager agnostic runtime in first place?

How about going following route:

1. Do not ship binaries of npm or any other packager manager with Node.js binaries and installers, but instead:
2. In installer provide a prompt asking which package manager to install. List all popular package managers, sorted by popularity, by default check the most popular one, allow to install many package managers. Installer should download and install chosen package managers.
3. In binaries provide an installation script, that will download and install a chosen package manager
4. Have specific versions of package managers bound to specific Node.js versions (so as it's today Node.js v14.13.1 in all cases installs npm v6.14.8 and no other)

Wouldn't that be simple, clear for users, and solve the problem?

@medikoo I'd really appreciate if we could keep "brainstormings" to #15244 so that this PR can focus on the implementation. This thread has already 70 comments, and if we go into design Q&A I'm concerned it would become even harder to follow.

To answer you quickly, your plan is actually not unlike what's actually implemented here. In particular, the "[binary which] provide an installation script, that will download and install a chosen package manager" is very much what Corepack itself is. The main difference is that you're suggesting to remove npm from Node, which I'm personally not interested to pursue.

The main reason why this PR has a chance to land is that it's "strictly better" than what Node currently has, enabling more workflows than exist now without hurting the existing ones. Removing Node would be a different game, because it would remove workflows. I don't believe this could ever land in the current landscape, and we need to be pragmatic.

@medikoo I'd really appreciate if we could keep "brainstormings" to #15244

Indeed, sorry for putting this out of proper context (I came here out of twitter, and didn't explore fully the path it derived from)

Do you see any remaining blocker to address before this PR can be merged?

we should def follow up ASAP with the idea of creating a Corepack WG (or bring that conversation under one of the existing ones) so that we can actually measure/evaluate the results of the experiment and in general have the right place to follow up with the many discussions brought up during the pmm/corepack proposal

I believe this will be part of what @jasnell planned for #35682

I believe this will be part of what @jasnell planned for #35682

good point! @jasnell it's unclear to me whether #35682 is just a tangential or if its intention is to replace/fulfill the need for a WG/place of discussion for corepack as raised by @wesleytodd in nodejs/admin#558

That'll be on the agenda for the summit, yes

In terms of "Do you see any remaining blocker to address before this PR can be merged?"

I think there are lots of open questions (for example all those in #35398 (comment)), and an explicit objection on the PR...

EDIT: I should probably have also said, my expectation was that upcoming summit was the place to discuss a number of those questions and make progress before this would land.

I think there are lots of open questions (for example all those in #35398 (comment)),

Most of those meta questions have been answered here or in the original thread.

an explicit objection on the PR...

Similarly, @mmarchini's requests have been addressed. As a reminder, they were to:

• move the project to the org, this is now done
• find other potential contributors, this is now done (cf @zkochan's comment here, the commits made by @merceyz, ...)
• make sure this ships as experimental somehow, this is now done, Corepack being opt-in and marked as experimental in the documentation.

From my point of view the first iteration is ready and everything in this thread has been addressed one way or another, so if you see remaining blockers I'd appreciate them being restated here under the form of concrete action items, so that we're sure we're all on the same page 😊

+1 ... I think @mmarchini's objections have been addressed. What I'd like to see at this point is for this to land in the main branch but hold off on moving it into a release until at least after we have the binary manager summit. In that regard, it would be similar to the QUIC work -- available for folks to try and experiment with if they build from source, but not delivered in a supported release until it's had time to bake a bit more.

For a change like this I personally think it should have more TSC approvals, at the very least a good number of collaborator approvals.

@jasnell with respect to "it would be similar to the QUIC work -- available for folks to try and experiment with if they build from source" is there a compile time flag that gates it being included?

Most of those meta questions have been answered here or in the original thread.

@bnb are you happy that your questions that you raised have been addressed through the references above?