Skip to content

Use ncc and remove node_modules#70

Closed
robertbrignull wants to merge 5 commits into
mainfrom
ncc
Closed

Use ncc and remove node_modules#70
robertbrignull wants to merge 5 commits into
mainfrom
ncc

Conversation

@robertbrignull
Copy link
Copy Markdown
Contributor

@robertbrignull robertbrignull commented Jun 18, 2020

This is a prototype of using https://www.npmjs.com/package/@zeit/ncc to bundle all of our code into a js single file per action, instead of having to ship everything. This means we can remove the lib and node_modules directories from the repository.

Ncc is essentially a wrapper around webpack, but it provides a nicer interface for node projects. Using webpack directly would also be an option, but it would require slightly more setup. If it turns out ncc doesn't offer us enough flexibility then I wouldn be opposed to that.

The diff is a bit big for GitHub to display, but here's a summary of what has changed (excluding the node_modules and lib directories).

> git diff --stat master . ':!node_modules' ':!lib'
 .github/workflows/pr-checks.yml             |   30 +-
 .gitignore                                  |    2 +
 analyze/action.yml                          |    2 +-
 analyze/lib/README.md                       |    1 +
 analyze/lib/index.js                        |    2 +
 analyze/lib/index.js.map                    |    1 +
 analyze/lib/sarif_v2.1.0_schema.json        | 3350 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 autobuild/action.yml                        |    2 +-
 autobuild/lib/README.md                     |    1 +
 autobuild/lib/index.js                      |    2 +
 autobuild/lib/index.js.map                  |    1 +
 init/action.yml                             |    2 +-
 init/lib/README.md                          |    1 +
 init/lib/index.js                           |    2 +
 init/lib/index.js.map                       |    1 +
 init/lib/inject-tracer.ps1                  |    9 +
 src/tracer-env.ts => init/lib/tracer-env.js |    9 +-
 package-lock.json                           |   12 +-
 package.json                                |   13 +-
 src/setup-tracer.ts                         |    4 +-
 src/tracer-env.js                           |   18 +
 upload-sarif/action.yml                     |    2 +-
 upload-sarif/lib/README.md                  |    1 +
 upload-sarif/lib/index.js                   |    2 +
 upload-sarif/lib/index.js.map               |    1 +
 upload-sarif/lib/sarif_v2.1.0_schema.json   | 3350 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 26 files changed, 6773 insertions(+), 48 deletions(-)

Some things to note:

  • The setup-tracer.js file wasn't being compiled because it isn't executed in the normal way but is instead passed to CodeQL. To get around this I've converted it to javascript manually.
  • Those sarif_v2.1.0_schema.json and other files have been copied from src to the output directories. I view this is a bit unnnecessary and annoying but I haven't found a nice way to stop it. We could probably change the place where we reference this file to comfuse the bundler, but that seems counterproductive. In fact for setup-tracer.js and inject-tracer.ps1 I modified the code so the builder would recognise the reference and copy the file.
  • There's going to be a bit of duplicate between the various index.js files, but this keeps the entrypoints separate so we can just run the file. We could probably change this if we wanted so there's only one file.
  • The -m arguments means the index.js files are minimised. This isn't really necessary or hugely helpful other than making our repository a bit smaller.
  • The -s argument means we're generating source maps, but I explicitly pass --no-source-map-register to avoid generating sourcemap-register.js files. I haven't seen these files before, and googling didn't help me find out what they are for, so I disabled them.
  • Unfortunately ava doesn't support running typescript files directly and you have to pre-compile them and then run ava of the js files. Therefore I've added a call to tsc when running the tests. This means the lib directory is still getting used, but only for the tests, and it is excluded from the repository. It also slows down the tests a bit but for me they take 11 seconds which is hopefully still acceptable.

Merge / deployment checklist

  • Run test builds as necessary. Can be on this repository or elsewhere as needed in order to test the change - please include links to tests in other repos!
    • CodeQL using init/analyze actions
    • 3rd party tool using upload action
  • Confirm this change is backwards compatible with existing workflows.
  • Confirm the readme has been updated if necessary.

@alexkappa
Copy link
Copy Markdown
Contributor

This is awesome @robertbrignull! I am obviously a big fan of this change 😃

Perhaps as a next step we could even compile a single "binary" which all actions could use as their runs.main and control what command to run by means of an input argument. For example:

name: 'CodeQL: Init'
description: 'Setup the CodeQL tracer'
author: 'GitHub'
inputs:
  tools:
    description: URL of CodeQL tools
    required: false
    default: https://github.com/github/codeql-action/releases/download/codeql-bundle-20200601/codeql-bundle.tar.gz
  languages:
    description: The languages to be analysed
    required: false
  token:
    default: ${{ github.token }}
  matrix:
    default: ${{ toJson(matrix) }}
  config-file:
    description: Path of the config file to use
    required: false
+ command:
+   default: "init"
runs:
  using: 'node12'
- main: '../lib/setup-tracer.js'
+ main: '../lib/main.js'

../lib/main.js would then have to multiplex the command input and call the right function from there...

Still setup-tracer.js would remain a special case I suppose.

Thoughts? Btw, I would love to help in any way I can.

@sampart sampart changed the base branch from master to main June 23, 2020 08:55
@chrisgavin
Copy link
Copy Markdown
Contributor

Doesn't the fact that all the code gets bundled into a single minified file make conflicts much more likely?

@robertbrignull
Copy link
Copy Markdown
Contributor Author

Perhaps as a next step we could even compile a single "binary" which all actions could use as their runs.main and control what command to run by means of an input argument.

@alexkappa, I like the idea of only having one "binary" that we compile. I thought of a different way of then hooking into it by manually writing simple js files like

require('main.js').init()

and

require('main.js').analyze()

I'm not really sure which is best. I find putting the command as an input to the action a little odd as it could produce odd behaviour if a user changed it, and we might be obligated to document what it is. However introducing extra files is also a bit annoying.

Doesn't the fact that all the code gets bundled into a single minified file make conflicts much more likely?

@chrisgavin, yes, you're right it'll increase the chance of conflicts in the minified code. If we bundle all the files but don't minify them that could reduce it. I think we'd still get conflicts in the map file though almost every time. Fixing these conflicts is not hard to do but it is a pain as you'd have to do it locally and not by clicking a button in the UI. I can't think of a way around this if we want to keep the map files.

@alexkappa
Copy link
Copy Markdown
Contributor

require('main.js').init()

Totally agree, this is much cleaner. It will also enable us to test with more flexibility as we won't have that pesky run() call at the end of certain files.

@robertbrignull
Copy link
Copy Markdown
Contributor Author

Changes made include:

  • Compile all code to a single file that contains logical entrypoints to all of the actions. The physical entrypoint to each action is then a very simple file that calls into this single compiled file.
  • Don't minimise the generated javascript, to avoid getting conflicts on every single change. I worry it will probably still happen because the map file is all on one line, but we could try it and see how bad it is.

@robertbrignull
Copy link
Copy Markdown
Contributor Author

I think I've addressed all comments and this is ready for a proper review and discussion again.

So far I think the only outstanding negative point is that we'll likely get conflicts on every code change that will require running commands locally to fix.

@robertbrignull
Copy link
Copy Markdown
Contributor Author

I'm going to close this issue as there isn't enough support to follow through on this and outweigh the downsides.
If we decide we want to do this in the future then this PR will still be here if we want to use it as a starting point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants