Skip to content

Swift: generalize open redirection on both platforms and rework output rewriting#11571

Merged
redsun82 merged 24 commits into
mainfrom
redsun82/swift-open-redirection
Jan 17, 2023
Merged

Swift: generalize open redirection on both platforms and rework output rewriting#11571
redsun82 merged 24 commits into
mainfrom
redsun82/swift-open-redirection

Conversation

@redsun82
Copy link
Copy Markdown
Contributor

@redsun82 redsun82 commented Dec 5, 2022

The hash map mechanism that was already in use for reading swiftmodule
files on macOS is now in use also on Linux. The output replacing
mechanism has been also reworked so that the remapping is done directly in
the internal frontend options instead of painstakingly modifying input flags.

All outputs that are not swiftmodule files are either removed or redirected to /dev/null

The open file read redirection uses a global weak pointer instance to
maximize robustness in the face of possibly multi-threaded calls to open
happening while main is exiting. Possibly overkill, but better safe
than sorry.

@github-actions github-actions Bot added the Swift label Dec 5, 2022
@redsun82 redsun82 force-pushed the redsun82/swift-open-redirection branch 9 times, most recently from 41b6d2a to 1b48dbf Compare December 8, 2022 10:59
@redsun82 redsun82 changed the title Swift: generalize open redirection on both platforms and on outputs Swift: generalize open redirection on both platforms and rework output rewriting Dec 8, 2022
The hash map mechanism that was already in use for reading swiftmodule
files on macOS is now in use also on Linux. The output replacing
mechanism has been also reworked so that:
* frontend module emission modes have the remapping done directly in
  the internal frontend options instead of painstakingly modifying input
  flags (this requires a patch on the swift headers though)
* object emission mode is silenced to be just a type checking pass,
  thus producing no output files
* all other passes but some debugging and version related ones become
  noops

The open file read redirection uses a global weak pointer instance to
maximize robustness in the face of possibly multi-threaded calls to open
happening while `main` is exiting. Possibly overkill, but better safe
than sorry.
@redsun82 redsun82 force-pushed the redsun82/swift-open-redirection branch from be5e084 to fd2d16c Compare December 8, 2022 15:11
Downgrading the emit object action to a type check one has some
unexpected side effects, that seem however acceptable:
* experimental false static assertions do not make compilation fail in
  type check mode
* the implicit module loading of `SwiftOnoneSupport` is not happening.
  That module contains some "pre-specializations", it does not seem
  really relevant for analysis
@redsun82 redsun82 force-pushed the redsun82/swift-open-redirection branch from fd2d16c to 26ae8f1 Compare December 8, 2022 16:13
@redsun82 redsun82 marked this pull request as ready for review December 8, 2022 16:29
@redsun82 redsun82 requested review from a team as code owners December 8, 2022 16:29
@redsun82 redsun82 requested a review from AlexDenisov December 8, 2022 16:29
@AlexDenisov
Copy link
Copy Markdown
Contributor

Tested it locally against two big projects - works great! Let's wait for DCA and merge?

@redsun82
Copy link
Copy Markdown
Contributor Author

redsun82 commented Dec 9, 2022

Tested it locally against two big projects - works great! Let's wait for DCA and merge?

Thanks for that! I aborted the latest expriment as I wanted to add 7162692 in before that. I'll be restarting the experiment right away.

I wonder whether this has a good impact on performance, as we probably avoid a significant amount of compiler work duplication with this.

@redsun82 redsun82 force-pushed the redsun82/swift-open-redirection branch from 342c87b to 208388e Compare December 15, 2022 12:30
Copy link
Copy Markdown
Contributor

@andersfugmann andersfugmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work on unifying artifact handling across platforms!
Just a drive by commenting - I did not do a full review.

Comment thread swift/extractor/remapping/SwiftFileInterception.cpp Outdated
Comment thread swift/extractor/remapping/SwiftFileInterception.cpp Outdated
Comment thread swift/extractor/remapping/SwiftFileInterception.cpp Outdated
Copy link
Copy Markdown
Contributor

@andersfugmann andersfugmann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great work on unifying artifact handling across platforms!
Just a drive by commenting - I did not do a full review.

File hashing is now done internally in `SwiftFileInterception` (and
exported as a `getHashOfRealFile` function for future use in linkage
awareness), and using a per-process in-memory cache. The persistent
caching of paths is removed, so the solution is now robust against input
file changes during the build.

For the same reason, the hash to artifact mapping have the symlinks
reversed now. The artifacts themselves are stored using the hash as
filenames, and the original paths of the artifacts are reacreated in the
scratch dir with symlinks mostly for debugging purposes (to understand
what artifact each hash corresponds to, and to follow what was built by
the extractor).
Copy link
Copy Markdown
Contributor

@AlexDenisov AlexDenisov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@redsun82 redsun82 merged commit d9bd41b into main Jan 17, 2023
@redsun82 redsun82 deleted the redsun82/swift-open-redirection branch January 17, 2023 14:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants