[Linux] Support case-sensitive filesystems #1412

chrisd8088 · 2019-08-05T06:26:24Z

This PR looks to resolve a number of case-sensitivity issues with the current provider code and test suite, including:

Fixing case-differing references to files and path, e.g., FastFetch.dll not fastfetch.dll. This includes references to Git object files, for which the SHA1 value needs to be lowercased.
Fixing hard-coded Windows path separators in messages and tests.
Ensuring DiffHelper functions correctly on case-sensitive filesystems; among other things, this allows FastFetch to succeed.
Adding case-sensitivity to the GVFS provider's databases and file path comparisons, so that, for example, the GitCommandsTests.CaseOnlyRenameFileAndChangeBranches() functional test succeeds.

The latter's failure on a case-sensitive filesystem is due to case-insensitivity in the current modified paths database and virtualization code, among other things, which results in case-differing-only filenames being dropped from the data sent back from the GVFS provider, via the virtual-filesystem hook, to Git itself after it reads its index file.

Introduce case-sensitive file path matching and sorting of file and folder names within the repository as well as the modified paths and placeholder list databases, but only on Linux and other case-sensitive file systems, while retaining existing case-insensitive behaviour on Windows
and Mac platforms.

Use filesystem-specific case matching when comparing path throughout the functional test suite.
Also make sure to exclude the case-sensitive filesystem unit tests when running on Windows or Mac platforms.

Also fix the path matching in the HydrateEntireRepo() helper for MultiEnlistmentTests.SharedCacheTests, which was not excluding {repoRoot}/.git/ paths on Mac (or Linux) due to hard-coded Windows path separators.

Divide the HydratingFileUsesNameCaseFromRepo() and HydratingNestedFileUsesNameCaseFromRepo() functional tests
into pairs, one for case-sensitive filesystems, and one for case-insensitive filesystems.

Includes some filename case corrections to prevent test case regressions on case-sensitive filesystems, use of lowercase paths generated from SHA1 values because that ensures consistency with Git's own naming scheme for files under .git/objects on case-sensitive filesystems, and fixes for some hard-coded uses of Windows path separators.

Resolves github#29.

/cc @jrbriggs, @kivikakk.

chrisd8088 · 2019-08-05T11:29:25Z

/azp run GitHub VFSForGit Mac Functional Tests

azure-pipelines · 2019-08-05T11:29:30Z

No pipelines are associated with this pull request.

chrisd8088 · 2019-08-05T11:37:08Z

/azp run GitHub VFSForGit Mac Functional Tests

azure-pipelines · 2019-08-05T11:37:12Z

No pipelines are associated with this pull request.

derrickstolee

LGTM, but please let @kewillford or @wilbaker give their thoughts on the SQL changes before merging.

GVFS/GVFS.Common/Database/PlaceholderTable.cs

GVFS/GVFS.Common/Git/GitObjects.cs

GVFS/GVFS.Platform.Mac/MacPlatform.cs

chrisd8088 · 2019-08-08T20:35:04Z

Just to expand on our conversation today, I hope to add some further refinements in the non-common, platform-specific code (e.g., GVFS.Platform.Mac, etc.), which I think would help reduce the likelihood of re-introducing case-sensitivity issues in the common code. The PR at the moment is the base set of changes needed, but there are a few more OrdinalIgnoreCase lurking around in some of the platform-specific classes.

chrisd8088 · 2019-08-12T16:28:19Z

Looks like the sparse mode changes have added a single new test failure ... will investigate today.

chrisd8088 · 2019-08-13T01:26:18Z

/azp run PR - Windows - Functional Tests

azure-pipelines · 2019-08-13T01:26:25Z

Azure Pipelines successfully started running 1 pipeline(s).

chrisd8088 · 2019-08-13T03:34:35Z

/azp run PR - Windows - Functional Tests

azure-pipelines · 2019-08-13T03:34:43Z

Azure Pipelines successfully started running 1 pipeline(s).

On Linux, file paths are case-sensitive, so we always use lowercase paths generated from SHA1 values because that ensures consistency with Git's own naming scheme for files under .git/objects.

Make sure to use the exact case-sensitive name of FastFetch.dll in the test suite.

Simplify a few more platform-specific case-sensitivity checks in the functional tests and CmdRunner by using FileSystemHelpers, as suggested by kewillford on PR review.

Per PR suggestion from kewillford.

Per PR suggestion from wilbaker, we hoist the check on whether to perform a case-sensitive string comparison or not out of the core loop of the LazyUTF8String.Compare() method in GitIndexProjection. This should reduce the number of additional conditional tests to just one per Compare() invocation, as opposed to one per compared byte. (We could, if need be, further reduce the overhead by creating two independent Compare() methods and not passing in a bool flag, and then hoisting the comparsion-mode check higher in SortedFolderEntries.GetSortedEntriesIndexOfName() so that different versions of that method were executed depending on the mode. However, we reserve this option for now, depending on the results of performance tests.) We also add labels to the bool flags used in some of the helper methods used by the unit tests, per PR advice.

Per PR advice from wilbaker.

As wilbaker noted in PR review, we really don't need the Case[In]SensitiveFileSystem functional test categories because we've now created the public boolean flag in FileSystemHelpers, so we can just toggle case-sensitive behaviour on that flag instead of duplicating tests and moving code around. Nice catch!

wilbaker

Approving now, but please make the final minor functional tests changes discussed in the comments before merging.

chrisd8088 · 2019-08-23T16:21:39Z

As we discussed, @wilbaker, I'll put the minor functional test changes into a small followup PR. Thanks for all the help!!

On case-insensitive filesystems, we can restore the use of non-case-matching mixed-case file and folder paths in several PrefetchVerbTests, while retaining case-sensitive exact matches on Linux. These changes are a follow-up based on PR feedback from wilbaker on PR microsoft#1412 and commit 5c78902.

chrisd8088 · 2019-09-04T05:19:42Z

As a postscript, #1503 should deal with some minor leftover issues, notably the functional test revisions @wilbaker wanted, and I also captured the question of whether it would be realistically eliminate all sources of uppercase SHA1s in github#35.

Add initial Linux platform and scripts This is an initial port of the GVFS.Platform.Linux classes and the Scripts/Linux scripts from both the Mac equivalents in this repository as well as the corresponding classes in the VFSForGit repository. Note that some useful refactoring which exists in the VFSForGit features/linuxprototype branch, such as microsoft/VFSForGit@7b70850, has not yet been ported; this would eliminate some of the duplication between the Mac and Linux platforms and move common code into the POSIX one. This PR also does not yet include the case-sensitive filename support in VFSForGit's master branch from microsoft/VFSForGit#1412. Quoting from the primary VFSForGit commit microsoft/VFSForGit@b304cf7 from which this work was derived: We add the core GVFS.Platform.Linux classes, derived from their GVFS.Platform.Mac equivalents but with appropriate changes for syscall argument signatures and flag values, type definitions, and structure fields (e.g., mode_t is a uint, and the layout of struct stat is quite different).

TODO -- see VFSForGit#1503 and #1412 for more work to be refactored. TODO -- since we have no failing tests without this work, add some. Use filesystem-specific case matching when comparing path throughout Scalar and the functional test suite. From microsoft/VFSForGit#1412 and microsoft/VFSForGit#1503. From microsoft/VFSForGit@e6eb054, microsoft/VFSForGit@66e3e21, microsoft/VFSForGit@6067f04, and microsoft/VFSForGit@7100179. Co-authored-by: Ashe Connor <ashe@kivikakk.ee>

TODO -- since we have no failing tests without this work, add some. Use filesystem-specific case matching when comparing path throughout Scalar and the functional test suite. From microsoft/VFSForGit#1232, microsoft/VFSForGit#1412, and microsoft/VFSForGit#1503. From microsoft/VFSForGit@e6eb054, microsoft/VFSForGit@66e3e21, microsoft/VFSForGit@6067f04, and microsoft/VFSForGit@7100179. Co-authored-by: Ashe Connor <ashe@kivikakk.ee>

Use filesystem-specific case matching when comparing path throughout Scalar and the functional test suite. From microsoft/VFSForGit#1232, microsoft/VFSForGit#1412, and microsoft/VFSForGit#1503. From microsoft/VFSForGit@e6eb054, microsoft/VFSForGit@66e3e21, microsoft/VFSForGit@6067f04, and microsoft/VFSForGit@7100179. Co-authored-by: Ashe Connor <ashe@kivikakk.ee>

chrisd8088 added the WIP label Aug 5, 2019

chrisd8088 removed the WIP label Aug 5, 2019

chrisd8088 changed the title ~~[WIP] Support case-sensitive filesystems~~ [Linux] Support case-sensitive filesystems Aug 5, 2019

chrisd8088 requested review from wilbaker and derrickstolee Aug 5, 2019

chrisd8088 added the platform: linux label Aug 5, 2019

chrisd8088 marked this pull request as ready for review Aug 5, 2019

derrickstolee approved these changes Aug 5, 2019

View changes

GVFS/GVFS.Common/Database/PlaceholderTable.cs Show resolved Hide resolved

GVFS/GVFS.Common/Git/GitObjects.cs Outdated Show resolved Hide resolved

turbonaitis reviewed Aug 6, 2019

View changes

GVFS/GVFS.Platform.Mac/MacPlatform.cs Show resolved Hide resolved

This was referenced Aug 7, 2019

Tracking: Run VFSForGit functional tests on Linux github/VFSForGit#16

Closed

Tracking: Set up unit and functional test CI for Linux github/VFSForGit#18

Closed

chrisd8088 force-pushed the github:case-sensitive-repos branch from 655673e to d242285 Aug 7, 2019

chrisd8088 mentioned this pull request Aug 8, 2019

[Linux] enable most functional tests on Linux #1432

Merged

chrisd8088 force-pushed the github:case-sensitive-repos branch from d242285 to 0b66909 Aug 8, 2019

chrisd8088 force-pushed the github:case-sensitive-repos branch 5 times, most recently from 26ef76b to fa6915c Aug 9, 2019

chrisd8088 force-pushed the github:case-sensitive-repos branch from fa6915c to b536003 Aug 13, 2019

chrisd8088 added 9 commits Jun 13, 2019

always lowercase SHA1 filenames for Linux

7b7d581

On Linux, file paths are case-sensitive, so we always use lowercase paths generated from SHA1 values because that ensures consistency with Git's own naming scheme for files under .git/objects.

use mixed-case FastFetch.dll in functional tests

8c05870

Make sure to use the exact case-sensitive name of FastFetch.dll in the test suite.

use FileSystemHelpers in additional func tests

6067f04

Simplify a few more platform-specific case-sensitivity checks in the functional tests and CmdRunner by using FileSystemHelpers, as suggested by kewillford on PR review.

add case-sensitive filesystem DiffHelperTests test

124b6b7

Per PR suggestion from kewillford.

Add case-differing-only SortedFolderEntriesTests

2886081

Per PR suggestion from kewillford.

only down-case SHA1s on case-sensitive filesystems

d4c8260

Per PR advice from wilbaker.

fix abd8a62

e1ef0c1

chrisd8088 force-pushed the github:case-sensitive-repos branch from 0ee3268 to 7100179 Aug 23, 2019

wilbaker approved these changes Aug 23, 2019

View changes

chrisd8088 deleted the github:case-sensitive-repos branch Aug 23, 2019

chrisd8088 mentioned this pull request Sep 3, 2019

restore mixed-case prefetch folder list tests and tidy MirrorProviders #1503

Merged

alameenshah added this to the M159 milestone Sep 12, 2019

jeschu1 mentioned this pull request Sep 12, 2019

Merge milestones/M158 to releases/shipped #1515

Merged

chrisd8088 mentioned this pull request Aug 17, 2020

Add initial Linux platform and scripts microsoft/scalar#413

Merged

chrisd8088 mentioned this pull request Sep 15, 2020

use platform-specific filepath matching and add or revise relevant tests microsoft/scalar#432

Merged

microsoft / VFSForGit

[Linux] Support case-sensitive filesystems #1412

[Linux] Support case-sensitive filesystems #1412

chrisd8088 commented Aug 5, 2019 •

edited

chrisd8088 commented Aug 5, 2019

azure-pipelines bot commented Aug 5, 2019

chrisd8088 commented Aug 5, 2019

azure-pipelines bot commented Aug 5, 2019

derrickstolee left a comment

chrisd8088 commented Aug 8, 2019

chrisd8088 commented Aug 12, 2019

chrisd8088 commented Aug 13, 2019

azure-pipelines bot commented Aug 13, 2019

chrisd8088 commented Aug 13, 2019

azure-pipelines bot commented Aug 13, 2019

wilbaker left a comment

chrisd8088 commented Aug 23, 2019

chrisd8088 commented Sep 4, 2019

microsoft / VFSForGit

Join GitHub today

GitHub is where the world builds software

[Linux] Support case-sensitive filesystems #1412

[Linux] Support case-sensitive filesystems #1412

Conversation

chrisd8088 commented Aug 5, 2019 • edited

chrisd8088 commented Aug 5, 2019

azure-pipelines bot commented Aug 5, 2019

chrisd8088 commented Aug 5, 2019

azure-pipelines bot commented Aug 5, 2019

derrickstolee left a comment

chrisd8088 commented Aug 8, 2019

chrisd8088 commented Aug 12, 2019

chrisd8088 commented Aug 13, 2019

azure-pipelines bot commented Aug 13, 2019

chrisd8088 commented Aug 13, 2019

azure-pipelines bot commented Aug 13, 2019

wilbaker left a comment

chrisd8088 commented Aug 23, 2019

chrisd8088 commented Sep 4, 2019

Essential cookies

Always active

Analytics cookies

chrisd8088 commented Aug 5, 2019 •

edited