I'd suggest we rename the local variables here to just sslKeyOK and sslCertOK, since they're not specific to the host name any more.

The same is also true in the getRootCAsForURLFromGitconfig() function below (i.e., sslKey and sslCert).

One nice thing is that it's an opportunity to more closely adhere to Google's Go naming style guidelines for initialisms, so for example, we can replace hostSslKeyOk with sslKeyOK and keep the cases of ssl and OK internally consistent.

Let's change this trace log message to include the full URL used to look for a client TLS certificate, e.g.:

We might as well also use %q here, just to make the output a bit tidier.

Note that when we make this change, there will be no need to initialize the local host variable at all in this method, so we can just drop that line.

In my previous comment I alluded to our development history and how it's left us with a slightly confusing and redundant code design:

... the current state of affairs likely stems from an incomplete original implementation from PRs #1787 and #1839 which was then incompletely migrated in PR #2160 to our full emulation of Git's matching rules for http.<url>.* options when that was added in PRs #1912 and #3392.

I've expanded on the details below, and I'll put notes on the whole tangled history in a comment, but here's what I think we should do:

We can then remove the SkipSSLVerify field from the Client structure entirely, as this function is the only place it's used, and we don't need it here at all if we rewrite the function as in my suggestion above.

That does mean we also have to move and refactor the two Go tests in the lfshttp/client_test.go source file.

As well, there's another test in the lfshttp/client_test.go source file which sets multiple http.<url>.sslVerify options, but so far as I can tell, these are unnecessary and can be removed. Other tests just set the http.sslVerify option only, and given the limitation this PR addresses, the extra http.<url>.sslVerify options in the TestClientRedirect() test function would never have had any effect. (The TestClientRedirect() function was added first, in PR #2235, while the TestHttp2() and TestHttpVersion() functions were added later.)

Git Implementation

It's worth reviewing how the precedence and parsing of the http.sslVerify/http.<url>.sslVerify option and the GIT_SSL_NO_VERIFY environment variable are implemented by Git.

First, URL matching rules are applied to look for the most-specific variant of the http.sslVerify/http.<url>.sslVerify options. These options are expected to have Boolean values, which are parsed such that the terms false, no, and off, along with the integer 0, all count as false, while their opposite terms and all non-zero integers count as true.

Next, if the GIT_SSL_NO_VERIFY environment variable is defined at all, with any value (even an empty string), this overrides the configuration option settings with a false value. Finally, if no option and no environment variable were found, the default value used is true.

Current Git LFS Implementation

Our implementation is different from Git's in a few respects, some of which stem from the early development history of our project.

In commit 8009d17 of PR #1787 (which was part of a larger set of PRs that were all merged into the main development branch in PR #1839), we implemented a simplified version of Git's URL-matching logic for http.<url>.* configuration options, which just searched for an http.https://<host>/.sslVerify option.

There was already a discrepancy in this initial version of the code between how the generic http.sslVerify option and the URL-specific variant of that option were treated. The former was parsed using the environment.Bool() method from our config package, which recognizes a number of possible values like 0 and false as being equivalent to false, whereas the latter was retrieved with the environment.Get() method and then checked only for the value false. The following example illustrates how this discrepancy remains the current code:

$ git init test
$ cd test
$ git lfs track "*.bin"
$ echo foo >foo.bin
$ git add .gitattributes foo.bin
$ git commit -m foo

$ git remote add origin https://expired.badssl.com/test/test.git
$ git lfs push --object-id origin b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c                
batch response: Post "https://expired.badssl.com/test/test.git/info/lfs/objects/batch": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: “*.badssl.com” certificate is expired

# "false" is accepted for both "http.sslVerify" and "http.https://<host>/.sslVerify"
$ git -c http.sslVerify=false lfs push --object-id origin b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
batch response: Repository or object not found: https://expired.badssl.com/test/test.git/info/lfs/objects/batch
$ git -c http.https://expired.badssl.com/.sslVerify=false lfs push --object-id origin b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c 
batch response: Repository or object not found: https://expired.badssl.com/test/test.git/info/lfs/objects/batch

# "0" is accepted only for "http.sslVerify"
$ git -c http.sslVerify=0 lfs push --object-id origin b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c
batch response: Repository or object not found: https://expired.badssl.com/test/test.git/info/lfs/objects/batch
$ git -c http.https://expired.badssl.com/.sslVerify=0 lfs push --object-id origin b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c                     
batch response: Post "https://expired.badssl.com/test/test.git/info/lfs/objects/batch": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: “*.badssl.com” certificate is expired

So that's one problem: we use our config.Bool() function to parse only the generic version of the http.sslVerify / http.<url>.sslVerify option, whereas for the URL-specific version we only accept false as a possible value, which doesn't match Git's implementation.

(As a sidebar, our config.Bool() function doesn't parse Boolean values in exactly the same way as Git, so for instance Git treats any non-zero integer as true, but we only accept the integer value 1 as equivalent to true. But this is a discrepancy we should postpone resolving to another, future PR.)

Another discrepancy pertains to how we handle the GIT_SSL_NO_VERIFY environment variable. If the variable is defined with any value, even the empty string, then Git does not perform TLS/SSL certificate validation. As the Git documentation states:

Setting and exporting this environment variable to any value tells Git not to verify the SSL certificate when fetching or pushing over HTTPS.

When we first introduced support for the GIT_SSL_NO_VERIFY variable, in commit f83b028 of PR #145, we accepted any non-empty value of the variable as indicating that certificate validation should be disabled. That was almost equivalent to Git's implementation, except for the case of an empty string.

However, in commit 410830a of PR #401 we began parsing the value of the GIT_SSL_NO_VERIFY variable as a Boolean field, which is quite different from how Git interprets it, we continue to mis-parse this variable as a Boolean field today.

Finally, there's a lesser issue that we currently often read the value of the http.sslVerify option twice.

We parse it first when we set the value of the SkipSSLVerify field of our lfshttp.Client structure. That design dates back to our original introduction of support for this configuration option in PRs #1787 and #1839.

Later, in commit 9a17bd2 of PR #2160, we started using the initial version of our config.URLConfig structure's Get() method to retrieve the value of an http.https://<host>/.sslVerify option if one was set. Because we still constructed the <url> part from just a <host> value, this change was largely ineffective at properly supporting the full range of possible http.<url>.sslVerify options. But one consequence was that we now often read the value of the generic version of the http.sslVerify / http.<url>.sslVerify option a second time, because when no host-specific version matching http.https://<host>/.sslVerify is found the Get() method falls back to looking for the generic version of the option.

Revised Git LFS Implementation

With all that said, I believe my proposed revision to the isCertVerificationDisabledForURL() function makes the following corrections and refinements:

1. We check the environment variable using the os.LookupEnv() function, and if the variable has any defined value at all, we interpret that to mean we should not perform TLS/SSL certificate verification (i.e., the isCertVerificationDisabledForURL() function should return true). This allows us to skip the more complicated URL-matching phase entirely if the variable is defined, and corrects our historical mis-parsing of the variable.

2. If the environment variable isn't defined, then we use the Bool() method of our config.URLConfig structure to look for the best-matching http.sslVerify/http.<url>.sslVerify option, passing a default value of true in case none of these options are defined. This avoids any extra lookups of the generic version of the option and allows us to fully support Boolean values for the URL-specific version of the option.

These two new TestRootCAsPathScoped() and TestRootCAsRespectsScheme() functions both exercise the getRootCAsForURLFromGitconfig() function, which is what the first six test functions in this source file also validate, so at a minimum we should order them together with those existing functions.

But I think we can also consolidate all these test functions and also expand the set of checks we perform. That's going to require quite a large refactoring of the tests, though.

We can simplify this section by using the URLConfig.Bool() method:

I think we should probably just put these tests in our t/t-config.sh file, as they pertain to the validation of how we read and apply configuration settings.

We apparently have never supported http.<url>.sslCAPath configuration options, even though Git does.

Like some of our issues with the http.sslVerify/http.<url>.sslVerify option, this problem goes back to the original implementation in commit 8009d17 of PRs #1787 and #1839.

If that was the only issue, I would recommend just the following change:

 		return appendCertsFromFilesInDir(pool, cadir)
 	}
 	// http.sslcapath
-	if cadir, ok := c.gitEnv.Get("http.sslcapath"); ok {
+	if cadir, ok := uc.Get("http", rawurl, "sslcapath"); ok {
 		return appendCertsFromFilesInDir(pool, cadir)
 	}

However, I believe there's another couple of problems here as well.

First, we treat the GIT_SSL_CAINFO environment variable and the http.sslCAInfo / http.<url>.sslCAInfo option as if they should override the GIT_SSL_CAPATH environment variable and the http.sslCAPath / http.<url>.sslCAPath option.

And second, we also ignore the GIT_SSL_CAPATH environment variable and the http.sslCAPath / http.<url>.sslCAPath option if the http.sslBackend / http.<url>.sslBackend option is set to schannel and the http.schannelUseSSLCAInfo / http.<url>.schannelUseSSLCAInfo option is not set to true.

However, that's not how Git handles these options. Regardless of any other configuration settings, Git sets the libcurl library's CURLOPT_CAPATH option from the GIT_SSL_CAPATH environment variable or the http.sslCAPath / http.<url>.sslCAPath configuration option.

Then, if the http.sslBackend / http.<url>.sslBackend option is not set to schannel or the http.schannelUseSSLCAInfo / http.<url>.schannelUseSSLCAInfo option is set to true, Git sets the libcurl library's CURLOPT_CAINFO option from the GIT_SSL_CAINFO environment variable or the http.sslCAInfo / http.<url>.sslCAInfo option.

In other words, the "certificate file" (i.e., "info") and "certificate path" options are not incompatible, but we treat them as such right now, and let the "info" setting override any "path" setting. In particular, if either is set, the libcurl library passes both through to the OpenSSL library (assuming that's the SSL/TLS backend in use) via the X509_STORE_load_file() and X509_STORE_load_path() functions. These either directly or indirectly invoke the X509_STORE_add_lookup() function, which pushes the appropriate lookup function into the list of locations to check for valid certificates.

Note that when Git reads these environment variables, even an empty string value is accepted, while we ignore empty strings, which is another discrepancy between our implementation and Git's.

Note, too, that the CURLOPT_CAPATH option is documented to have no effect on Windows due to an OpenSSL limitation (which is likely why there's no http.schannelUseSSLCAPath option in Git). But we read certificates from the path specified by the GIT_SSL_CAPATH environment variable or the http.sslCAPath / http.<url>.sslCAPath option and parse them even on Windows, so we should correct that inconsistency.

All of which is to say that I think we need to rewrite this whole section to look something like this:

	if runtime.GOOS != "windows" {
		// GIT_SSL_CAPATH overrides http.sslCAPath/http.<url>.sslCAPath
		if cadir, ok := c.osEnv.Get("GIT_SSL_CAPATH"); ok {
			pool = appendCertsFromFilesInDir(pool, cadir)
		} else if cadir, ok := uc.Get("http", rawurl, "sslcapath"); ok {
			pool = appendCertsFromFilesInDir(pool, cadir)
		}
	}

	backend, _ := uc.Get("http", rawurl, "sslbackend")
	schannelUseSslCaInfo := uc.Bool("http", rawurl, "schannelusesslcainfo", false)

	if backend == "schannel" && !schannelUseSslCaInfo {
		return pool
	}

	// GIT_SSL_CAINFO overrides http.sslCAInfo/http.<url>.sslCAInfo
	if cafile, ok := c.osEnv.Get("GIT_SSL_CAINFO"); ok {
		pool = appendCertsFromFile(pool, cafile)
	} else if cafile, ok := uc.Get("http", rawurl, "sslcainfo"); ok {
		pool = appendCertsFromFile(pool, cafile)
	}

Since we want to to ensure that the certificate and key files don't exist, I think we want to use paths which we can guarantee are unique and local to the tests, rather than using paths which are just unlikely to exist on a system.

Admittedly, it's very unlikely that a system would have a root-level directory named nonexistent with those particular files in it, but partly for the sake of consistency with other tests which check that files don't exist, I'd suggest we do:

We should make the same change to the similar lines in the subsequent test as well, of course.

This is an interesting case, because if the git lfs push command fails, we'd ideally like to check if it did so specifically because it tried to load the nonexistent certificate file, rather than for some other spurious reason (e.g., because the test server had crashed).

A common pattern in our shell tests is to check that a command failed and then check that a specific message appears in the logged output from that command.

We certainly also have lots of instances in our shell tests where we check that a log file does not contain a given string, but we normally do this along with various "positive" assertions, such as that the log file does contain other strings or that a command succeeded. Occasionally, though, these may even follow a check that a command has failed.

What's unusual about this situation, though, is that we don't have any such positive assertions, and that if the command does unexpectedly fail, we'd like to check for a specific error message first before checking the command's exit status. Otherwise we're making an assumption that the command failed for a specific reason which may not actually be the case.

So I'd suggest we adopt a modified version of a pattern that turns up in a few other shell tests, where we save the exit status from the command into a variable, then check the log for the a specific message, and then check if the exit status. Something like this:

So first we confirm that there are zero instances of the string Error reading client cert file in the log file, and then we confirm that the command returned successfully with an exit status of zero.

One detail I should perhaps mention here is that we don't need to explicitly output the push.log file in the case of an error, because our test framework, combined with the use of the tee(1) command, will do that for us. In fact, that's why we use the tee command: we get a log file local to the test which it can search for specific patterns, but the command's output is also written to stdout, which our test harness captures. If a test fails, then the harness outputs those captured logs (which are separate from any local log file created within the test).

So in the case of this test, with my suggested changes above, if it fails then we get diagnostic test output like the following, where the stderr section indicates that the grep(1) command failed and the stdout section includes the full output from the git lfs push command:

# -- stdout --
...
#     batch response: Error reading client cert file "/.../git-lfs_TEMP.9O8m3R7c6R/t-config.sh-72750/nonexistent/cert.pem": open /.../git-lfs_TEMP.9O8m3R7c6R/t-config.sh-72750/nonexistent/cert.pem: no such file or directory
#     Uploading LFS objects:   0% (0/1), 0 B | 0 B/s, done.
# -- stderr --
...
#     + git lfs push origin main
#     + tee push.log
#     + res=2
#     ++ grep -c -F 'Error reading client cert file' push.log
#     + '[' 0 -eq 1 ']'
#     + test_status=1

My only suggestion here is that we could add the -F option to the grep(1) command since we're just searching for a fixed string rather than a pattern.

Admittedly, this is an optimization we neglect to use in many of our shell tests, but it's something we might as well include whenever we can going forward.