CI Build Updates by ethomson · Pull Request #5308 · libgit2/libgit2

ethomson · 2019-11-23T01:49:08Z

Our CI builds are currently flaky. Some of these issues are related to the upgrade to xenial for our Linux build and test runs. Some of them snuck in because that move destabilized our CI and we didn't catch problems.

Here are some issues:

valgrind is reporting errors that it should be suppressing. This is because our valgrind suppression is suppressing a number of errors in symbols in libssh2.so, but we're currently statically linking against libssh2. Thus the suppression file can't match those symbols and we see a number of valgrind errors.
valgrind errors aren't failing the builds. The version of valgrind that we were using from the PPA is broken and --exit-errorcode is ignored. 😳
OpenSSL has several new issues that need to be suppressed. It appears that this is an interaction between the versions of OpenSSL, gcc and valgrind that we have on the build/test image reporting a false positive for uninitialized memory. As best I can tell, valgrind misses some subtle memory initializations with some gcc optimizations. Amusingly, these are only noticed when talking to Azure Repos. Presumably the Windows TLS stack or the ciphers in use are sufficiently different for valgrind to notice these. (But only sometimes, sigh.)

Fixes:

Build a shared library during the libssh2 compilation step so that we can target suppressions against symbols in libssh2.so.
Build the latest version of valgrind instead of using the PPA, so that --exit-errorcode works again.
Quiet valgrind down.
i. Extend the suppressions for valgrind to include uninitialized usage stemming from OpenSSL.
ii. Mark memory returned from OpenSSL as initialized explicitly.

In addition, there are a few build process enhancements:

Provide a default CMake generator in the build script. It explicitly passes -G, so it needs a default to supply if none was given in the environment.
Show the distribution information again.
Break the dockerfile up into stages. This allows it to cache stages and re-use unchanged ones, to prevent (eg) long apt installs that don't actually change the dependencies. Now docker build will pick up again from the first modified stage, re-using the unchanged stages from the cache.
Don't delete the apt cache. This aides in local debugging, so that you can apt-get install ... tools in the container. This bloats the image somewhat, but since we don't store it in a registry, this is of limited pain.
Cache docker layers in Azure Pipelines. This prevents us from needing to rebuild the entire image on every build.

Provide a sane default for `CMAKE_GENERATOR` in the build script so that it can be invoked without having to set that in the environment.

The lsb-release command is missing on our images; just show the information from the file instead of relying on it.

libcrypto will read uninitialized memory as entropy. Suppress warnings from this behavior.

ethomson · 2019-11-23T08:36:53Z

It gets better: the version of valgrind on the Ubuntu Xenial images has a broken --error-exitcode.

root@40e03a14632b:/tmp# valgrind -q --leak-check=full --show-reachable=yes --error-exitcode=2 --suppressions=/src/script/valgrind.supp  ./foo
==7019== 1,024 bytes in 1 blocks are definitely lost in loss record 1 of 1
==7019==    at 0x4C2DB4F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so)
==7019==    by 0x40059E: main (in /tmp/foo)
==7019==
root@40e03a14632b:/tmp# echo $?
0

Sigh.

Deleting the apt cache can be helpful for reducing the size of a container, but since we don't push it anywhere, it only hinders our ability to debug problems while working on the container. Keep it.

Use a multi-stage docker build so that we can cache early stages and not need to download the apt-provided dependencies during every build (when only later stages change).

The valgrind in the PPA is broken and ignores `--exit-errorcode`. Build and install our own.

Our docker builds are getting expensive, let's cache some of this.

We currently talk to Azure Repos for executing an online test (online::clone::path_whitespace). Add a simpler test to talk to Azure Repos to make it obvious that strange test failures are not likely the whitespace in the path, but actually a function of talking to Azure Repos itself.

valgrind will warn that OpenSSL will use undefined data in connect/read when talking to certain other TLS stacks. Thankfully, this only seems to occur when gcc is the compiler, so hopefully valgrind is just misunderstanding an optimization. Regardless, suppress this warning.

Provide usage hints to valgrind. We trust the data coming back from OpenSSL to have been properly initialized. (And if it has not, it's an OpenSSL bug, not a libgit2 bug.) We previously took the `VALGRIND` option to CMake as a hint to disable mmap. Remove that; it's broken. Now use it to pass on the `VALGRIND` definition so that sources can provide valgrind hints.

ethomson · 2019-11-27T03:22:17Z

Merging this so that CI is reliable again.

pks-t

Thanks a lot for these changes, they all look good to me. I'm always baffled by the amount of broken stuff we're dealing with when using Ubuntu. Broken libssh2 as it does produce memory leaks and other stuff by default, non-working valgrind options, a whole lot of suppressions...
Anyway, thanks for taking care of these issues!

pks-t · 2019-11-28T11:28:51Z

+    path: /tmp/dockercache
+- script: |
+    if [ -f /tmp/dockercache/${{parameters.docker.image}}.tar ]; then docker load < /tmp/dockercache/${{parameters.docker.image}}.tar; fi
+  displayName: 'Load Docker cache'


🎉 🎉 🎉 Great change

ethomson added 3 commits November 23, 2019 10:58

ci: provide a default for xcode generator

91ba65a

Provide a sane default for `CMAKE_GENERATOR` in the build script so that it can be invoked without having to set that in the environment.

ci: show distribution information

767990e

The lsb-release command is missing on our images; just show the information from the file instead of relying on it.

valgrind: suppress uninitialized reads in libcrypto

5dc1be8

libcrypto will read uninitialized memory as entropy. Suppress warnings from this behavior.

ethomson force-pushed the ethomson/cifix branch 7 times, most recently from 2c16b20 to 2ceeb9d Compare November 23, 2019 12:01

ethomson added 8 commits November 24, 2019 16:49

ci: don't install libssh2 since we build it

f592c73

ci: don't delete the apt cache

7a3d04d

Deleting the apt cache can be helpful for reducing the size of a container, but since we don't push it anywhere, it only hinders our ability to debug problems while working on the container. Keep it.

ci: break dockerfile into stages

8480788

Use a multi-stage docker build so that we can cache early stages and not need to download the apt-provided dependencies during every build (when only later stages change).

ci: build shared libssh2

fd83127

valgrind: suppress kexinit leaks

7adc32d

ci: build our own valgrind

c64b7aa

The valgrind in the PPA is broken and ignores `--exit-errorcode`. Build and install our own.

valgrind: suppress libssh2_rsa_sha1_sign leaks

6df3ec4

ci: cache docker layers

b8e00b9

Our docker builds are getting expensive, let's cache some of this.

ethomson force-pushed the ethomson/cifix branch from 61469fc to 6dd6742 Compare November 24, 2019 05:49

ethomson force-pushed the ethomson/cifix branch from 6dd6742 to eb3e31c Compare November 24, 2019 06:03

ethomson added 3 commits November 24, 2019 17:29

ci: enable the VALGRIND flag on builds

c863b3c

ethomson force-pushed the ethomson/cifix branch from eb3e31c to c863b3c Compare November 24, 2019 06:39

ethomson mentioned this pull request Nov 25, 2019

diff: complete support for git patchid #5306

Merged

ethomson merged commit 7805122 into master Nov 27, 2019

pks-t reviewed Nov 28, 2019

View reviewed changes

ethomson deleted the ethomson/cifix branch February 2, 2020 13:07

snyk-bot mentioned this pull request Feb 23, 2020

[Snyk] Upgrade nodegit from 0.4.1 to 0.26.4 saurabharch/Breezeblocks#1

Open

snyk-bot mentioned this pull request Mar 12, 2020

[Snyk] Upgrade nodegit from 0.26.0 to 0.26.4 thisconnect/nodegit-kit#95

Merged

snyk-bot mentioned this pull request Apr 22, 2020

[Snyk] Upgrade nodegit from 0.24.3 to 0.26.5 aminatakonate000/Graviton-App#4

Open

snyk-bot mentioned this pull request May 5, 2020

[Snyk] Upgrade nodegit from 0.24.3 to 0.26.5 Barnstorm-Online/ngp-openapi-generator#1

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

CI Build Updates#5308

CI Build Updates#5308
ethomson merged 15 commits into
masterfrom
ethomson/cifix

ethomson commented Nov 23, 2019 •

edited

Loading

Uh oh!

ethomson commented Nov 23, 2019

Uh oh!

ethomson commented Nov 27, 2019

Uh oh!

pks-t left a comment

Uh oh!

pks-t Nov 28, 2019

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

ethomson commented Nov 23, 2019 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ethomson commented Nov 23, 2019

Uh oh!

ethomson commented Nov 27, 2019

Uh oh!

pks-t left a comment

Choose a reason for hiding this comment

Uh oh!

pks-t Nov 28, 2019

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

ethomson commented Nov 23, 2019 •

edited

Loading