Skip to content

Refactor packfile code to use zstream abstraction#5340

Merged
ethomson merged 3 commits into
libgit2:masterfrom
pks-t:pks/pack-zstream
Jan 9, 2020
Merged

Refactor packfile code to use zstream abstraction#5340
ethomson merged 3 commits into
libgit2:masterfrom
pks-t:pks/pack-zstream

Conversation

@pks-t
Copy link
Copy Markdown
Member

@pks-t pks-t commented Dec 14, 2019

While investigating #5324, I noticed that the zlib-related code in packfile looks mighty complicated and could use some love. I thus refactored it to use our own git_zstream abstraction, which is a much nicer interface.

@pks-t
Copy link
Copy Markdown
Member Author

pks-t commented Jan 9, 2020

/rebuild

@libgit2-azure-pipelines
Copy link
Copy Markdown

Sorry @pks-t, an error occurred while trying to requeue the build.

@pks-t pks-t force-pushed the pks/pack-zstream branch from 4f17339 to e0cf8f9 Compare January 9, 2020 09:56
pks-t added 3 commits January 9, 2020 12:22
While we do have a zstream abstraction that encapsulates all the calls
to zlib as well as its error handling, we do not use it in our pack file
code. Refactor it to make the code a lot easier to understand.
While we do have a `git_zstream` abstraction that encapsulates all the
calls to zlib as well as its error handling, we do not use it in our
pack file code. Refactor it to make the code a lot easier to understand.
@pks-t
Copy link
Copy Markdown
Member Author

pks-t commented Jan 9, 2020

The code tended to get stuck randomly depending on how fast bytes came in through the network layer. The issue is taht git_zstream_get_output reads until all output bytes have been filled or the stream is at end. If the amount of input bytes cannot fill the output buffer and will not cause the current object to be finished, then we get stuck in an endless loop around git_zstream_get_output_chunk.

I tried fixing git_zstream_get_output, but it's near impossible to get right as other callers then fail. By converting to git_zstream_get_output_chunk, though, we are able to get the streaming code correct, where it will now return GIT_EBUFS as expected in case we didn't get enough input bytes. That fixes the endless loop.

Thus: never use git_zstream_get_output in a streaming scenario, but only ever git_zstream_get_output_chunk.

@pks-t pks-t force-pushed the pks/pack-zstream branch from e0cf8f9 to 2dc7b5e Compare January 9, 2020 11:26
@pks-t
Copy link
Copy Markdown
Member Author

pks-t commented Jan 9, 2020

Yup, green now. @ethomson, want to have another look? As said, the only change was to replace git_zstream_get_output with git_zstream_get_output_chunk.

@ethomson
Copy link
Copy Markdown
Member

ethomson commented Jan 9, 2020

Thanks for the ping @pks-t - I restarted the build and then forgot about it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants