Skip to content

/ datadogpy

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[dogstatsd] Handle EAGAIN socket error when dropping packets #515

Merged
merged 4 commits into from Jan 21, 2020
Merged

[dogstatsd] Handle EAGAIN socket error when dropping packets #515

merged 4 commits into from Jan 21, 2020

Conversation

@mrknmc
Copy link
Contributor

mrknmc commented Jan 14, 2020 •
edited by jirikuncar

What does this PR do?

Fixes #514 by catching socket.errorand checking errno == 11 (EAGAIN) and logs a warning without closing the socket.

Description of the Change

As mentioned in the issue, socket.send raises a BlockingIOError (socket.error on py2) exception when the socket is "full" rather than a socket.timeout exception. Hence, we create a new branch in the try-raise where we catch socket.error, check the errno and log a warning without closing the socket. This works across both python 2 and 3.

I left the socket.timeout branch in place because I wasn't sure if there is another way to set up DogStatsd where socket.timeout is thrown when the socket is full. This way, the change shouldn't break compatibility for people who rely on that behaviour.

Alternate Designs

I considered not logging a message when a socket.error with errno == 11 occurs but thought that people might want to be informed about this happening.

Possible Drawbacks

N/A

Verification Process

I created a unit test and checked that it passes.

Additional Notes

N/A

Release Notes

N/A

Review checklist (to be filled by reviewers)

  • Feature or bug fix MUST have appropriate tests (unit, integration, etc...)
  • PR title must be written as a CHANGELOG entry (see why)
  • Files changes must correspond to the primary purpose of the PR as described in the title (small unrelated changes should have their own PR)
  • PR must have one changelog/ label attached. If applicable it should have the backward-incompatible label attached.
  • PR should not have do-not-merge/ label attached.
  • If Applicable, issue must have kind/ and severity/ labels attached at least.
@mrknmc mrknmc requested a review from DataDog/integrations-tools-and-libraries as a code owner Jan 14, 2020
@mrknmc mrknmc force-pushed the mrknmc:blocking-io-error-catch branch from 75a4278 to f22e0f5 Jan 14, 2020
@mrknmc
Handle EAGAIN socket error
Verified
This commit was signed with a verified signature.
mrknmc Mark Nemec
GPG key ID: 388F515104E7D1F5 Learn about signing commits
Loading status checks…
9fef0b2
@mrknmc mrknmc force-pushed the mrknmc:blocking-io-error-catch branch from f22e0f5 to 9fef0b2 Jan 14, 2020
@mrknmc mrknmc changed the title Handle BlockingIOError when dropping packets Handle EAGAIN socket error when dropping packets Jan 14, 2020
@jd
jd requested changes Jan 15, 2020
View changes
datadog/dogstatsd/base.py Outdated
@@ -399,8 +399,11 @@ def _send_to_server(self, packet):
# dogstatsd is overflowing, drop the packets (mimicks the UDP behaviour)
pass
except (socket.error, socket.herror, socket.gaierror) as se:
log.warning("Error submitting packet: {}, dropping the packet and closing the socket".format(se))
self.close_socket()
if se.errno == 11:

This comment has been minimized.

Sign in to view
@jd

jd Jan 15, 2020 Member

This should be errno.EAGAIN.
The number is different depending on the platform.
I'd also prefer to see that under a specific socket.error except statement to be clearer. Catching everything like this is done is not a good thing, we should make clear things a bit.

This comment has been minimized.

Sign in to view
@mrknmc

mrknmc Jan 15, 2020 Author Contributor

Good point on errno.EAGAIN.

What do you mean by a specific socket.error except statement? I can create a separate statement but if we want to keep the logic the same for non-EAGAIN errors, I'd have to add the close_socket logic there as well.

This comment has been minimized.

Sign in to view
@jd

jd Jan 15, 2020 Member

I mean:

except socket.error:
   if not EAGAIN:
     close()
except (socket.herror, socket.gaierror):
     close()

in summary :)
tests/unit/dogstatsd/test_statsd.py Outdated

def send(self, payload):
error = socker.error("Socker error")
error.errno = 11

This comment has been minimized.

Sign in to view
@jd

jd Jan 15, 2020 Member

ditto use errno.EAGAIN
@mrknmc
Add separate except for socket.error
Verified
This commit was signed with a verified signature.
mrknmc Mark Nemec
GPG key ID: 388F515104E7D1F5 Learn about signing commits
Loading status checks…
93a184e
@jd
jd requested changes Jan 15, 2020
View changes
datadog/dogstatsd/base.py Show resolved Hide resolved
@mrknmc
Re-order socket.error and socket.herror, socket.gaierror exception ha…
Verified
This commit was signed with a verified signature.
mrknmc Mark Nemec
GPG key ID: 388F515104E7D1F5 Learn about signing commits
Loading status checks…
f4bec10 
…ndling
@jd
jd approved these changes Jan 15, 2020
View changes
Copy link
Member

jd left a comment

Thanks for the fix!
@jd
Copy link
Member

jd commented Jan 15, 2020

/azp run DataDog.datadogpy.integration
@azure-pipelines
Copy link

azure-pipelines bot commented Jan 15, 2020

Azure Pipelines successfully started running 1 pipeline(s).
@jd
jd reviewed Jan 17, 2020
View changes
Copy link
Member

jd left a comment

We might want to handle BlockingIOError for Python 3.
@dabcoder
Copy link
Contributor

dabcoder commented Jan 17, 2020

So we should be good with that for Python 2 and Python 3!

@mrknmc Many thanks for following up on our comments!
@mrknmc
Copy link
Contributor Author

mrknmc commented Jan 17, 2020

So we should be good with that for Python 2 and Python 3!

@mrknmc Many thanks for following up on our comments!

No problem, thanks for looking over it! :)
@dabcoder
dabcoder approved these changes Jan 17, 2020
View changes
@dabcoder
Copy link
Contributor

dabcoder commented Jan 20, 2020

@mrknmc Sorry almost ready to go, but the branch is now out of sync with the master branch. Can you update it and we should be good to go.
@mrknmc
Copy link
Contributor Author

mrknmc commented Jan 20, 2020

@mrknmc Sorry almost ready to go, but the branch is now out of sync with the master branch. Can you update it and we should be good to go.

Are you ok with a rebase or do you want me to merge master into this branch?
@dabcoder
Copy link
Contributor

dabcoder commented Jan 20, 2020

Are you ok with a rebase or do you want me to merge master into this branch?

I would say the latter, merge master into your branch please, thanks!
@mrknmc
Merge branch 'master' into blocking-io-error-catch
Verified
This commit was created on GitHub.com and signed with a verified signature using GitHub’s key.
GPG key ID: 4AEE18F83AFDEB23 Learn about signing commits
Loading status checks…
2b9cb49
@dabcoder
Copy link
Contributor

dabcoder commented Jan 20, 2020

/azp run DataDog.datadogpy.integration
@azure-pipelines
Copy link

azure-pipelines bot commented Jan 20, 2020

Azure Pipelines successfully started running 1 pipeline(s).
@mrknmc
Copy link
Contributor Author

mrknmc commented Jan 21, 2020

@dabcoder we should be good to go now, correct?
@dabcoder
Copy link
Contributor

dabcoder commented Jan 21, 2020

Yes all good thanks
@jirikuncar
jirikuncar approved these changes Jan 21, 2020
View changes
@jirikuncar jirikuncar merged commit 657bca9 into DataDog:master Jan 21, 2020
11 checks passed
11 checks passed
triage
Details
DataDog.datadogpy.integration Build #IntegrationTests succeeded
Details
DataDog.datadogpy.unit Build #20200120.1 succeeded
Details
DataDog.datadogpy.unit (UnitTests Lint) UnitTests Lint succeeded
Details
DataDog.datadogpy.unit (UnitTests Py27) UnitTests Py27 succeeded
Details
DataDog.datadogpy.unit (UnitTests Py35) UnitTests Py35 succeeded
Details
DataDog.datadogpy.unit (UnitTests Py36) UnitTests Py36 succeeded
Details
DataDog.datadogpy.unit (UnitTests Py37) UnitTests Py37 succeeded
Details
DataDog.datadogpy.unit (UnitTests Py38) UnitTests Py38 succeeded
Details
DataDog.datadogpy.unit (UnitTests Pypy2) UnitTests Pypy2 succeeded
Details
DataDog.datadogpy.unit (UnitTests Pypy3) UnitTests Pypy3 succeeded
Details
@hush-hush
hush-hush approved these changes Jan 22, 2020
View changes
@zippolyte zippolyte changed the title Handle EAGAIN socket error when dropping packets [dogstatsd] Handle EAGAIN socket error when dropping packets Feb 4, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jd jd

@jirikuncar jirikuncar

@hush-hush hush-hush

@dabcoder dabcoder

Assignees
No one assigned
Labels
changelog/Fixed kind/bug resource/dogstatsd
Projects
None yet
Milestone
No milestone
Linked issues

Successfully merging this pull request may close these issues.

DogStatsd closing socket unnecessarily when dropping packets
5 participants
@mrknmc @jd @dabcoder @jirikuncar @hush-hush
You can’t perform that action at this time.