[Python-Dev] Track ResourceWarning warnings with tracemalloc

[Victor Stinner]

Hi,

I'm trying to write an example of usage of the new
tracemalloc.get_object_traceback() function. Last month I proposed to
use it to give the traceback where a file/socket was allocated on
ResourceWarning:

   https://mail.python.org/pipermail/python-dev/2013-October/129923.html

I found a working solution using monkey patching of the socket, _pyio
and builtins modules:

   https://bitbucket.org/haypo/misc/src/tip/python/res_warn.py

This hack uses tracemalloc to retrieve the traceback where the
file/socket was allocated, but you can do the same using
traceback.extract_stack() for older Python version. So it's just an
example to show how tracemalloc can be used.


Example with test_poplib (without res_warn.py):

[1/1] test_poplib
/home/haypo/prog/python/default/Lib/test/support/__init__.py:1331:
ResourceWarning: unclosed <ssl.SSLSocket fd=5,
family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('127.0.0.1',
45747), raddr=('127.0.0.1', 48933)>
  gc.collect()

Ok, you now that the socket was destroyed be the garbage collector...
But which test created this socket???


Example of res_warn.py with test_poplib:

[1/1] test_poplib
/home/haypo/prog/python/default/Lib/test/support/__init__.py:1331:
ResourceWarning: unclosed <ssl.SSLSocket fd=5,
family=AddressFamily.AF_INET, type=2049, proto=6, laddr=('127.0.0.1',
49715), raddr=('127.0.0.1', 43904)>
  gc.collect()
Allocation traceback (most recent first):
  File '/home/haypo/prog/python/default/Lib/ssl.py', line 340
    _context=self)
  File '/home/haypo/prog/python/default/Lib/poplib.py', line 390
    self.sock = context.wrap_socket(self.sock)
  File '/home/haypo/prog/python/default/Lib/test/test_poplib.py', line 407
    self.client.stls()
  File '/home/haypo/prog/python/default/Lib/unittest/case.py', line 567
    self.setUp()
  File '/home/haypo/prog/python/default/Lib/unittest/case.py', line 610
    return self.run(*args, **kwds)
  (...)

You can see that the socket was created by test_poplib at line 407
TestPOP3_TLSClass.setUp). It's more useful than the previous warning
:-)


I found two issues in _pyio and tracemalloc modules. These issues
should be fixed to use res_warn.py on text or buffered files and use
it during Python shutdown:

   http://bugs.python.org/issue19831
   http://bugs.python.org/issue19829

Victor