Issue 35502: Memory leak in xml.etree.ElementTree.iterparse

When given xml that that would raise a ParseError, but parsing is stopped before the ParseError is raised, xml.etree.ElementTree.iterparse leaks memory.

Example:


import gc
from io import StringIO
import xml.etree.ElementTree as etree

import objgraph


def parse_xml():
    xml = """
      <LEVEL1>
      </LEVEL1>
    </ROOT>
    """
    parser = etree.iterparse(StringIO(initial_value=xml))
    for _, elem in parser:
        if elem.tag == 'LEVEL1':
            break


def run():
    parse_xml()

    gc.collect()
    uncollected_elems = objgraph.by_type('Element')
    print(uncollected_elems)
    objgraph.show_backrefs(uncollected_elems, max_depth=15)


if __name__ == "__main__":
    run()


Output:
[<Element 'LEVEL1' at 0x10df712c8>]

Also see this gist which has an image showing the objects that are retained in memory: https://gist.github.com/grokcode/f89d5c5f1831c6bc373be6494f843de3

I wrote attached run.py which confirms a leak using tracemalloc:

$ python3 run.py 
1 calls: 15.3B / call (total: 15.3 kB)
100 calls: 15.3B / call (total: 1527.7 kB)
1000 calls: 15.3B / call (total: 15265.0 kB)

Oops, there was a typo, you should read kB:

1 calls: 15.3 kB / call (total: 15.3 kB)
100 calls: 15.3 kB / call (total: 1527.7 kB)
1000 calls: 15.3 kB / call (total: 15265.0 kB)

The problem was with detecting a reference cycle containing a TreeBuilder.

Oops, my PR 11169 used the wrong issue number: bpo-35257 instead of bpo-35502. Anyway, I closed it, the change is too complex.

--

IMHO the root issue is the handling of the SyntaxError exception in XMLPullParser.feed(). I wrote a fix, but I don't have the bandwidth to write an unit test checking that the reference cycle is broken.

commit 9f3354d36a89d7898bdb631e5119cc37e9a74840 (fix_etree_leak)
Author: Victor Stinner <vstinner@redhat.com>
Date:   Fri Dec 14 22:03:16 2018 +0100

    bpo-35257: Fix memory leak in XMLPullParser.feed()
    
    Fix memory leak in XMLPullParser.feed() of xml.etree: on syntax
    error, clear the traceback to break a reference cycle.

diff --git a/Lib/xml/etree/ElementTree.py b/Lib/xml/etree/ElementTree.py
index c1cf483cf5..f17c52541b 100644
--- a/Lib/xml/etree/ElementTree.py
+++ b/Lib/xml/etree/ElementTree.py
@@ -1266,6 +1266,8 @@ class XMLPullParser:
             try:
                 self._parser.feed(data)
             except SyntaxError as exc:
+                # bpo-35502: Break reference cycle
+                #exc.__traceback__ = None
                 self._events_queue.append(exc)
 
     def _close_and_return_root(self):


I don't see any behavior difference in XMLPullParser.read_events() which raise again the exception:

        events = self._events_queue
        while events:
            event = events.popleft()
            if isinstance(event, Exception):
                raise event
            else:
                yield event

--

PR 11170 is also a nice enhancement (fix treebuilder_gc_traverse()), but maybe we should also prevent creating reference cycles in the first place?

It is not easy to avoid reference cycles if use a generator function. And generator function is much faster than an implementation as a class with the __next__ method. We need to access the iterator object from the code of the generator function, and this creates a cycle.

New changeset d2a75c67830d7c9f59e4e9b60f36974234c829ef by Serhiy Storchaka in branch 'master':
bpo-35502: Fix reference leaks in ElementTree.TreeBuilder. (GH-11170)
https://github.com/python/cpython/commit/d2a75c67830d7c9f59e4e9b60f36974234c829ef

New changeset 60c919b58bd3cf8730947a00ddc6a527d6922ff1 by Miss Islington (bot) in branch '3.7':
bpo-35502: Fix reference leaks in ElementTree.TreeBuilder. (GH-11170)
https://github.com/python/cpython/commit/60c919b58bd3cf8730947a00ddc6a527d6922ff1

This ticket looks like it's done for 3.7/8. Can it be closed?
I guess 3.6 isn't relevant anymore, right?

The 3.6 branch no longer accept bugfixes.

messages: + msg331875
versions: + Python 3.8

nosy: + serhiy.storchaka

messages: + msg331864