Issue 1223381: PEP 342 Generator enhancements

The attached patch implements code and test changes for:

* yield expressions
* bare yield (short for yield None)
* yield in try/finally
* generator.send(value) (send value into generator;
substituted for PEP 342's next(arg))
* generator.throw(typ[,val[,tb]]) (raise error in
generator)
* generator.close() (per PEP 343)
* GeneratorExit built-in exception type
* generator.__del__ (well, the C equivalent)
* All necessary mods to the compiler, parser module,
and Python 'compiler' package to support these changes.

It was necessary to change a small part of the eval
loop (well, the initialization, not the loop) and the
gc module's has_finalizer() logic in order to support a
C equivalent to __del__.  Specialists in these areas
should probably scrutinize this patch!

There is one additional implementation detail that was
not contemplated in either PEP. in order to prevent
used-up generators from retaining unnecessary
references to their frame's contents, I set the
generator's gi_frame member to None whenever the
generator finishes normally or with an error.  Thus, an
exhausted generator cannot be part of a cycle, and it
releases its frame object sooner than in previous
Python versions.  For generators used only in a direct
"for" loop, this makes no difference, but for
generators used with the iterator protocol (i.e.
"gen.next()") from Python, this avoids stranding the
generator's frame in a traceback cycle.

Logged In: YES 
user_id=1188172

Minor nit: on line 642, 'yield_stmt' should be changed to
'yield_expr'.

Logged In: YES 
user_id=6656

Argh, I'd hoped you'd done with, too :)

Have you done any work on that yet?

Logged In: YES 
user_id=56214

No, I haven't, and I never really planned to be the person
implementing "with".  Doesn't mean I might not end up doing
it eventually, but it's not currently in my plans.  I'm
going to have my hands full just updating PEP 342 (Guido has
asked me to take it over), and getting docs written to go
with this patch.  Between all that and my distutils-related
work, my pro-bono Python project time is booked up solid for
maybe the next 2 months or so.

Logged In: YES 
user_id=6656

Fair enough.  I want to use with statements in a presentation at 
EuroPython next week, so I've hacked together a truly horrible 
implementation that will do for now and will see about a real one some time 
after...

Logged In: YES 
user_id=6380

(Removing 343 from the subject since all this is now in PEP
342.)

Playing with the code a bit, I notice that throw() doesn't
like string exceptions. That could cause some problems.

Also, I think the tb argument should be allowed to be None.

Otherwise, great! In a few hours you can check it in. :-)

Logged In: YES 
user_id=6656

Heh heh heh.

Logged In: YES 
user_id=56214

I had hoped to be able to avoid dealing with string
exceptions, but I guess it makes sense that if somebody has
thrown one, it should be able to propagate. So I guess I'll
take care of that along with the rest.

Logged In: YES 
user_id=6380

Since I've accepted PEP 342+343 last night in my Europython
keynote, you can check this in now.

Logged In: YES 
user_id=6656

You missed an incref in gen_throw (symptom was

Fatal Python error: ../Objects/tupleobject.c:169 object at
0x4013807c has negative ref count -606348326

on exit after running test_generators).  New patch attached.

Logged In: YES 
user_id=56214

mwh, is the only difference in your patch the addition of
the incref?

Logged In: YES 
user_id=6656

I think so, yes :)

Logged In: YES 
user_id=56214

Checked in to CVS.  throw() still needs string support and
to allow None as the third argument, but I figure it's
better to get this thing into mainline CVS and address those
items separately.  I did include mwh's Py_INCREF fix, however.