Issue 5671: Speed up pickling of lists in cPickle

The attached patch adds another version of cPickle.c's batch_list(),
batch_list_exact(), which is specialized for "type(x) is list". This
provides a nice performance boost when pickling objects that use
lists. This is similar to the approach taken in issue 5670, though the
performance boost on our benchmark is smaller:

Pickle:
Min: 2.231 -> 2.200: 1.39% faster
Avg: 2.266 -> 2.227: 1.72% faster
Significant (t=10.994064, a=0.95)


Benchmark is at
http://code.google.com/p/unladen-swallow/source/browse/tests/performance/macro_pickle.py
(driver is  ../perf.py; perf.py was run with "--rigorous -b pickle").
Workloads involving more lists will benefit more.

This patch passes all the tests added in issue 5665. I would recommend
reviewing that patch first. I'll port to py3k once this is reviewed for
trunk.

Out of curiousity, are you also benchmarking against marshal and json? 
ISTM, that there is always one of them that will be the fastest and that
the others should mimic that approach.

No, we haven't started looking at other serialization formats yet.
Marshal will probably be my next target, with json being a lower
priority. There were enough instances of low-hanging fruit in cPickle
that I didn't go looking at the other implementations for ideas.

Here's a patch for py3k. I also added a special-case for 1-item lists.

A micro-benchmark of Collin's patch:

python -m timeit -s "import cPickle; l=range(150)" "cPickle.dumps(l,
protocol=-1)"

* before: 12.1 usec per loop
* after: 10.1 usec per loop

=> 15% faster on a favorable case

I've added a microbenchmark to perf.py called pickle_list. Running that
on this change (perf.py -r -b pickle_list):

pickle_list:
Min: 1.126 -> 0.888: 26.86% faster
Avg: 1.154 -> 0.906: 27.43% faster
Significant (t=115.404547, a=0.95)

That's probably the upper bound on the performance improvement to be
realized from this patch.

Still applies cleanly (with a little fuzz) to the trunk after applying
the issue 5683 patch.  Tests all still pass (including xpickle w/ 2.4,
2.5, 2.6 available).

pickle_batch_list_exact_py3k.diff applies cleanly on current py3k trunk and passes tests.  cpickle_list.patch applies cleanly against 2.x trunk and passes "make test".

I don't see any objections brought up about this set of patches, so we can get these applied?

It is too late now for the 2.x version. And, the huge patch in issue 9410 includes an updated version of this patch for 3.x.

superseder: Add Unladen Swallow's optimizations to Python 3's pickle.
stage: resolved

nosy: + jafo
messages: + msg101461

assignee: alexandre.vassalotti