[Mono-dev] IronPython Performance
lupus at ximian.com
Mon Apr 24 05:52:39 EDT 2006
[I continued your large cross-post, even if I'm not subscribed to
python-dev: hopefully the moderators will approve the post if needed.]
On 04/21/06 Brent Fulgham wrote:
> A while ago (nearly two years) an interesting paper was published by Jim Hugunin
> (http://www.python.org/pycon/dc2004/papers/9/) crowing about the significant
> speed advantage observed in this interpreter running on top of Microsoft's .NET
> VM stack. I remember being surprised by these results, as Python has always
> seemed fairly fast for an interpreted language.
> I've been working on the Programming Language Shootout for a few years now, and
> after growing tired of the repeated requests to include scores for IronPython I
> finally added the implementation this week.
> Comparison of IronPython (1.0 Beta 5) to Python 2.4.3  and IronPython (1.0 Beta 1)
> to Python 2.4.2  do not match the results reported in the 2004 paper. In fact,
> IronPython is consistenly 3 to 4 times slower (in some cases worse than that), and
> seemed to suffer from recursion/stack limitations.
You're comparing different benchmarks, so it's not a surprise that you
get different numbers.
I only tried two benchmarks from the paper, pystone and the globals one
(attached) and the results are largely equivalent to the ones from the
time mono IronPython-1.0-Beta6/IronPythonConsole.exe -O python-globals-bench2.py
(Note the above includes the startup time that I guess was not
considered in the paper and is 0.620 seconds running test(1): just
increasing the number of iterations will make the difference converge at
about 10x faster mono vs python2.4).
time python2.4 -O python-globals-bench2.py
python2.4 -O /usr/lib/python2.4/test/pystone.py
Pystone(1.1) time for 50000 passes = 1.4
This machine benchmarks at 35714.3 pystones/second
mono IronPython-1.0-Beta6/IronPythonConsole.exe -O /usr/lib/python2.4/test/pystone.py
Pystone(1.1) time for 50000 passes = 0.989288
This machine benchmarks at 50541.4 pystones/second
So IronPython is 40+% faster than CPython, like in the paper.
The mono version I used is roughtly 1.1.14.
As for the recursion limitations: what are they? I took the recursive
benchmark from the shootout page and both CPython and IronPython need
the call to setrecursionlimit (actually cpython needed it earlier for
me, anyway). This might be a difference between IronPython beta 5 and
the beta6 I'm using, though. It is also interesting because on my system
running the attached python-ack.py with 8 as argument, mono/IronPython
is 10% faster than CPython, while the shootout page reports CPython 2.4
50% faster. Feel free to investigate if this is because of improvements
in mono or IronPython:)
> I am aware of the following issues that might affect the results:
> 1) Mono vs. Microsoft's CLR.
This can certainly be a factor: some things in the MS runtime run faster and
some slower than in the Mono runtime, so it depends which features are
exercised the most in a particular benchmark. It is very likely that in
most benchmarks the MS runtime will score better.
> 2) Python 2.1 vs. Python 2.4
The python folks will have more details, but python2.4 seems to be
faster than 2.1 and 2.3 in several areas.
> 3) Changes in IronPython over the last two years.
For some time the IronPython folks have focused on features and
correctness issues, so I guess they'll start caring about performance in
the future. It is also to note that IronPython now uses generics which
have yet to see optimizations work in the mono runtime and libraries
(they are a 2.0 feature).
> I thought these findings might be interesting to some of the Python, IronPython,
> and Mono developers. Let the flames begin! ;-)
Why flames? Hard numbers are just that: useful information. It's just
your conclusions that might be incorrect;-)
I think the summary to take away is:
*) Sometimes IronPython on mono is faster than CPython
*) Often CPython is faster than IronPython on mono
*) IronPython and mono are improving quickly.
Thanks for the numbers: it would be nice if someone with time in his
hands could rerun the benchmarks with ironpython beta6, mono from svn
(or at least mono 1.1.14/1.1.15) and also with the MS runtime on windows.
Better if this includes running pystone to relate to improvements in the
published paper and the piethon benchmarks (on the MS runtime, in mono
we need to implement some things that newer ironpythons use). In the
meantime here are the piethon results running with IronPython 0.6 with a
b0 = 2.49623870849609 -- [2.59130096435547, 2.40117645263672]
b1 = 0.473392486572266 -- [0.496170043945312, 0.450614929199219]
b2 = 0.453083038330078 -- [0.543968200683594, 0.362197875976562]
b3 = 0.967914581298828 -- [0.995307922363281, 0.940521240234375]
b4 = 1.11585998535156 -- [1.23159027099609, 1.00012969970703]
b5 = 3.56014633178711 -- [3.70928192138672, 3.4110107421875]
b6 = 1.43364334106445 -- [1.42024230957031, 1.44704437255859]
all done in 21.01 sec
Python 2.4 results:
b0 = 2.185000 -- [2.1800000000000002, 2.1900000000000004]
b1 = 0.915000 -- [0.93999999999999995, 0.88999999999999879]
b2 = 0.395000 -- [0.39999999999999991, 0.39000000000000057]
b3 = 1.015000 -- [1.02, 1.0099999999999998]
b4 = 0.585000 -- [0.58999999999999986, 0.58000000000000007]
b5 = 0.995000 -- [0.99000000000000021, 1.0]
b6 = 1.410000 -- [1.4199999999999999, 1.4000000000000004]
all done in 15.00 sec
Python 2.3 results:
b0 = 2.270000 -- [2.2800000000000002, 2.2600000000000016]
b1 = 0.955000 -- [0.98999999999999977, 0.91999999999999993]
b2 = 0.395000 -- [0.39000000000000012, 0.39999999999999858]
b3 = 1.230000 -- [1.2300000000000004, 1.2300000000000004]
b4 = 0.805000 -- [0.80999999999999961, 0.80000000000000071]
b5 = 1.145000 -- [1.1500000000000004, 1.1399999999999988]
b6 = 1.500000 -- [1.5099999999999989, 1.4900000000000002]
all done in 16.60 sec
Since Guido designed it to excercise the tricky corners of the python
implementation this could be a better general evaluation of the speed
of a python runtime. Note that in the oscon paper, mono was reported to
be 23 time slower than CPython (but 2 times slower excluding b5 where we
had a really unoptimized codepath). Current mono is 27% slower than
CPython 2.3 and 40% slower than CPython 2.4. So while the CPython
performance increased, mono's rate of improvement is significantly
While having mono/IronPython be faster than CPython in more benchmarks
would be nice (and I guess we could run a few profile runs if there is
interest to see if some quick specific improvement could be done),
it is not really necessary.
Having mono run python code 1-2 times slower than CPython is pretty
good. People interested in raw performance could just write the code in
C#, gluelessly use it from python code and get the 1-2 orders of
magnitude improvements visible also in the shootout comparison of C# vs
lupus at debian.org debian/rules
lupus at ximian.com Monkeys do it better
More information about the Mono-devel-list