Technical Mailinglisten Archive


Re: PyPy (it work's !)

- 09.06.2015 13:24:44
Hi Ruben,

Yes, I guess you are right, it seems that CPython is best on database stuff and PyPy on Python.

Here are my latest findings:
- First of all, amauryfa's cffi version of lxml does not compile on PyPy 2.6 with cffi 1.1 
( It's a pity because it seems that this project 
is not very active recently.

- Still, I managed to have Odoo working on PyPy 2.6 with the standard lxml. It seems to work,
 but Odoo randomly segfaults after a while. I supposed it is linked, but I have no proof. It 
did not improve the speed either.

- Running on PyPy 2.6 allowed me to profile Odoo (at least for the time before it segfaults) 
with the new vmprof. 
Results are there:
I don't know if they are relevant or not, but if they are with only 2% of jitted code no 
wonder it is so slow !

- After reading I focused on removing locals(), globals(), 
sys._getframe(), sys.exc_info(), and sys.settrace from Odoo and also from logging module 
which seems to be known for slowing down applications under PyPy.
Sadly, although I removed all of them it did not improve the speed of Odoo under PyPy nor 
the jitted percentage of wmprof.

That's where I am now. I suppose I should have a look at psycopg2cffi, but on their 
website they say they should be faster than CPython already... So I don't really know 
what to look next for now.

I'll check memory usage, and I'll let you know.


----- Mail original ----- 
De: "Ruben De Smet" <> 

Let me take a (very wild) guess. I think CPython is faster on things
that are done in the database and that PyPy is faster on things that are
done in Python.
The last one sounds logical. I think CPython could be faster on database
stuff because its connection manager could be faster.

Also: the XML backend is different between both Python implementations,
from which I think to know that the PyPy implementation is horribly slow.

In general I'd look after different back-end implementation between the two.

Do you happen to have had a look on the memory usage difference?

Oh, and please keep this list posted. Awesome job!


On 01/06/15 15:18, Nicolas PIGANEAU wrote:
> Hi all,
> I have done a few tests to check the speed of my running Odoo on PyPy server vs Odoo on CPython server.
> Tests were made on two docker instances (built on top of official python and pypy docker images)
> pointing to the same database docker. The same database was duplicated, one for each docker instance.
> Server call duration were measured with Firebug.
> Suprisingly, on most common calls PyPy instance was 2-3 times slower than CPython instance (I did
> make a few requests before to warm up the JIT, but it doesn't seem to bring it to the level of
> CPython).
> But on specific calls where CPython is slow, we have a performance improvement from PyPy version.
> Here is an example: On Odoo v8, go to warehouse->all operations and clicking on "ready" operations of
> receipts (I had ~30 ready receipts) gave the following:
> (First column is CPython, second is PyPy) in ms
> load                 123  159
> stock.picking/fields_view_get     31  186
> stock.picking/fields_view_get	   64  357
> get_filters	           16   27
> stock.picking/fields_get       32   81
> stock.picking.type/name_get      19   44
> stock.picking/fields_view_get     71  234
> search_read             2170  934
> Most notable is the last line which repeatedly gave best performance by 2-3 times for PyPy.
> Any idea on which part of Odoo could be slower on PyPy is very welcome !
> Nicolas PIGANEAU
> -------------------
> NDP Systèmes
> 06 68 93 87 56
> ----- Mail original -----
> De: "Nicolas PIGANEAU" <>
> À: "Odoo S.A. Community: Framework" <>
> Envoyé: Dimanche 31 Mai 2015 23:08:07
> Objet: Re: PyPy (it work's !)
> Thank you for your answers, Raphaël and Ruben.
> I've got further today, and managed to have my first Odoo server on PyPy running !
> But there is still some way to go.
> As I said before, I came across 2 issues:
> 1. Database creation does not work. After having rapidly inspected the generated database, only the base module has been loaded, although the logs show that all the modules have been loaded. I suppose that there has been a rollback somewhere, but I'll have to check.
> 2. When using an already created database, I had issues with inherited views, as you mentioned. With the fact that only one every other line of the inherited view was inserted inside the base view, I managed to locate the issue, and applying the following patch solves it:
> ===================================================================
> --- openerp/addons/base/ir/	(revision e9f802425cf8ca0f82e7075774585d4ca897ad08)
> +++ openerp/addons/base/ir/	(revision )
> @@ -449,7 +449,7 @@
> if node.getparent() is None:
> source = copy.deepcopy(spec[0])
> else:
> -            for child in spec:
> +            for child in spec[:]:
> node.addprevious(child)
> node.getparent().remove(node)
> elif pos == 'attributes':
> @@ -461,7 +461,7 @@
> del node.attrib[attribute[0]]
> else:
> sib = node.getnext()
> -          for child in spec:
> +          for child in spec[:]:
> if pos == 'inside':
> node.append(child)
> elif pos == 'after':
> Maybe there are other places in the Odoo code where the same sort of modifications should be made, but this was enough to get it running.
> For the few quick tests I made, I get QWebException complaining about void* not being known and giving a code 500. But if you reload several times, you get it running in the end. And when you get the interface, then no more problems.
> Now, for the big question: is it really faster than Odoo in CPython ?
> Well, I did not have time to make a lot of benchmarking, but:
> - The interface seems the same speed as the CPython version, I'll have to make deeper tests to see if the server calls are faster or not.
> - On one specific stock operation (validating a several hundred lines picking) which took me just above 1 minute on CPython server took only 27 seconds on the PyPy server.
> => So it seems to be significantly faster (~x2), at least on operations with a lot of python calculation like stock operations. Now it needs to be confirmed with full benchmark.
> Nicolas PIGANEAU
> -------------------
> NDP Systèmes
> 06 68 93 87 56
> ----- Mail original -----
> De: "Raphaël Valyi" <>
> À: "Community: Framework" <>
> Envoyé: Samedi 30 Mai 2015 15:02:09
> Objet: Re: PyPy
> Hello Nicolas,
> a few months ago I tried again and commented here
> Basically it seems the Pypy lxml implementation differs from the CPython one and eventually this is what breaks the view inheritance system. Also I pioneered tests on Jython back around 2011 and I had the same issues with the Java based lxml adapter not having the exact same internals of the one of CPython. May be it's not this issue anymore but I bet it is. That's interesting to see efforts here however. Let us know if you make any progress on this.
> Regards.
> On Fri, May 29, 2015 at 9:00 PM, Nicolas PIGANEAU < > wrote:
> Hi,
> Has anyone tried to run Odoo on PyPy recently ?
> On my side, I used:
> - lxml-cffi from git:// - latest gevent sources (after merging )
> - psycopg2cffi and psycopg2cffi-compat (pip install)
> I had also to patch Odoo sources to add pypy opcodes to safe_eval:
> --- ../GitHub/odoo/openerp/tools/	(date 1432802291000)
> +++ ../GitHub/odoo/openerp/tools/	(date 1432901508000)
> @@ -69,6 +69,7 @@
> _SAFE_OPCODES = _EXPR_OPCODES.union(set(opmap[x] for x in [
> And also add this one if I wanted to start in multi-worker mode (for some reason pypy complained they were float instead of int):
> --- ../GitHub/odoo/openerp/service/	(date 1432802291000)
> +++ ../GitHub/odoo/openerp/service/	(date 1432901508000)
> @@ -713,7 +713,7 @@
> raise Exception('CPU time limit exceeded.')
> signal.signal(signal.SIGXCPU, time_expired)
> soft, hard = resource.getrlimit(resource.RLIMIT_CPU)
> -    resource.setrlimit(resource.RLIMIT_CPU, (cpu_time + config['limit_time_cpu'], hard))
> +    resource.setrlimit(resource.RLIMIT_CPU, (int(cpu_time + config['limit_time_cpu']), hard))
> def process_work(self):
> pass
> Then my server starts all right and I am able to go to the database manager page.
> Creating a database seems to work, but it is then impossible to load the created database. Indeed, it seems some imports have been skipped.
> In particular, ir_model_data is only filled with xml_ids of the base module.
> So I created a database from a cpython instance and accessed it through my pypy instance. I could login, but then the loading stopped after
> displaying the top menu. A look in firebug showed that JS complained about "nv" not being defined in graph_widget.js.
> So I checked what had been loaded, and found out that only one every other scripts had been loaded from this module, but also from all
> other modules. Same goes for css files: one loaded, one skipped, one loaded, etc. for each line of the assets_backend extension template
> (web_graph/views/web_graph.xml in this case).
> I suppose this an issue linked with lxml, that sort of misses one line every two lines but I haven't got a clue on where to go now.
> Any idea or experience on this?
> Thanks !
> Nicolas PIGANEAU
> -------------------
> NDP Systèmes
> 06 68 93 87 56
> _______________________________________________
> Mailing-List:
> Post to: mailto:
> Unsubscribe:
> --
> Raphaël Valyi
> Founder and consultant
> +55 21 3942-2434
> _______________________________________________
> Mailing-List:
> Post to:
> Unsubscribe:
> _______________________________________________
> Mailing-List:
> Post to:
> Unsubscribe:

Post to: