Community: Framework mailing list archives

expert-framework@mail.odoo.com

Re: PyPy (it work's !)

by
ruben.de.smet
- 06/09/2015 09:46:07
Let me take a (very wild) guess. I think CPython is faster on things
that are done in the database and that PyPy is faster on things that are
done in Python.
The last one sounds logical. I think CPython could be faster on database
stuff because its connection manager could be faster.

Also: the XML backend is different between both Python implementations,
from which I think to know that the PyPy implementation is horribly slow.

In general I'd look after different back-end implementation between the two.

Do you happen to have had a look on the memory usage difference?

Oh, and please keep this list posted. Awesome job!

Ruben

On 01/06/15 15:18, Nicolas PIGANEAU wrote:
> Hi all,
> I have done a few tests to check the speed of my running Odoo on PyPy server vs Odoo on CPython server.
> Tests were made on two docker instances (built on top of official python and pypy docker images)
> pointing to the same database docker. The same database was duplicated, one for each docker instance.
> Server call duration were measured with Firebug.
> Suprisingly, on most common calls PyPy instance was 2-3 times slower than CPython instance (I did
> make a few requests before to warm up the JIT, but it doesn't seem to bring it to the level of
> CPython).
> But on specific calls where CPython is slow, we have a performance improvement from PyPy version.
> Here is an example: On Odoo v8, go to warehouse->all operations and clicking on "ready" operations of
> receipts (I had ~30 ready receipts) gave the following:
> 
> (First column is CPython, second is PyPy) in ms
> load                 123  159
> stock.picking/fields_view_get     31  186
> stock.picking/fields_view_get	   64  357
> get_filters	           16   27
> stock.picking/fields_get       32   81
> stock.picking.type/name_get      19   44
> stock.picking/fields_view_get     71  234
> search_read             2170  934
> Most notable is the last line which repeatedly gave best performance by 2-3 times for PyPy.
> Any idea on which part of Odoo could be slower on PyPy is very welcome !
> Nicolas PIGANEAU
> -------------------
> NDP Systèmes
> 06 68 93 87 56
> http://www.ndp-systemes.fr
> ----- Mail original -----
> De: "Nicolas PIGANEAU" <nicolas.piganeau@ndp-systemes.fr>
> À: "Odoo S.A. Community: Framework" <expert-framework@mail.odoo.com>
> Envoyé: Dimanche 31 Mai 2015 23:08:07
> Objet: Re: PyPy (it work's !)
> Thank you for your answers, Raphaël and Ruben.
> I've got further today, and managed to have my first Odoo server on PyPy running !
> But there is still some way to go.
> As I said before, I came across 2 issues:
> 1. Database creation does not work. After having rapidly inspected the generated database, only the base module has been loaded, although the logs show that all the modules have been loaded. I suppose that there has been a rollback somewhere, but I'll have to check.
> 2. When using an already created database, I had issues with inherited views, as you mentioned. With the fact that only one every other line of the inherited view was inserted inside the base view, I managed to locate the issue, and applying the following patch solves it:
> ===================================================================
> --- openerp/addons/base/ir/ir_ui_view.py	(revision e9f802425cf8ca0f82e7075774585d4ca897ad08)
> +++ openerp/addons/base/ir/ir_ui_view.py	(revision )
> @@ -449,7 +449,7 @@
> if node.getparent() is None:
> source = copy.deepcopy(spec[0])
> else:
> -            for child in spec:
> +            for child in spec[:]:
> node.addprevious(child)
> node.getparent().remove(node)
> elif pos == 'attributes':
> @@ -461,7 +461,7 @@
> del node.attrib[attribute[0]]
> else:
> sib = node.getnext()
> -          for child in spec:
> +          for child in spec[:]:
> if pos == 'inside':
> node.append(child)
> elif pos == 'after':
> Maybe there are other places in the Odoo code where the same sort of modifications should be made, but this was enough to get it running.
> For the few quick tests I made, I get QWebException complaining about void* not being known and giving a code 500. But if you reload several times, you get it running in the end. And when you get the interface, then no more problems.
> Now, for the big question: is it really faster than Odoo in CPython ?
> Well, I did not have time to make a lot of benchmarking, but:
> - The interface seems the same speed as the CPython version, I'll have to make deeper tests to see if the server calls are faster or not.
> - On one specific stock operation (validating a several hundred lines picking) which took me just above 1 minute on CPython server took only 27 seconds on the PyPy server.
> => So it seems to be significantly faster (~x2), at least on operations with a lot of python calculation like stock operations. Now it needs to be confirmed with full benchmark.
> 
> Nicolas PIGANEAU
> -------------------
> NDP Systèmes
> 06 68 93 87 56
> http://www.ndp-systemes.fr
> ----- Mail original -----
> De: "Raphaël Valyi" <rvalyi@akretion.com>
> À: "Community: Framework" <expert-framework@mail.odoo.com>
> Envoyé: Samedi 30 Mai 2015 15:02:09
> Objet: Re: PyPy
> Hello Nicolas,
> a few months ago I tried again and commented here https://www.odoo.com/groups/community-framework-62/community-framework-10089512
> Basically it seems the Pypy lxml implementation differs from the CPython one and eventually this is what breaks the view inheritance system. Also I pioneered tests on Jython back around 2011 and I had the same issues with the Java based lxml adapter not having the exact same internals of the one of CPython. May be it's not this issue anymore but I bet it is. That's interesting to see efforts here however. Let us know if you make any progress on this.
> Regards.
> On Fri, May 29, 2015 at 9:00 PM, Nicolas PIGANEAU < nicolas.piganeau@ndp-systemes.fr > wrote:
> 
> Hi,
> Has anyone tried to run Odoo on PyPy recently ?
> On my side, I used:
> - lxml-cffi from git:// github.com/amauryfa/lxml.git - latest gevent sources (after merging https://github.com/gevent/gevent/pull/552 )
> - psycopg2cffi and psycopg2cffi-compat (pip install)
> I had also to patch Odoo sources to add pypy opcodes to safe_eval:
> --- ../GitHub/odoo/openerp/tools/safe_eval.py	(date 1432802291000)
> +++ ../GitHub/odoo/openerp/tools/safe_eval.py	(date 1432901508000)
> @@ -69,6 +69,7 @@
> _SAFE_OPCODES = _EXPR_OPCODES.union(set(opmap[x] for x in [
> 'LOAD_NAME', 'CALL_FUNCTION', 'COMPARE_OP', 'LOAD_ATTR',
> +  'CALL_METHOD','LOOKUP_METHOD',
> 'STORE_NAME', 'GET_ITER', 'FOR_ITER', 'LIST_APPEND', 'DELETE_NAME',
> 'JUMP_FORWARD', 'JUMP_IF_TRUE', 'JUMP_IF_FALSE', 'JUMP_ABSOLUTE',
> 'MAKE_FUNCTION', 'SLICE+0', 'SLICE+1', 'SLICE+2', 'SLICE+3', 'BREAK_LOOP',
> 
> And also add this one if I wanted to start in multi-worker mode (for some reason pypy complained they were float instead of int):
> --- ../GitHub/odoo/openerp/service/server.py	(date 1432802291000)
> +++ ../GitHub/odoo/openerp/service/server.py	(date 1432901508000)
> @@ -713,7 +713,7 @@
> raise Exception('CPU time limit exceeded.')
> signal.signal(signal.SIGXCPU, time_expired)
> soft, hard = resource.getrlimit(resource.RLIMIT_CPU)
> -    resource.setrlimit(resource.RLIMIT_CPU, (cpu_time + config['limit_time_cpu'], hard))
> +    resource.setrlimit(resource.RLIMIT_CPU, (int(cpu_time + config['limit_time_cpu']), hard))
> def process_work(self):
> pass
> 
> Then my server starts all right and I am able to go to the database manager page.
> Creating a database seems to work, but it is then impossible to load the created database. Indeed, it seems some imports have been skipped.
> In particular, ir_model_data is only filled with xml_ids of the base module.
> So I created a database from a cpython instance and accessed it through my pypy instance. I could login, but then the loading stopped after
> displaying the top menu. A look in firebug showed that JS complained about "nv" not being defined in graph_widget.js.
> So I checked what had been loaded, and found out that only one every other scripts had been loaded from this module, but also from all
> other modules. Same goes for css files: one loaded, one skipped, one loaded, etc. for each line of the assets_backend extension template
> (web_graph/views/web_graph.xml in this case).
> I suppose this an issue linked with lxml, that sort of misses one line every two lines but I haven't got a clue on where to go now.
> Any idea or experience on this?
> Thanks !
> Nicolas PIGANEAU
> -------------------
> NDP Systèmes
> 06 68 93 87 56 http://www.ndp-systemes.fr
> _______________________________________________
> Mailing-List: https://www.odoo.com/groups/community-framework-62
> Post to: mailto: expert-framework@mail.odoo.com
> Unsubscribe: https://www.odoo.com/groups?unsubscribe
> 
> 
> 
> --
> Raphaël Valyi
> Founder and consultant
> http://twitter.com/rvalyi
> +55 21 3942-2434
> www.akretion.com
> 
> _______________________________________________
> Mailing-List: https://www.odoo.com/groups/community-framework-62
> Post to: mailto:expert-framework@mail.odoo.com
> Unsubscribe: https://www.odoo.com/groups?unsubscribe
> _______________________________________________
> Mailing-List: https://www.odoo.com/groups/community-framework-62
> Post to: mailto:expert-framework@mail.odoo.com
> Unsubscribe: https://www.odoo.com/groups?unsubscribe
>