Community: Framework mailing list archives

expert-framework@mail.odoo.com

Re: PyPy

by
ruben.de.smet
- 12/12/2014 07:33:42
Apparently, gevents didn't get installed. I guess that could be the
reason things don't work.

(venv)$ pip install gevents
doesn't do the job, because it's mostly cpython-minded

This however should do it:
(venv)$ pip install git+git://github.com/schmir/gevent@pypy-hacks
(venv)$ pip install cffi
(venv)$ pip install git+git://github.com/gevent-on-pypy/pypycore
(venv)$ export GEVENT_LOOP=pypycore.loop

Don't forget to install libev-devel or libev-dev before running the
fourth command.

Using this package didn't change anything noticable.

>From here I erased my virtualenv and database and started over. I edited
the requirements.txt to have the new gevent and lxml libraries in it. (I
attached it) Using pip install -r requirements.txt in the virtualenv,
everything should become installed.
Then I started a new database and... nope. Same error.

Is it possible that the routing gets corrupted at some point?

2014-12-12 12:30:31,844 20915 INFO testpypy
openerp.addons.base.ir.ir_http: Generating routing map
2014-12-12 12:30:33,690 20915 INFO testpypy werkzeug: 127.0.0.1 - -
[12/Dec/2014 12:30:33] "GET /web HTTP/1.1" 500 -
2014-12-12 12:30:33,721 20915 ERROR testpypy werkzeug: Error on request:


These are the last lines before pypy drops its backtrace. I'm pretty
much out of clues, any ideas are welcome!

Ruben

On 11/12/14 22:58, Ruben De Smet wrote:
> I'm listing my findings for this evening in here:
> 
> - Installing lxml using pip in a virtualenv with PyPy is a PITA on
> Fedora 20. Had to use some export CFLAGS="-I ..." for the libxml and
> libxslt libraries.
> - The guys on IRC #pypy tell me there is a lxml-cffi available (after
> bouncing on several segfaults [1] with the default one), which is
> preferred over default lxml.
> $ pip install -e git+git://github.com/amauryfa/lxml.git@cffi#egg=lxml-cffi
> should do the job of installing it.
> 
> Thereafter things started to run:
> The server started without a lot of problems (except begging for more
> modules). It didn't want to initialize the database, I'll investigate
> that on another moment.
> 
> Then I copied over my production server odoo files and I set up an SSH
> tunnel for postgresql.
> I can't login on existing databases: [2]
> 
> Now it's generating a new database (over SSH, slow!) using the files
> copied from the production server. It's saying "You may not believe it,
> but the application is actually loading". No issues in my terminal
> window so far. I'll keep you posted. As it's too late for coffee ("Take
> a minute to get a coffee, because it's still loading..."), I'll call it
> a night and post my next findings tomorrow.
> 
> EDIT: Wow, almost went to bed. It gives the same error as the
> new-install now [3]. Apparently, some XML-id cannot be found. Would this
> be a bug in lxml?
> 
> Good night!
> 
> Ruben
> 
> [1] http://paste.fedoraproject.org/158870/18326131
> [2] http://paste.fedoraproject.org/158916/3344931
> [3] http://paste.fedoraproject.org/158918/33499214
> 
> On 11/12/14 15:57, Raphaël Valyi wrote:
>> Hello Ruben,
>> some 2 months ago I tried to run Odoo v8 on PyPy again following the hints from Sandy. Basically the old style Postgresql cursor has been changed in v8 and there was no need for any hack here anymore.
>> That being said, it was still crashing(segfault I believe) badly in fields_view_get when doing complex things with lxml (C extension). That would work from some trivial views but crash in any complex views.
>> In fact, several Python projects using lxml have similar issues on PyPy it seems.
>> Amazingly back to 2009 when Oracle didn't had totally dis-moralized the Java world yet and when Jython was still alive (I mean really), I managed to get to the same point getting OpenERP running on Jython (there is even an old Launchpad branch) but that was crashing exactly on the same point.
>> Possibly, lxml is using low level C Python API in a way that is not possible with alternative implementations. So for instance, lxml now "officially" supports PyPy but mention that they use a different implementation: weakrefs instead of back pointers:  https://github.com/lxml/lxml/blob/e01a81740c77/src/lxml/proxy.pxi#L7 [1]
>> This kind of known lxml issues on PyPy may very well be related with our trouble with Odoo  http://blog.gmane.org/gmane.comp.python.pypy.bugs/month=20130101 [2]
>> I'm not sure if the micro-benchmarks benefits will hold true on real world cases like Odoo, but still if anyone manage to makes its way in the lxml portability pitfalls, I'm curious to know.
>> Best regards.
>>
>> On Thu, Dec 11, 2014 at 7:57 AM, Ruben De Smet < ruben.de.smet@glycos.org [3] > wrote:
>> Hello Odoo developers
>> I was reading some things on PyPy, the JIT python interpreter and I
>> thought this could be something very interesting for Odoo, to reduce
>> memory usage and CPU time.
>> According to this [1] (simple) benchmark, PyPy cuts memory usage in half
>> and reduces runtime to one third. This counts for memory intensive
>> applications and I think Odoo is among those.
>> Think of it: cutting memory usage and cpu usage in half means you can
>> double the amount of Odoo instances run on one server and therefore
>> could cut the TCO in half, especially for large Odoo installations.
>> According to this [2] article written by an Odoo developer, OpenERP
>> 8-trunk ran in PyPy back then, using some minor changes.
>> Caveats included
>> 1) Using psycop2cffi instead of psycop2 because it's faster and more
>> elegant (and psycop2 doesn't work using pypy). That means that the
>> (legacy) psycopg1cursor calls had to be replaced by something else which
>> is explained in the article.
>> 2) There was a segfault (with regard to unicode) in psycop2cffi which
>> was fixed by him [3]
>> 3) Another segfault: "After a bit of digging, it would seem the segfault
>> happens sometimes when calling the function on line 923 of
>> openerp-server/openerp/tools/convert.py"
>> I haven't tried anything regarding PyPy myself yet, but I'm planning to
>> in the near future.
>> For 1) I'd suggest I take a look if those call can be replaced by the
>> newer (non-legacy) variant. Regarding to the article, "and this should
>> serve as note to core developers to drop the psycopg1.cursor and use the
>> DictCursor in the psycopg.extras modules which implements a much better
>> and faster algorithm."
>> 2) Has been fixed already
>> For 3) I'd suggest the same. I'll try out Odoo on PyPy, list the caveats
>> and try to produce a smooth fix, be it in Odoo, in PyPy or in some
>> project else.
>> My questions to this list:
>> - Are there people, other than me, interested in using Odoo on PyPy?
>> - Could, once tested and stable, PyPy become officially supported? I'm
>> willing to take the testing and developing on me.
>> - Would Odoo accept patches (on github) to make Odoo work on PyPy?
>> In any case, I'll report back my findings (being caveats and performance
>> statistics, if I can gather those) when trying this out.
>> Thank you for reading through! Any feedback is welcome
>>
>> Ruben De Smet
>> [1]
>> http://blog.irukado.org/2013/06/pypy-vs-cpython-speed-memory-usage-benchmarks/ [4]
>> [2]
>> https://bwrsandman.wordpress.com/2014/04/25/running-openerp-8-0-trunk-with-pypy-and-psycopg2cffi/ [5]
>> [3]
>> https://github.com/chtd/psycopg2cffi/commit/760db9d807e17c109b5f0ae11d7bc327b14cba7a [6]
>> _______________________________________________
>> Mailing-List: https://www.odoo.com/groups/community-framework-62 [7]
>> Post to: mailto: expert-framework@mail.odoo.com [8]
>> Unsubscribe: https://www.odoo.com/groups?unsubscribe [9]
>>
>>
>> --
>> Raphaël Valyi Founder and consultant  http://twitter.com/rvalyi [10]   +55 21 3942-2434  www.akretion.com [11]
>>
>>
>> _______________________________________________
>> Mailing-List: https://www.odoo.com/groups/community-framework-62
>> Post to: mailto:expert-framework@mail.odoo.com
>> Unsubscribe: https://www.odoo.com/groups?unsubscribe
>>
>>
>>
>> [1] https://github.com/lxml/lxml/blob/e01a81740c77/src/lxml/proxy.pxi#L7
>> [2] http://blog.gmane.org/gmane.comp.python.pypy.bugs/month=20130101
>> [3] mailto:ruben.de.smet@glycos.org
>> [4] http://blog.irukado.org/2013/06/pypy-vs-cpython-speed-memory-usage-benchmarks/
>> [5] https://bwrsandman.wordpress.com/2014/04/25/running-openerp-8-0-trunk-with-pypy-and-psycopg2cffi/
>> [6] https://github.com/chtd/psycopg2cffi/commit/760db9d807e17c109b5f0ae11d7bc327b14cba7a
>> [7] https://www.odoo.com/groups/community-framework-62
>> [8] mailto:expert-framework@mail.odoo.com
>> [9] https://www.odoo.com/groups?unsubscribe
>> [10] http://twitter.com/#!/rvalyi
>> [11] http://www.akretion.com/
>>
> 

  • PyPy

    by
    ruben.de.smet
    - 12/11/2014 04:54:45 - 0

    6 replies 6 replies