Community mailing list archives

community@mail.odoo.com

Re: Odoo Performance and concurrency locks

by
Camptocamp France SAS, Alexandre Fayolle - Camptocamp
- 09/16/2015 03:51:44
On 15/09/2015 20:34, Olivier Dony wrote:
> On 09/15/2015 11:13 AM, Alexandre Fayolle wrote:
>> On 14/09/2015 23:17, Antony Lesuisse wrote:
>> base_concurrency does not handle the longpolling bus. But it allows:
>>
>> 1. importing csv files in parallel
>> 2. connecting as admin while long transactions are in progress (e.g. mrp
>> on a large db, large CSV file import)
>>
>> Having these benefits natively in v9 would be great.
> 
> AFAICS, the only purpose of base_concurrency is to avoid login transactions 
> grabbing exclusive locks on res.users rows, because of the potential conflicts 
> with share locks via create_uid/write_uid FKs to res.users.
> 
> Avoiding those conflicts is exactly the purpose of the FOR KEY SHARE/FOR NO KEY 
> UPDATE locking levels introduced in PG 9.3. So upgrading to PG9.3 gives the 
> same result natively, with more benefits and performance included, and less 
> hacky code... ;-)

Hello Olivier,

I'd advocate that the hacky feeling about base_concurreny comes from the
requirement to preserve the fields presented to other modules.

Note: I use Postgresql 9.3.9.

A typical use case I have for base_concurrency is the following: I need
to perform a large import of data in an Odoo database. For instance 150k
partners, for which I have a csv file. Importing this in one go is
impractical as each line in the csv file takes ~500ms to be processed.
The obvious approach to speed up the import to split the csv file and to
run the import using N different processes. Since the import process is
fairly CPU intensive on the Odoo side, and I have lots of cores, this
should help. Besides having smaller chunks provide a smaller penalty
from the checkpoints, by forcing more commits.

In this case, not having base_concurrency, even with Pg 9.3, will cause
rollbacks:

Traceback (most recent call last):
  File "server/openerp/sql_db.py", line 234, in execute
    res = self._obj.execute(query, params)
TransactionRollbackError: could not serialize access due to concurrent
update
CONTEXT:  SQL statement "SELECT 1 FROM ONLY "public"."res_users" x WHERE
"id" OPERATOR(pg_catalog.=) $1 FOR KEY SHARE OF x"

So yes this is exactly the situation you are describing as "unusual" in
your other email, but this is a very real case, and not unusual at all,
at least for me. I'm pretty certain that a similar situation can be
achieved with the procurement scheduler on large instances.



-- 
Alexandre Fayolle
Chef de Projet
Tel : +33 4 58 48 20 30

Camptocamp France SAS
Savoie Technolac, BP 352
73377 Le Bourget du Lac Cedex
http://www.camptocamp.com