Technical mailing list archives

technical@mail.odoo.com

Avatar

Re: Troubleshooting loading/locking issue

by
foo
- 03/27/2019 07:40:48

Thank you!


Yes, we are - Odoo plus Postgres on 1 system with 1 CPU and 1GB RAM (2GB swap). With a 2.5GB database and about 5-10 people using Odoo daily.

I’ll put in a request to split Postgres and Odoo on two different systems. I wonder, do you think Odoo and Postgres each on its own system with 1GM RAM/1 CPU is ok? 

Probably best to start there and then watch performance and upgrade if necessary. Also easier to tune Postgres/Odoo if it’s on dedicated resources. Agreed?

And thank you for the tool list!



On Wednesday, March 27, 2019, Andre de Kock <adekock11@gmail.com> wrote:
Hi Matt,

Are you running the database and the Odoo on one AWS server with only 1 CPU and 1 GB RAM? If so, you will definitely need to give more resources. Also, consider running the PostgreSQL on a separate server and only run Odoo on the one container. I think you have allocated too little resources.

>From a server point of view you can take a look at the following tools as well:
- htop (Monitor CPU and Memory)
- iotop (Monitor disk read/write)
- iftop (Monitor network)

Kind regards

On Wed, Mar 27, 2019 at 6:13 AM Matt Gx <mattgx@gmail.com> wrote:
Thank you Andre de Kock! 

I had log_min_duration_statement = 5000 set, then I dropped down to log_min_duration_statement = 1500 recently because I realized I could have many queries blocked that take 1.5 seconds or such. I won't be able to see in realtime what is happening in pg_activity (although I did install, thank you, there's a small chance I'll catch it).

Is there a way to know if all workers are busy? Or is there no log or error message, anywhere, that shows that? I'm thinking that'll be the next test we try to rule out that possibility. 

We've got 1 CPU. 3 workers total. 

https://www.odoo.com/documentation/11.0/setup/deploy.html said "Rule of thumb : (#CPU * 2) + 1" – maybe I misread. I could drop down to 2 workers but in that case probably want to add a CPU first... we also have 3 workers plus a cron worker. 

And 1GB RAM.

I'm thinking we'll double the RAM and CPU and increase workers unless anyone else spots anything obvious here. I wish there was a specific error when there is no workers available but I realize that may not be a possibility.  

Grateful for your time, thank you!


On Tue, Mar 26, 2019 at 1:06 PM Andre de Kock <adekock11@gmail.com> wrote:
Good evening!

I've noticed that loading issues are mostly due to slow queries on PostgreSQL database. To monitor this, install a tool called PG Activity and watch those logs for slow queries, especially when the loading screen appears. If there is nothing in those logs, then maybe all of your workers are busy, so I would increase the workers to number of CPU x 2. It depends on the amount of users using concurrently and also *what* they are doing.

Kind regards

On Tue, Mar 26, 2019 at 12:10 AM Matt Gx <mattgx@gmail.com> wrote:
Howdy all.

I've been tracking down a loading / performance issue for weeks and am baffled. I can't seem to narrow it down. 

Odoo 11.

In this file, addons/web/static/src/js/core/misc.js , bolded messages are getting shown to the user (in a popup):

var messages_by_seconds = function() {
    return [
        [0, _t("Loading...")],
        [20, _t("Still loading...")],
        [60, _t("Still loading...<br />Please be patient.")],
        [120, _t("Don't leave yet,<br />it's still loading...")],
        [300, _t("You may not believe it,<br />but the application is actually loading...")],
        [420, _t("Take a minute to get a coffee,<br />because it's loading...")],
        [3600, _t("Maybe you should consider reloading the application by pressing F5...")]
    ];
};

This happened about 6 times this morning (which is the most we've seen in a while). Usually it's once a week or a few times a week... been happening for months. I've tried various solutions I thought might resolve it. No luck.

A few possible culprits based off comparing dozens of timestamps...

This odoo.modules.loading always happens around this time:

2019-03-25 13:41:03,420 29502 INFO db odoo.modules.loading: loading 1 modules...
2019-03-25 13:41:04,472 29502 INFO db odoo.modules.loading: 1 modules loaded in 1.02s, 0 queries
2019-03-25 13:41:05,429 29502 INFO db odoo.modules.loading: loading 48 modules...
2019-03-25 13:41:07,933 29502 INFO db odoo.modules.loading: 48 modules loaded in 2.46s, 0 queries
2019-03-25 13:41:12,536 29502 INFO db odoo.modules.loading: Modules loaded.

I can't tell if this is supposed to be happening or if Odoo is reloading because it crashed?

Email is always getting processed during locking (1 to many emails).

IMAP from Gmail, specifically. We do a bit of custom e-mail handling and I've removed / cleaned up / combed over the code a few times and can't find any issues that might be causing this there. Per the timestamps, everything there seems to be processing fine with no long gaps (I've added more verbosity here, too).

The database is growing. ~2.5GB.

# SELECT pg_size_pretty( pg_database_size('db') );
 pg_size_pretty
----------------
 2467 MB
(1 row)

We have it at AWS.
I have log_min_duration_statement = 3000 (previously 5000). Although it takes longer than this (per the error messages), no slow queries getting logged so I don't think this is it. 

It's possible there is some postgres locking issue but I think I'd see a slow query if that was the case.

A RAM issue? We have 1GB with 2GB swap.

# free -m
              total        used        free      shared  buff/cache   available
Mem:            996         654          92         133         249          80
Swap:          2011         630        1381

Odoo config issue?

workers = 3
max_cron_threads = 1
proxy_mode = True
longpolling_port = 8072
limit_time_cpu = 3600
limit_time_real = 3600


Does anyone have any pointers or ideas on what is causing this or things to try? Or further questions? We can increase RAM at AWS. I'm wondering if there is something else I can be doing to troubleshoot, too, if there is nothing obvious that jumps out.

Grateful for any time in helping to resolve this, thank you! 

_______________________________________________
Mailing-List: https://www.odoo.com/groups/technical-62
Post to: mailto:technical@mail.odoo.com
Unsubscribe: https://www.odoo.com/groups?unsubscribe

_______________________________________________
Mailing-List: https://www.odoo.com/groups/technical-62
Post to: mailto:technical@mail.odoo.com
Unsubscribe: https://www.odoo.com/groups?unsubscribe

_______________________________________________
Mailing-List: https://www.odoo.com/groups/technical-62
Post to: mailto:technical@mail.odoo.com
Unsubscribe: https://www.odoo.com/groups?unsubscribe

_______________________________________________
Mailing-List: https://www.odoo.com/groups/technical-62
Post to: mailto:technical@mail.odoo.com
Unsubscribe: https://www.odoo.com/groups?unsubscribe

Reference