Skip to Content
Menu
This question has been flagged

Persistent Real-time Chat (Longpolling) & ir_cron Lock Errors

Subject: Odoo 18 Real-time Chat Not Working (Requires Refresh) & Recurring ir_cron Lock Errors in Multi-DB Docker Setup

Odoo Version: 18.0 (Community Edition) Deployment: Docker Compose on Ubuntu Server 20.04.6 LTS Reverse Proxy: Caddy Database: PostgreSQL 15 (Docker container) Issue Type: Real-time communication (longpolling/websockets) and database contention (ir_cron locks)

Problem Description:

We are experiencing two persistent and critical issues with our Odoo 18 setup, which runs multiple Odoo databases (one per domain) on a single live18-odoo Docker container connected to a shared live18-db PostgreSQL container:

  1. OdooBot/Discuss Real-time Chat Not Working:
    • When chatting with OdooBot or in Discuss channels, replies and new messages do not appear in real-time. The page must be manually refreshed to see new content.
    • Direct access via http://144.126.135.5:18069 (bypassing Caddy/SSL) still shows "Real-time connection lost...", confirming the issue is internal to Odoo.
  2. Recurring ir_cron Database Lock Errors (Root Cause Identified):
    • We frequently observe ERROR: could not obtain lock on row in relation "ir_cron" in our PostgreSQL logs, specifically for id IN (6) ("Mail: Fetchmail Service").
    • CRITICAL NEW FINDING: Odoo logs also show RuntimeError: OrderedDict mutated during iteration. This error occurs when Odoo's cron system tries to loop through its list of active databases, but that list unexpectedly changes or becomes unstable. This is the root cause of the ir_cron locking errors and indicates fundamental instability in Odoo's multi-database management for cron jobs.
    • This database registry instability leads to cron job failures, memory spikes, and general Odoo unresponsiveness.

Server and Odoo Configuration Details:

  • Server IP: 144.126.135.5
  • Odoo Docker Compose (~/docker-apps/odoo18/live18/docker-compose.yml): Standard setup with Odoo 18, PostgreSQL 15, exposing ports 18069 (main) and 18072 (longpolling). Uses ODOO_PROXY_MODE=True and ODOO_DB_FILTER=^%h$.
  • Odoo Configuration (~/docker-apps/odoo18/live18/config/odoo.conf): log_level = debug, longpolling_port = 8072, workers = 4, max_cron_threads = 2, proxy_mode = True, dbfilter = ^%h$. web.base.url is not frozen.
  • Caddy Configuration: Standard setup for Odoo 18, proxying to 18069 and 18072 with necessary headers for websockets/longpolling.
  • Firewall (UFW): Ports 80, 443, 18069, 18072 are ALLOW IN. Mail ports (993, 995, 465, 587) are ALLOW OUT.
  • Cloudflare: A/CNAME records for web traffic are Proxied (orange cloud). Mail-related records are DNS Only (grey cloud). SSL/TLS is "Full". No custom WAF rules.
[options]
admin_passwd = ##################
; # Database connection details are set via environment variables in docker-compose.yml
; # Addons paths - Explicitly list subdirs like old server + Odoo standard path
addons_path = /mnt/extra-addons, /mnt/extra-addons/odooapps/, /mnt/extra-addons/industry/, /mnt/extra-addons/CybroAddons/, /mnt/extra-addons/social,/usr/lib/python3/dist-packages/odoo/addons
http_timeout = 1800
logfile = /var/log/odoo/odoo-server.log
logrotate = True
log_level = debug
longpolling_port = 8072
session_store = db
workers = 4
max_cron_threads = 2
proxy_mode = True
dbfilter = ^%h$
; # Database Connection Pool
;db_maxconn = 95
;web.base.url = https://coqui.cloud  ; <--- ADD/ENSURE THIS LINE (use your primary domain)
;web.base.url.freeze = True

Troubleshooting Steps Attempted and Ruled Out:

  • Caddy Configuration: Extensive testing and multiple Caddyfile iterations (including adding timeouts, explicit websocket directives) did not resolve the longpolling issue and sometimes caused Caddy to crash. Ruled out as primary cause.
  • Browser Caching: Hard refresh and clearing browser cache/cookies. Ruled out.
  • SSL/TLS Issues (from Caddy or Cloudflare): Direct HTTP access to Odoo's port 18069 still shows the longpolling issue. Ruled out.
  • Nginx/Nginx Proxy Manager: Attempts to switch were unsuccessful due to setup issues. Ruled out for now.
  • Odoo 15 Instances: Odoo 15 instances with similar Caddy configs work perfectly for real-time chat. Suggests issue is specific to Odoo 18.

Remaining Unresolved Issues & Hypotheses:

  1. RuntimeError: OrderedDict mutated during iteration (Root Cause of Cron Instability):
    • This is the most critical server-side error. It's a fundamental issue in Odoo's multi-database registry management.
    • Hypothesis: Likely a bug in Odoo 18's core or a specific module interaction causing database registry instability when cron threads iterate.
    • Specific Symptom: "Multiple password values not supported" from mail server logs for Fetchmail authentication, even after manual password re-typing and with empty mailboxes. This indicates Fetchmail is failing in an unusual way.
  2. Odoo 18 Real-time Chat Issue (Longpolling):
    • Hypothesis: Problem is definitively internal to Odoo 18's core longpolling/bus mechanism. Could be a subtle bug in Odoo 18's real-time stack, a conflict with a specific Odoo 18 module, or an interaction with its worker management when dbfilter is active. Logs at debug level show no errors, making diagnosis difficult.

Questions for the Community:

  1. For the RuntimeError: OrderedDict mutated during iteration: What are common causes and fixes for this specific Odoo error in multi-database setups (especially with dbfilter=^%h$)? Are there known Odoo 18 bugs or specific module interactions that cause this?
  2. For the Odoo 18 Real-time Chat issue: Given it's internal to Odoo and not Caddy/network, what are common causes for "Real-time connection lost..." when longpolling_port is listening and debug logs show no errors? Are there specific Odoo 18 modules known to interfere with bus or longpolling?
  3. For "Multiple password values not supported" (from mail server): What could cause Odoo 18 to send multiple password values during IMAP/POP3 authentication, even after manual re-typing and with simple passwords?

Any help or insights would be greatly appreciated. Thank you!

Avatar
Discard
Author Best Answer

After an extensive and challenging troubleshooting journey, I wanted to share the solutions we found for the persistent real-time chat issues and ir_cron database lock errors in our Odoo 18 multi-database Docker setup.

This was a complex problem with several interconnected root causes, but we've successfully stabilized the system.

Summary of the Problems:

  1. Real-time Chat (OdooBot/Discuss) Not Working: Messages required manual refresh.
  2. Recurring ir_cron Database Lock Errors: PostgreSQL logs showed ERROR: could not obtain lock on row in relation "ir_cron".
  3. Root Cause of ir_cron Errors: RuntimeError: OrderedDict mutated during iteration in Odoo logs, indicating database registry instability during cron job execution in a multi-database environment.

Solutions Found:

1. Fixing RuntimeError: OrderedDict mutated during iteration and ir_cron Locks (Server Stability)

This was the most critical server-side issue. The OrderedDict mutated error caused cron jobs to crash and hold database locks.

  • Diagnosis: The error occurred when Odoo's cron system iterated through active databases, and the registry unexpectedly changed. This was exacerbated by multiple problematic cron jobs (like Fetchmail and Sitemap generation) trying to run.
  • Solution:
    1. Deactivated ALL non-essential scheduled actions (cron jobs) in Odoo across all instances (e.g., "Mail: Fetchmail Service", "Create sitemap", "Recurring Invoice"). This was crucial to stop the OrderedDict mutation and ir_cron locks.
    2. Increased limit_time_real_cron in odoo.conf: Initially set to 2 seconds, which was too aggressive. Increasing it to 120 seconds (2 minutes) allowed cron jobs more time to complete without timing out.
      • odoo.conf setting: limit_time_real_cron = 120
    3. Systematic Reactivation: After achieving global stability (no errors with all non-essential cron jobs off), we systematically reactivated them one by one, troubleshooting specific mail server issues (e.g., "Multiple password values not supported" from mail server logs, which was a symptom of Fetchmail getting stuck) as they arose.
  • Result: The RuntimeError: OrderedDict mutated during iteration and ir_cron locking errors are now gone. The database is stable.

2. Fixing Odoo 18 Real-time Chat (Longpolling/WebSockets)

The real-time chat issue was internal to Odoo 18, not related to Caddy's basic proxying or network exposure.

  • Diagnosis:
    1. Direct IP Access Test: Accessing Odoo directly via http://144.126.135.5:18069 (bypassing Caddy/SSL) still showed "Real-time connection lost...", confirming the issue was Odoo-internal.
    2. GeoIP Crash: Odoo logs showed ValueError: '{http.request.remote.ip}' does not appear to be an IPv4 or IPv6 address. leading to KeyError: 'ir.http'. This happened because Caddy was passing a literal string instead of the actual client IP.
    3. Odoo 15 vs. Odoo 18: Odoo 15 instances worked, suggesting Odoo 18 had different requirements.
  • Solution:
    1. Corrected Caddy header_up directives: To pass the actual client IP from Cloudflare.
      • In Caddyfile, for both main and /websocket proxy blocks: header_up X-Real-IP {http.request.header.Cf-Connecting-Ip} header_up X-Forwarded-For {http.request.header.Cf-Connecting-Ip}
    2. Applied Comprehensive odoo.conf settings: This was the crucial step for Odoo 18's real-time stack.
      • odoo.conf settings (ensure these are present and active):
[options]
admin_passwd = ############
addons_path = /mnt/extra-addons,/usr/lib/python3/dist-packaes/odoo/addons
session_store = db
session_dir = False
limit_memory_hard = 2415919104
limit_memory_soft = 2013265920
limit_request = 8192
limit_time_cpu = 360
limit_real_time = 3600
limit_time_real_cron = 120
longpolling_port=8072
;logfile = /var/log/odoo/odoo-server.log
;logrotate = True
;log_level = debug
http_port =
;http_timeout = 1800
max_cron_threads = 2
workers = 5
websocket_keep_alive_timeout = 600
websocket_rate_limit_burst = 10
websocket_rate_limit_delay = 0.2
xmlrpc = True
xmlrpc_interface =
xmlrpc_port = 8069
xmlrpcs = True
xmlrpcs_interface =
xmlrpcs_port = 8071
proxy_mode = True
dbfilter = ^%h$

(Note: workers and max_cron_threads were also set, but their optimal values depend on server resources. We used workers = 3 and max_cron_threads = 1 from a working example, but later increased max_cron_threads to 2 for other reasons.)

  • Reult: Real-time chat (OdooBot/Discuss) is now working perfectly!

Final Caddyfile Snippet (Only tested with Odoo 17, 18 Instances)

# --- Odoo 18 Instances (Main Group) ---
your.domain.com, www.your.domain.com, other.domain.com { # Your list of domains
    encode zstd gzip

    reverse_proxy http://127.0.0.1:18069 { # Main Odoo port
        header_up X-Real-IP {http.request.header.Cf-Connecting-Ip}
        header_up X-Forwarded-For {http.request.header.Cf-Connecting-Ip}
        header_up Host {host}
        header_up X-Forwarded-Host {host}
        header_up X-Forwarded-Proto {scheme}
    }
    # Odoo WebSocket / Longpolling traffic
    reverse_proxy /websocket {
        to http://127.0.0.1:18072 # Longpolling port
        header_up Host {host}
        header_up X-Real-IP {http.request.header.Cf-Connecting-Ip}
        header_up X-Forwarded-For {http.request.header.Cf-Connecting-Ip}
        header_up X-Forwarded-Host {host}
        header_up X-Forwarded-Proto {scheme}
        header_up Upgrade {http.request.header.Upgrade}
        header_up Connection {http.request.header.Connection}
    }
}

This was a challenging but ultimately successful journey. I hope this detailed breakdown helps others facing similar complex Odoo 18 issues. We've created a full Blog Post about this experience with more details.
                 
 Epic Journey Through Real-time Chat & Cron Job

Best regards,

Ramon Rios Jr. 

Coqui Cloud

Avatar
Discard

Thanks for the update - where did you find a docker with limit_time_real_cron set to 2 seconds?

Author

I found it in another blog post and I assumed it was 2 for 2 mins, shortly after applying those settings I started receiving massive amounts of errors within seconds

Related Posts Replies Views Activity
1
Dec 22
4088
1
Mar 22
3443
0
Nov 17
3771
1
Jul 25
447
4
Apr 25
4486