Skip to Content
Menu
This question has been flagged
1 Reply
6535 Views

Hello Community, 

I have a problem with the cron job that stops on a regular basis running after a while. I have been looking into the logs and I came across this error:

RuntimeError: OrderedDict mutated during iteration

Full log:

2022-03-28 12:26:57,168 344737 INFO test-server: xxx\.xxx\.xxx\.xxx\ \-\ \-\ \[28\ Mar2022\ 12:26:57\\]\\ "POST\\ /longpolling/poll\\ HTTP/1\\.0"\\ 200\\ \\-\\ 8\\ 0\\.002\\ 50\\.007
Exception\ in\ thread\ odoo\.service\.cron\.cron1:
Traceback\ \(most\ recent\ call\ last\):
\\ \\ File\\ "/usr/lib/python3\\.9/threading\\.py",\\ line\\ 954,\\ in\\ _bootstrap_inner
2022\\-03\\-28\\ 12:26:59,512\\ 344737\\ INFO\\ test\\-server\\ werkzeug:\\ 84\\.192\\.32\\.235\\ \\-\\ \\-\\ \\[28/Mar/2022\\ 12:26:59\\]\\ "POST\\ /longpolling/poll\\ HTTP/1\\.0"\\ 200\\ \\-\\ 8\\ 0\\.003\\ 50\\.009
\\ \\ \\ \\ self\\.run\\(\\)
\\ \\ File\\ "/usr/lib/python3\\.9/threading\\.py",\\ line\\ 892,\\ in\\ run
\\ \\ \\ \\ self\\._target\\(\\*self\\._args,\\ \\*\\*self\\._kwargs\\)
\\ \\ File\\ "/opt/odoo14/odoo/odoo/service/server\\.py",\\ line\\ 432,\\ in\\ target
\\ \\ \\ \\ self\\.cron_thread\\(i\\)
\\ \\ File\\ "/opt/odoo14/odoo/odoo/service/server\\.py",\\ line\\ 408,\\ in\\ cron_thread
\\ \\ \\ \\ for\\ db_name,\\ registry\\ in\\ registries\\.d\\.items\\(\\):
RuntimeError:\\ OrderedDict\\ mutated\\ during\\ iteration

This\\ error\\ happens\\ when\\ the\\ cron\\ job\\ runs\\.

I\\ found\\ this\\ website:\\ https://chowdera.com/2021/09/20210913213439937A.html

Explaining that the problem is related to the python version. Python >=3 has this problem. But oodo14 requests python >=3.6

They suggest a solution, in server.py  in the function

    def cron_thread(self, number):

change the line

for db_name, registry in registries.d.items():
into
for db_name, registry in list(registries.d.items()):
This sounds reasonable, but I wonder why this isn't done by odoo itself?
Could this be the solution? Is this an unreported bug? Or is there something else that is the cause of this?
Thanks for your reply.




Avatar
Discard

my customer bumped into this similar problem, I wonder why, even this post still hasn't got any replies

I've been struggling with this problem myself. It kills the cron jobs after it happens twice (assuming it's because default cron workers is 2). Going to try your solution

Best Answer

Subject: Solution for RuntimeError: OrderedDict mutated during iteration in Odoo 17 CRON Jobs

I ran into the same issue on Odoo 17 on-premise. The issue lies in the registries.d being modified while being iterated over by the cron_thread method.

Initially, I thought that registries.d would only be modified upon creation, editing, or deleting of a database. However, this is not the case. The registries.d is updated more frequently, for example, when you install a new module on a database.

Upon investigating the /odoo/modules/registry.py file, I found that there are multiple threads or processes accessing and modifying the registry concurrently. 

This concurrent access can lead to the observed behavior, especially if there are background jobs or operations interacting with the registry while CRON jobs are running.

If you run CRON jobs very often, as I do, it is almost inevitable that you will encounter this error at some point. 

The solution suggested by Dries should work. It involves modifying the cron_thread method to iterate over a list copy of registries.d.items(). This prevents the iteration from being disrupted by concurrent modifications.

Here’s the change I’m going to apply:

  1. Locate the cron_thread method in server.py:
    def cron_thread(self, number):
        from odoo.addons.base.models.ir_cron import ir_cron
        conn = odoo.sql_db.db_connect('postgres')
        with conn.cursor() as cr:
            pg_conn = cr._cnx
            cr.execute("SELECT pg_is_in_recovery()")
            in_recovery = cr.fetchone()[0]
            if not in_recovery:
                cr.execute("LISTEN cron_trigger")
            else:
                _logger.warning("PG cluster in recovery mode, cron trigger not activated")
            cr.commit()
    
            while True:
                select.select([pg_conn], [], [], SLEEP_INTERVAL + number)
                time.sleep(number / 100)
                pg_conn.poll()
    
                registries = odoo.modules.registry.Registry.registries
                _logger.debug('cron%d polling for jobs', number)
    
                # Create a copy of the items to avoid mutation during iteration
                for db_name, registry in list(registries.d.items()):
                    if registry.ready:
                        thread = threading.current_thread()
                        thread.start_time = time.time()
                        try:
                            ir_cron._process_jobs(db_name)
                        except Exception:
                            _logger.warning('cron%d encountered an Exception:', number, exc_info=True)
                        finally:
                            thread.start_time = None
    
  2. Restart Odoo:
    sudo systemctl restart odoo
    
  3. Monitor the logs to ensure the error does not reoccur and that CRON jobs are running smoothly:
    sudo tail -f /var/log/odoo/odoo-server.log
    

By making this change, I hope to prevent the OrderedDict mutated during iteration error and ensure that my CRON jobs run without interruption. 

If you encounter similar issues, this approach should help.

Best of luck!


Avatar
Discard
Related Posts Replies Views Activity
3
Sep 23
10953
0
Oct 24
1643
1
Aug 24
4612
1
Apr 24
1480
4
May 22
4294