Problem: Odoo CRON Jobs not running due to RuntimeError

This is an issue some people struggle with when hosting on-premise databases. To check the status of your server and diagnose this issue, follow these steps:

Check Server Status

1. Run the following command in bash:

   systemctl status odoo
   

2. Look for Errors in the Output: If you encounter something like this, you have run into the problem:

   May 14 13:30:46 mengelmann-odoo-1 odoo[21985]: Traceback (most recent call last):
   May 14 13:30:46 mengelmann-odoo-1 odoo[21985]:   File "/usr/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
   May 14 13:30:46 mengelmann-odoo-1 odoo[21985]:     self.run()
   May 14 13:30:46 mengelmann-odoo-1 odoo[21985]:   File "/usr/lib/python3.10/threading.py", line 953, in run
   May 14 13:30:46 mengelmann-odoo-1 odoo[21985]:     self._target(*self._args, **self._kwargs)
   May 14 13:30:46 mengelmann-odoo-1 odoo[21985]:   File "/usr/lib/python3/dist-packages/odoo/service/server.py", line 497, in target
   May 14 13:30:46 mengelmann-odoo-1 odoo[21985]:     self.cron_thread(i)
   May 14 13:30:46 mengelmann-odoo-1 odoo[21985]:   File "/usr/lib/python3/dist-packages/odoo/service/server.py", line 473, in cron_thread
   May 14 13:30:46 mengelmann-odoo-1 odoo[21985]:     for db_name, registry in registries.d.items():
   May 14 13:30:46 mengelmann-odoo-1 odoo[21985]: RuntimeError: OrderedDict mutated during iteration
   

You've encountered a common problem in Odoo: RuntimeError: OrderedDict mutated during iteration.

Explanation

I encountered this issue on Odoo 17 on-premise. The problem arises because the registries.d dictionary is being modified while being iterated over by the cron_thread method. Initially, I assumed registries.d would only be altered when creating, editing, or deleting a database. However, this is not the case. The registries.d gets updated more frequently, for example, when installing a new module on a database.

Upon investigating the /odoo/modules/registry.py file, I found that multiple threads or processes are accessing and modifying the registry concurrently. This concurrent access can lead to the observed behavior, especially if there are background jobs or operations interacting with the registry while CRON jobs are running.

If you run CRON jobs very often, as I do, it is almost inevitable that you will encounter this error at some point. The solution suggested by Dries works effectively. It involves modifying the cron_thread method to iterate over a list copy of registries.d.items(). This prevents the iteration from being disrupted by concurrent modifications.

Steps to Implement the Solution

1. Locate the cron_thread method in server.py (path to file for packaged installations: /usr/lib/python3/dist-packages/odoo/service/server.py):

   def cron_thread(self, number):
       from odoo.addons.base.models.ir_cron import ir_cron
       conn = odoo.sql_db.db_connect('postgres')
       with conn.cursor() as cr:
           pg_conn = cr._cnx
           cr.execute("SELECT pg_is_in_recovery()")
           in_recovery = cr.fetchone()[0]
           if not in_recovery:
               cr.execute("LISTEN cron_trigger")
           else:
               _logger.warning("PG cluster in recovery mode, cron trigger not activated")
           cr.commit()
   
           while True:
               select.select([pg_conn], [], [], SLEEP_INTERVAL + number)
               time.sleep(number / 100)
               pg_conn.poll()
   
               registries = odoo.modules.registry.Registry.registries
               _logger.debug('cron%d polling for jobs', number)
   
               # Create a copy of the items to avoid mutation during iteration
               for db_name, registry in list(registries.d.items()):
                   if registry.ready:
                       thread = threading.current_thread()
                       thread.start_time = time.time()
                       try:
                           ir_cron._process_jobs(db_name)
                       except Exception:
                           _logger.warning('cron%d encountered an Exception:', number, exc_info=True)
                       finally:
                           thread.start_time = None
   

2. Restart Odoo:

   systemctl restart odoo
   

3. Monitor the logs to ensure the error does not reoccur and that CRON jobs are running smoothly:

   tail -f /var/log/odoo/odoo-server.log
   

By making this change, I hope to prevent the OrderedDict mutated during iteration error and ensure that my CRON jobs run without interruption. If you encounter similar issues, this approach should help.

Best of luck!