Skip to Content
Odoo Menu
  • Sign in
  • Try it free
  • Apps
    Finance
    • Accounting
    • Invoicing
    • Expenses
    • Spreadsheet (BI)
    • Documents
    • Sign
    Sales
    • CRM
    • Sales
    • POS Shop
    • POS Restaurant
    • Subscriptions
    • Rental
    Websites
    • Website Builder
    • eCommerce
    • Blog
    • Forum
    • Live Chat
    • eLearning
    Supply Chain
    • Inventory
    • Manufacturing
    • PLM
    • Purchase
    • Maintenance
    • Quality
    Human Resources
    • Employees
    • Recruitment
    • Time Off
    • Appraisals
    • Referrals
    • Fleet
    Marketing
    • Social Marketing
    • Email Marketing
    • SMS Marketing
    • Events
    • Marketing Automation
    • Surveys
    Services
    • Project
    • Timesheets
    • Field Service
    • Helpdesk
    • Planning
    • Appointments
    Productivity
    • Discuss
    • Approvals
    • IoT
    • VoIP
    • Knowledge
    • WhatsApp
    Third party apps Odoo Studio Odoo Cloud Platform
  • Industries
    Retail
    • Book Store
    • Clothing Store
    • Furniture Store
    • Grocery Store
    • Hardware Store
    • Toy Store
    Food & Hospitality
    • Bar and Pub
    • Restaurant
    • Fast Food
    • Guest House
    • Beverage Distributor
    • Hotel
    Real Estate
    • Real Estate Agency
    • Architecture Firm
    • Construction
    • Estate Management
    • Gardening
    • Property Owner Association
    Consulting
    • Accounting Firm
    • Odoo Partner
    • Marketing Agency
    • Law firm
    • Talent Acquisition
    • Audit & Certification
    Manufacturing
    • Textile
    • Metal
    • Furnitures
    • Food
    • Brewery
    • Corporate Gifts
    Health & Fitness
    • Sports Club
    • Eyewear Store
    • Fitness Center
    • Wellness Practitioners
    • Pharmacy
    • Hair Salon
    Trades
    • Handyman
    • IT Hardware & Support
    • Solar Energy Systems
    • Shoe Maker
    • Cleaning Services
    • HVAC Services
    Others
    • Nonprofit Organization
    • Environmental Agency
    • Billboard Rental
    • Photography
    • Bike Leasing
    • Software Reseller
    Browse all Industries
  • Community
    Learn
    • Tutorials
    • Documentation
    • Certifications
    • Training
    • Blog
    • Podcast
    Empower Education
    • Education Program
    • Scale Up! Business Game
    • Visit Odoo
    Get the Software
    • Download
    • Compare Editions
    • Releases
    Collaborate
    • Github
    • Forum
    • Events
    • Translations
    • Become a Partner
    • Services for Partners
    • Register your Accounting Firm
    Get Services
    • Find a Partner
    • Find an Accountant
    • Meet an advisor
    • Implementation Services
    • Customer References
    • Support
    • Upgrades
    Github Youtube Twitter Linkedin Instagram Facebook Spotify
    +1 (650) 691-3277
    Get a demo
  • Pricing
  • Help

Odoo is the world's easiest all-in-one management software.
It includes hundreds of business apps:

  • CRM
  • e-Commerce
  • Accounting
  • Inventory
  • PoS
  • Project
  • MRP
All apps
You need to be registered to interact with the community.
All Posts People Badges
Tags (View all)
odoo accounting v14 pos v15
About this forum
You need to be registered to interact with the community.
All Posts People Badges
Tags (View all)
odoo accounting v14 pos v15
About this forum
Help

Cron job showing error: RuntimeError: OrderedDict mutated during iteration

Subscribe

Get notified when there's activity on this post

This question has been flagged
cronbugcronjobv14
1 Reply
8540 Views
Avatar
Dries Cox

Hello Community, 

I have a problem with the cron job that stops on a regular basis running after a while. I have been looking into the logs and I came across this error:

RuntimeError: OrderedDict mutated during iteration

Full log:

2022-03-28 12:26:57,168 344737 INFO test-server: xxx\.xxx\.xxx\.xxx\ \-\ \-\ \[28\ Mar2022\ 12:26:57\\]\\ "POST\\ /longpolling/poll\\ HTTP/1\\.0"\\ 200\\ \\-\\ 8\\ 0\\.002\\ 50\\.007
Exception\ in\ thread\ odoo\.service\.cron\.cron1:
Traceback\ \(most\ recent\ call\ last\):
\\ \\ File\\ "/usr/lib/python3\\.9/threading\\.py",\\ line\\ 954,\\ in\\ _bootstrap_inner
2022\\-03\\-28\\ 12:26:59,512\\ 344737\\ INFO\\ test\\-server\\ werkzeug:\\ 84\\.192\\.32\\.235\\ \\-\\ \\-\\ \\[28/Mar/2022\\ 12:26:59\\]\\ "POST\\ /longpolling/poll\\ HTTP/1\\.0"\\ 200\\ \\-\\ 8\\ 0\\.003\\ 50\\.009
\\ \\ \\ \\ self\\.run\\(\\)
\\ \\ File\\ "/usr/lib/python3\\.9/threading\\.py",\\ line\\ 892,\\ in\\ run
\\ \\ \\ \\ self\\._target\\(\\*self\\._args,\\ \\*\\*self\\._kwargs\\)
\\ \\ File\\ "/opt/odoo14/odoo/odoo/service/server\\.py",\\ line\\ 432,\\ in\\ target
\\ \\ \\ \\ self\\.cron_thread\\(i\\)
\\ \\ File\\ "/opt/odoo14/odoo/odoo/service/server\\.py",\\ line\\ 408,\\ in\\ cron_thread
\\ \\ \\ \\ for\\ db_name,\\ registry\\ in\\ registries\\.d\\.items\\(\\):
RuntimeError:\\ OrderedDict\\ mutated\\ during\\ iteration

This\\ error\\ happens\\ when\\ the\\ cron\\ job\\ runs\\.

I\\ found\\ this\\ website:\\ https://chowdera.com/2021/09/20210913213439937A.html

Explaining that the problem is related to the python version. Python >=3 has this problem. But oodo14 requests python >=3.6

They suggest a solution, in server.py  in the function

    def cron_thread(self, number):

change the line

for db_name, registry in registries.d.items():
into
for db_name, registry in list(registries.d.items()):
This sounds reasonable, but I wonder why this isn't done by odoo itself?
Could this be the solution? Is this an unreported bug? Or is there something else that is the cause of this?
Thanks for your reply.




2
Avatar
Discard
Faris Fathurrahman

my customer bumped into this similar problem, I wonder why, even this post still hasn't got any replies

José Moreno

I've been struggling with this problem myself. It kills the cron jobs after it happens twice (assuming it's because default cron workers is 2). Going to try your solution

Avatar
Reinhard Mengelberg
Best Answer

Subject: Solution for RuntimeError: OrderedDict mutated during iteration in Odoo 17 CRON Jobs

I ran into the same issue on Odoo 17 on-premise. The issue lies in the registries.d being modified while being iterated over by the cron_thread method.

Initially, I thought that registries.d would only be modified upon creation, editing, or deleting of a database. However, this is not the case. The registries.d is updated more frequently, for example, when you install a new module on a database.

Upon investigating the /odoo/modules/registry.py file, I found that there are multiple threads or processes accessing and modifying the registry concurrently. 

This concurrent access can lead to the observed behavior, especially if there are background jobs or operations interacting with the registry while CRON jobs are running.

If you run CRON jobs very often, as I do, it is almost inevitable that you will encounter this error at some point. 

The solution suggested by Dries should work. It involves modifying the cron_thread method to iterate over a list copy of registries.d.items(). This prevents the iteration from being disrupted by concurrent modifications.

Here’s the change I’m going to apply:

  1. Locate the cron_thread method in server.py:
    def cron_thread(self, number):
        from odoo.addons.base.models.ir_cron import ir_cron
        conn = odoo.sql_db.db_connect('postgres')
        with conn.cursor() as cr:
            pg_conn = cr._cnx
            cr.execute("SELECT pg_is_in_recovery()")
            in_recovery = cr.fetchone()[0]
            if not in_recovery:
                cr.execute("LISTEN cron_trigger")
            else:
                _logger.warning("PG cluster in recovery mode, cron trigger not activated")
            cr.commit()
    
            while True:
                select.select([pg_conn], [], [], SLEEP_INTERVAL + number)
                time.sleep(number / 100)
                pg_conn.poll()
    
                registries = odoo.modules.registry.Registry.registries
                _logger.debug('cron%d polling for jobs', number)
    
                # Create a copy of the items to avoid mutation during iteration
                for db_name, registry in list(registries.d.items()):
                    if registry.ready:
                        thread = threading.current_thread()
                        thread.start_time = time.time()
                        try:
                            ir_cron._process_jobs(db_name)
                        except Exception:
                            _logger.warning('cron%d encountered an Exception:', number, exc_info=True)
                        finally:
                            thread.start_time = None
    
  2. Restart Odoo:
    sudo systemctl restart odoo
    
  3. Monitor the logs to ensure the error does not reoccur and that CRON jobs are running smoothly:
    sudo tail -f /var/log/odoo/odoo-server.log
    

By making this change, I hope to prevent the OrderedDict mutated during iteration error and ensure that my CRON jobs run without interruption. 

If you encounter similar issues, this approach should help.

Best of luck!


1
Avatar
Discard
Enjoying the discussion? Don't just read, join in!

Create an account today to enjoy exclusive features and engage with our awesome community!

Sign up
Related Posts Replies Views Activity
Cron job does not execute python code on scheduler, only on Run Manually button. Solved
cron cronjob
Avatar
Avatar
Avatar
3
Sep 23
12902
Why does the automated scheduling occasionally stop running?
cronjob v14 Scheduled
Avatar
Avatar
1
Oct 25
3054
Using CRON job to download and attach pdf to binary field ODOO16
cron cronjob ODOO16
Avatar
0
Oct 24
3137
Cron getting timeout
cron timeout cronjob
Avatar
Avatar
1
Aug 24
6183
Possible Bug with Google Maps extension on v14 Solved
bug v14 googlemaps
Avatar
Avatar
Avatar
Avatar
4
May 22
5869
Community
  • Tutorials
  • Documentation
  • Forum
Open Source
  • Download
  • Github
  • Runbot
  • Translations
Services
  • Odoo.sh Hosting
  • Support
  • Upgrade
  • Custom Developments
  • Education
  • Find an Accountant
  • Find a Partner
  • Become a Partner
About us
  • Our company
  • Brand Assets
  • Contact us
  • Jobs
  • Events
  • Podcast
  • Blog
  • Customers
  • Legal • Privacy
  • Security
الْعَرَبيّة Català 简体中文 繁體中文 (台灣) Čeština Dansk Nederlands English Suomi Français Deutsch हिंदी Bahasa Indonesia Italiano 日本語 한국어 (KR) Lietuvių kalba Język polski Português (BR) română русский язык Slovenský jazyk slovenščina Español (América Latina) Español ภาษาไทย Türkçe українська Tiếng Việt

Odoo is a suite of open source business apps that cover all your company needs: CRM, eCommerce, accounting, inventory, point of sale, project management, etc.

Odoo's unique value proposition is to be at the same time very easy to use and fully integrated.

Website made with

Odoo Experience on YouTube

1. Use the live chat to ask your questions.
2. The operator answers within a few minutes.

Live support on Youtube
Watch now