Odoo Help

Welcome!

This community is for beginners and experts willing to share their Odoo knowledge. It's not a forum to discuss ideas, but a knowledge base of questions and their answers.

1

[FIXED] CSV Product Import Issues. Import preview failed due to: field larger than field limit (131072).

By
Luke Branch
on 7/11/14, 6:24 AM 4,945 views

[UPDATE 2015-03-01]

This issue has been fixed in the following PR:

https://github.com/odoo/odoo/commit/68f14c68709bbb50cb7fb66d288955e1d769c5ff


I've been trying to figure out a way to import product images effectively into our database and have been running into the same issues as most in this forum (ie. red status notification with no text inside). I therefore slowly started trying to validate imports using larger product field sets and pictures every time (started with 5 and moved on to 20 at a time with larger images).

I then came across this error:

Import preview failed due to: field larger than field limit (131072). The issue is usually an incorrect file encoding.

I have googled the error and come across this article regarding the error in python:

http://lethain.com/handling-very-large-csv-and-xml-files-in-python/

and a possible solution in this stackoverflow post:

http://stackoverflow.com/questions/15063936/csv-error-field-larger-than-field-limit-131072

I've then backed up the following file:

/opt/odoo/odoo-server/addons/base_import/models.py

and added the following to the top of the file:

import sys
import csv
maxInt = sys.maxsize
decrement = True

while decrement:
    # decrease the maxInt value by factor 10 
    # as long as the OverflowError occurs.

    decrement = False
    try:
        csv.field_size_limit(maxInt)
    except OverflowError:
        maxInt = int(maxInt/10)
        decrement = True

and then I have restarted the server. I understand that modifiying core code is not a smart idea, so if it works I will post back here as an answer to confirm, however does anyone know how this might be implemented as a module to override the default setting rather than modifying the core files?

0
Snehal Chauhan
On 11/21/14, 7:06 AM

above solution does not work on windows server, any suggestion

@Snehal, I think this solution should be platform independent. Have you tried restarting your odoo-server, or running an odoo-server --update=all when starting (make sure you're fully backed up first)?

Luke Branch
on 11/21/14, 9:47 PM
0

Luke Branch

--Luke Branch--
1531
| 6 7 8
Mui Wo, Lantau, Hong Kong
--Luke Branch--

...

Luke Branch
On 7/11/14, 8:22 AM

The fix above definitely works. I have just finished an import with thousands of large images inside a CSV.

  1. make a backup of /opt/odoo/odoo-server/addons/base_import/models.py (or whereever /addons/base_import/ is located in your instance).
  2. add the following lines at the very top of the file:
    1. import sys
      import csv
      maxInt = sys.maxsize
      decrement = True

      while decrement:
          # decrease the maxInt value by factor 10 
          # as long as the OverflowError occurs.

          decrement = False
          try:
              csv.field_size_limit(maxInt)
          except OverflowError:
              maxInt = int(maxInt/10)
              decrement = True

  3. Restart the odoo server.

The issue was resolved due to this stackoverflow post:

http://stackoverflow.com/questions/15063936/csv-error-field-larger-than-field-limit-131072

This needs to be packaged into a module though as this method relies on editing core files to achieve a solution.

Your Answer

Please try to give a substantial answer. If you wanted to comment on the question or answer, just use the commenting tool. Please remember that you can always revise your answers - no need to answer the same question twice. Also, please don't forget to vote - it really helps to select the best questions and answers!

About This Community

This community is for professionals and enthusiasts of our products and services. Read Guidelines

Question tools

1 follower(s)

Stats

Asked: 7/11/14, 6:24 AM
Seen: 4945 times
Last updated: 3/16/15, 8:10 AM