This question has been flagged
2 Replies
2855 Views

I am trying to import 30,000 products in openerp as well as I have to import category, products related supplier also. For this process I have made an addon. But it takes more than 7hours to import. I have used ORM objects only. 

Note : I have a lot of validation while importing like as file value checking, special character removing and etc... That's why I have created the addon.

Is any otherway to reduce the timing and increase the performance? I have to do more validation on my script otherwise I will use openerp import functionality itself or some other tools. Please give me instruction to increase the performance.

Avatar
Discard
Best Answer

I would go for your custom module. The standard import functions might take even longer than your custom module as they process many different cases. The only things that MIGHT help is multi-threading your custom module or optimizing code/server configuration.

Avatar
Discard
Author

Thanks for your advice Michael. In my code i have checked every single line of the csv or xls file and make it as list of dictionaries and check the conditions and import single product after importing the single product that i am importing the product related suppliers and then insert another product. If we insert the product one by one, is it taking time to import? Can i use the queries rather than ORM functionality?

While it is certainly possible to use raw querries, using ORM objects is preferable as there might be some unforseen impacts when manipulating the DB directly (Unknown impacts, different modules, etc...). There might also be future changes that might break your import method. However, if you intend to go forward with this kind of scenario, here is how I would suggest you proceed: - Backup database - Import a few objects using ORM method (try to pick a set that goes trought all edge cases you could think of) - Backup this version - Rollback to original backup - Import the same objects using database method - Backup this version - Compare the results from both import methods Doing so, ensures you process the objects closely enough to the original ORM model and that you haven't forgotten anything in the process. The pgsql backups being text based SQL dumps, they sould be easy enough to compare.

Best Answer

Maybe you could try to sanitize the data before inserting it into OpenERP? If you can run a script that validates the records and then inserts them using simple "create" method calls on XML-RPC it should take you no more than an hour or so.

Avatar
Discard