Community mailing list archives

community@mail.odoo.com

Re: About Attachment File Location (again)

by
Anybox, Christophe Combelles
- 02/03/2015 07:45:57

Le 3 février 2015 12:32:57 EET, David Arnold <dar@devco.co> a écrit :
>I though that (storing attachemnets in the db) is the default [v8] .
>but following this thread so far puts me in doubts a little
>can someone clarify for dummies? Also if it is not the default and the
>recommendation would be to have attachments stored in the db ... how do
>you achieve that 
>http://www.postgresql.org/docs/9.4/static/largeobjects.html [1]
>The keyword in the introduction is "special", arge objects are a
>special facility of Postgres just made for this use case. So far this
>facility never has been touched by odoo. (apart from Christophes
>implementation - kudos!) Stored in databse meant so far as base64

I personally didn't write a line of this module, it has been done by our team to achieve reliable attachment storage on a large project. But it still needs improvements, such as avoiding the base64 encoding/decoding which is unuseful (and maybe comes from the old client/server xmlrpc architecture before 6.0) and also avoiding to raise the whole attachment in memory. For media oriented client data with +100MB attachments, that would be the very first thing to improve.



>encoded objects in normal data table fields. This indeed has been even
>more madness :)     None [2]
>*Saludos Cordiales*   David Arnold    ​ [3]    David Arnold BA HSG
> / Analista
>315 304 13 68/  dar@devco.co [4]
>*devCO - empresa de consultoría de sistemas (en fundación)*
>http://www.devco.co [5]
>This e-mail message may contain confidential or legally privileged
>information and is intended only for the use of the intended
>recipient(s). Any unauthorized disclosure, dissemination, distribution,
>copying or the taking of any action in reliance on the information
>herein is prohibited. E-mails are not secure and cannot be guaranteed
>to be error free as they can be intercepted, amended, or contain
>viruses. Anyone who communicates with us by e-mail is deemed to have
>accepted these risks. devCO is not responsible for errors or omissions
>in this message and denies any responsibility for any damage arising
>from the use of e-mail. Any opinion and other statement contained in
>this message and any attachment are solely those of the author and do
>not necessarily represent those of the company.
>
>2015-02-03 5:22 GMT-05:00 Gunnar Wagner gunnar.wagner@irisgermanica.com [6] > :
>
>On 2/3/2015 6:12 PM, David Arnold
>wrote:
>
>
cite="mid:CAOLEt-F2Farq= >9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com [7] " >type="cite"> >Cristophe has a very stringent and knowledgable >line of thought. I also see *all* counter arguments invalidated. >Unless the local NAS which isn't served over VPN, but strictly >locally (by ethernet). But if it's a local setup, why not just >have the Databse local as well and reduce entropy in the >arguments. Here have been apples mixed with banana... > >I'd >recommend Large Objects as the default. > >I though that (storing attachemnets in the db) is the default [v8] . >but following this thread so far puts me in doubts a little >can someone clarify for dummies? Also if it is not the default and >the recommendation would be to have attachments stored in the db ... >how do you achieve that > > > > > > >
cite="mid:CAOLEt-F2Farq= >9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com [8] " >type="cite"> > >2015-02-03 3:42 GMT-05:00 Christophe >Combelles < ccomb@anybox.fr [9] > : > >Le 3 février 2015 04:08:15 EET, Phui Hock < phuihock@codekaki.com [10] >> a écrit : >>I regret the best theorical solution is rarely the default in Odoo. >>Practicality >>is always dependent on the context and you should rather let people >>choose what >>is practical for them. Hopefully it's easy to migrate data from the >>default >>storage to large objects. >>> >>> LOBs: >>> >>> - transcationals >>« transactional » means much more than a single word >>LOBs also : >>- are easier to setup >>- are more reliable >>- are replicable >>- are consistent (and not "eventually" consistent...) >>- can benefit from postgresql tools regarding backups, replication, >>etc. >>> >>> but >>> >>> - bloat the replication stream >>You need to transfer the file anyway. At least, there is only one >>stream to >>manage. No need to set up an rsync or anything else. Depending on the >>type of clients, saving binaries to database may be not acceptable at >>all. I have a client who is a media house and 90% of their files are >>several hundreds MB in size. One of their primary requirements is to >>have these files backup and accessible on their NAS-RAID. If the files >>are saved in the database, >How do you keep a total sync between what is in the NAS and the >ir.attachment metadata in the database? > >>it would incur unnecessary database overhead >>every time the files need to be accessed or manipulated. >With the default 7.0 storage maybe, with the blob there are no more >overhead than just reading a file from disk. > >>For instance, we needed to generate previews and thumbnails for each >of >>the known types. We also need to add watermark on-the-fly. These type >>of operations are easier and faster when files are saved on local >>filesystem. >It seems easier in the first place (but not faster). It's easier to >modify a file in the filesystem with a filesystem tool indeed, but it's >harder to do it while keeping real sync with what is in the database >anyway. And eventually I would bet it's easier in the long run to rely >on the database security and transactionality. >It all depend on the importance you give to your data. > >>Having these binaries saved on local filesystem also benefit from >>quicker streaming when served via web server such as nginx. >This is totally wrong, you can stream large blobs directly from the >database do the browser, even PHP offers such features. And regarding >small files they are supposed to be correctly cached anyway so they end >up being never served by the filesystem nor the database. >>> - esay to browse by a human >>I'm not sure humans are good at browsing such SHA1 trees : >>│ ├── 57 >>│ │ └── 57d4a8c7b652d6928d791c458cb04a246e3fb1ba >>│ ├── dc >>│ │ └── dcf00aacce882bbfd117c0277e514f829b4c5bf0 >>│ └── ef >>│ └── ef2c882a36dbe90fc1e7e28d816ad1ac1464cfbb >>└── v8 >>├── 35 >>│ └── 358d08bafbecf102728c217bcc3eb0 a24647443c >>├── 36 >>While I agree these are not human readable, I don't think it is hard >>too to derive another filestore that overrides SHA1 filename with >>actual file name. >I believe something similar was done in old 6.1 or 7.0 modules that >were abandonned due to their poor quality. Repoducing that with the >filestore would make people believe they can use and modify it as real >files which is wrong unfortunately, unless you maintain the sha1 >filename and directory manually... > >>_______________________________________________ >>Mailing-List: https://www.odoo.com/groups/community-59 [11] >>Post to: mailto: community@mail.odoo.com [12] >>Unsubscribe: https://www.odoo.com/groups?unsubscribe [13] > > >_______________________________________________ >Mailing-List: https://www.odoo.com/groups/community-59 [14] >Post to: mailto: community@mail.odoo.com [15] >Unsubscribe: https://www.odoo.com/groups?unsubscribe [16] > > > > > > >_______________________________________________ >Mailing-List: https://www.odoo.com/groups/community-59 [17] >Post to: mailto:community@mail.odoo.com [18] >Unsubscribe: https://www.odoo.com/groups?unsubscribe [19] > > > >-- > > > >Gunnar >Wagner | Iris Germanica Ltd. | JinQian Gong Lu 385, 8-201 >| FengXian >Qu, 201404 Shanghai | P.R. CHINA >+86 >159 0094 1702 | +49 (0)176 7808 9090 [20] | skype: >professorgunrad | >None [21] www.fashionsupermarket.net > > >_______________________________________________ >Mailing-List: https://www.odoo.com/groups/community-59 [22] >Post to: mailto: community@mail.odoo.com [23] >Unsubscribe: https://www.odoo.com/groups?unsubscribe [24] > > >_______________________________________________ >Mailing-List: https://www.odoo.com/groups/community-59 >Post to: mailto:community@mail.odoo.com >Unsubscribe: https://www.odoo.com/groups?unsubscribe > > > >[1] http://www.postgresql.org/docs/9.4/static/largeobjects.html >[2] http://www.elaleman.co/ >[3] http://www.elaleman.co/ >[4] mailto:dar@devco.co >[5] http://www.devco.co >[6] mailto:gunnar.wagner@irisgermanica.com >[7] mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com >[8] mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com >[9] mailto:ccomb@anybox.fr >[10] mailto:phuihock@codekaki.com >[11] https://www.odoo.com/groups/community-59 >[12] mailto:community@mail.odoo.com >[13] https://www.odoo.com/groups?unsubscribe >[14] https://www.odoo.com/groups/community-59 >[15] mailto:community@mail.odoo.com >[16] https://www.odoo.com/groups?unsubscribe >[17] https://www.odoo.com/groups/community-59 >[18] mailto:community@mail.odoo.com >[19] https://www.odoo.com/groups?unsubscribe >[20] tel:%2B49 %280%29176 7808 9090 >[21] http://www.fashionsupermarket.net >[22] https://www.odoo.com/groups/community-59 >[23] mailto:community@mail.odoo.com >[24] https://www.odoo.com/groups?unsubscribe