Community mailing list archives

Re: About Attachment File Location (again)

Anybox, Christophe Combelles
- 02/03/2015 07:45:57

Le 3 février 2015 12:32:57 EET, David Arnold <> a écrit :
>I though that (storing attachemnets in the db) is the default [v8] .
>but following this thread so far puts me in doubts a little
>can someone clarify for dummies? Also if it is not the default and the
>recommendation would be to have attachments stored in the db ... how do
>you achieve that 
> [1]
>The keyword in the introduction is "special", arge objects are a
>special facility of Postgres just made for this use case. So far this
>facility never has been touched by odoo. (apart from Christophes
>implementation - kudos!) Stored in databse meant so far as base64

I personally didn't write a line of this module, it has been done by our team to achieve reliable attachment storage on a large project. But it still needs improvements, such as avoiding the base64 encoding/decoding which is unuseful (and maybe comes from the old client/server xmlrpc architecture before 6.0) and also avoiding to raise the whole attachment in memory. For media oriented client data with +100MB attachments, that would be the very first thing to improve.

>encoded objects in normal data table fields. This indeed has been even
>more madness :)     None [2]
>*Saludos Cordiales*   David Arnold    ​ [3]    David Arnold BA HSG
> / Analista
>315 304 13 68/ [4]
>*devCO - empresa de consultoría de sistemas (en fundación)*
> [5]
>This e-mail message may contain confidential or legally privileged
>information and is intended only for the use of the intended
>recipient(s). Any unauthorized disclosure, dissemination, distribution,
>copying or the taking of any action in reliance on the information
>herein is prohibited. E-mails are not secure and cannot be guaranteed
>to be error free as they can be intercepted, amended, or contain
>viruses. Anyone who communicates with us by e-mail is deemed to have
>accepted these risks. devCO is not responsible for errors or omissions
>in this message and denies any responsibility for any damage arising
>from the use of e-mail. Any opinion and other statement contained in
>this message and any attachment are solely those of the author and do
>not necessarily represent those of the company.
>2015-02-03 5:22 GMT-05:00 Gunnar Wagner [6] > :
>On 2/3/2015 6:12 PM, David Arnold
cite="mid:CAOLEt-F2Farq= > [7] " >type="cite"> >Cristophe has a very stringent and knowledgable >line of thought. I also see *all* counter arguments invalidated. >Unless the local NAS which isn't served over VPN, but strictly >locally (by ethernet). But if it's a local setup, why not just >have the Databse local as well and reduce entropy in the >arguments. Here have been apples mixed with banana... > >I'd >recommend Large Objects as the default. > >I though that (storing attachemnets in the db) is the default [v8] . >but following this thread so far puts me in doubts a little >can someone clarify for dummies? Also if it is not the default and >the recommendation would be to have attachments stored in the db ... >how do you achieve that > > > > > > >
cite="mid:CAOLEt-F2Farq= > [8] " >type="cite"> > >2015-02-03 3:42 GMT-05:00 Christophe >Combelles < [9] > : > >Le 3 février 2015 04:08:15 EET, Phui Hock < [10] >> a écrit : >>I regret the best theorical solution is rarely the default in Odoo. >>Practicality >>is always dependent on the context and you should rather let people >>choose what >>is practical for them. Hopefully it's easy to migrate data from the >>default >>storage to large objects. >>> >>> LOBs: >>> >>> - transcationals >>« transactional » means much more than a single word >>LOBs also : >>- are easier to setup >>- are more reliable >>- are replicable >>- are consistent (and not "eventually" consistent...) >>- can benefit from postgresql tools regarding backups, replication, >>etc. >>> >>> but >>> >>> - bloat the replication stream >>You need to transfer the file anyway. At least, there is only one >>stream to >>manage. No need to set up an rsync or anything else. Depending on the >>type of clients, saving binaries to database may be not acceptable at >>all. I have a client who is a media house and 90% of their files are >>several hundreds MB in size. One of their primary requirements is to >>have these files backup and accessible on their NAS-RAID. If the files >>are saved in the database, >How do you keep a total sync between what is in the NAS and the >ir.attachment metadata in the database? > >>it would incur unnecessary database overhead >>every time the files need to be accessed or manipulated. >With the default 7.0 storage maybe, with the blob there are no more >overhead than just reading a file from disk. > >>For instance, we needed to generate previews and thumbnails for each >of >>the known types. We also need to add watermark on-the-fly. These type >>of operations are easier and faster when files are saved on local >>filesystem. >It seems easier in the first place (but not faster). It's easier to >modify a file in the filesystem with a filesystem tool indeed, but it's >harder to do it while keeping real sync with what is in the database >anyway. And eventually I would bet it's easier in the long run to rely >on the database security and transactionality. >It all depend on the importance you give to your data. > >>Having these binaries saved on local filesystem also benefit from >>quicker streaming when served via web server such as nginx. >This is totally wrong, you can stream large blobs directly from the >database do the browser, even PHP offers such features. And regarding >small files they are supposed to be correctly cached anyway so they end >up being never served by the filesystem nor the database. >>> - esay to browse by a human >>I'm not sure humans are good at browsing such SHA1 trees : >>│ ├── 57 >>│ │ └── 57d4a8c7b652d6928d791c458cb04a246e3fb1ba >>│ ├── dc >>│ │ └── dcf00aacce882bbfd117c0277e514f829b4c5bf0 >>│ └── ef >>│ └── ef2c882a36dbe90fc1e7e28d816ad1ac1464cfbb >>└── v8 >>├── 35 >>│ └── 358d08bafbecf102728c217bcc3eb0 a24647443c >>├── 36 >>While I agree these are not human readable, I don't think it is hard >>too to derive another filestore that overrides SHA1 filename with >>actual file name. >I believe something similar was done in old 6.1 or 7.0 modules that >were abandonned due to their poor quality. Repoducing that with the >filestore would make people believe they can use and modify it as real >files which is wrong unfortunately, unless you maintain the sha1 >filename and directory manually... > >>_______________________________________________ >>Mailing-List: [11] >>Post to: mailto: [12] >>Unsubscribe: [13] > > >_______________________________________________ >Mailing-List: [14] >Post to: mailto: [15] >Unsubscribe: [16] > > > > > > >_______________________________________________ >Mailing-List: [17] >Post to: [18] >Unsubscribe: [19] > > > >-- > > > >Gunnar >Wagner | Iris Germanica Ltd. | JinQian Gong Lu 385, 8-201 >| FengXian >Qu, 201404 Shanghai | P.R. CHINA >+86 >159 0094 1702 | +49 (0)176 7808 9090 [20] | skype: >professorgunrad | >None [21] > > >_______________________________________________ >Mailing-List: [22] >Post to: mailto: [23] >Unsubscribe: [24] > > >_______________________________________________ >Mailing-List: >Post to: >Unsubscribe: > > > >[1] >[2] >[3] >[4] >[5] >[6] >[7] >[8] >[9] >[10] >[11] >[12] >[13] >[14] >[15] >[16] >[17] >[18] >[19] >[20] tel:%2B49 %280%29176 7808 9090 >[21] >[22] >[23] >[24]