Community mailing list archives

community@mail.odoo.com

Re: About Attachment File Location (again)

by
Antony Lesuisse (al)
- 02/03/2015 18:02:08
To BLOB or Not To BLOB, or To LOB or Not To LOB...

We decided to make fs attachment the default in v8. I think it's a sensible 
but I understand that there are pro and con and this choice is not universal...

That's the reason we made it very easy to switch to blob or to implement any 
other storage strategy as a module.

It easy to switch to blob storing by setting ir.config.parameters 
'ir_attachment.location' to 'db'.

Implementing LOB storage in a module base_lob should also be very easy

- add a data_oid column to store the LOB id.
- override the _data_get _data_set method of ir.attachment


On 02/03/2015 11:18 PM, David Arnold wrote:
> ​It seems we still need to deepen our common understanding about Large Object
> Facilities​, it's benefits and it's pitfalls and how they differ from
> traditional db filstorage. The discussion is somewhat apple and bananas, so
> I'd like to ask Christophe to comprehensifly clarify and give structure.
>
> For example on the link referenced by Nhomar is actually an argument in favor
> of SQL FILSTREAM facility, which seems at first sight to me the counterpart to
> POSTGRESQL LOB.
>
> I wished to be able to contribute edge knowledge to keep this discussion
> informative and usefull...
>
> 
>
> *Saludos Cordiales*
> David Arnold
> ​ 
>
> David Arnold BA HSG / Analista
> 315 304 13 68/ dar@devco.co <mailto:dar@devco.co>
>
> *devCO - empresa de consultoría de sistemas (en fundación)*
> http://www.devco.co
>
> This e-mail message may contain confidential or legally privileged information
> and is intended only for the use of the intended recipient(s). Any
> unauthorized disclosure, dissemination, distribution, copying or the taking of
> any action in reliance on the information herein is prohibited. E-mails are
> not secure and cannot be guaranteed to be error free as they can be
> intercepted, amended, or contain viruses. Anyone who communicates with us by
> e-mail is deemed to have accepted these risks. devCO is not responsible for
> errors or omissions in this message and denies any responsibility for any
> damage arising from the use of e-mail. Any opinion and other statement
> contained in this message and any attachment are solely those of the author
> and do not necessarily represent those of the company.
>
>
> 2015-02-03 7:47 GMT-05:00 Christophe Combelles <ccomb@anybox.fr
> <mailto:ccomb@anybox.fr>>:
>
>     Le 3 février 2015 12:32:57 EET, David Arnold  a écrit :
>     >I though that (storing attachemnets in the db) is the default [v8] .
>     >but following this thread so far puts me in doubts a little
>     >can someone clarify for dummies? Also if it is not the default and the
>     >recommendation would be to have attachments stored in the db ... how do
>     >you achieve that
>     >http://www.postgresql.org/docs/9.4/static/largeobjects.html  [1]
>     >The keyword in the introduction is "special", arge objects are a
>     >special facility of Postgres just made for this use case. So far this
>     >facility never has been touched by odoo. (apart from Christophes
>     >implementation - kudos!) Stored in databse meant so far as base64
>
>     I personally didn't write a line of this module, it has been done by our team to achieve reliable attachment storage on a large project. But it still needs improvements, such as avoiding the base64 encoding/decoding which is unuseful (and maybe comes from the old client/server xmlrpc architecture before 6.0) and also avoiding to raise the whole attachment in memory. For media oriented client data with +100MB attachments, that would be the very first thing to improve.
>
>
>
>     >encoded objects in normal data table fields. This indeed has been even
>     >more madness :)     None [2]
>     >*Saludos Cordiales*   David Arnold    ​ [3]    David Arnold BA HSG
>     > / Analista
>     >315 304 13 68/dar@devco.co  <mailto:dar@devco.co>  [4]
>     >*devCO - empresa de consultoría de sistemas (en fundación)*
>     >http://www.devco.co  [5]
>     >This e-mail message may contain confidential or legally privileged
>     >information and is intended only for the use of the intended
>     >recipient(s). Any unauthorized disclosure, dissemination, distribution,
>     >copying or the taking of any action in reliance on the information
>     >herein is prohibited. E-mails are not secure and cannot be guaranteed
>     >to be error free as they can be intercepted, amended, or contain
>     >viruses. Anyone who communicates with us by e-mail is deemed to have
>     >accepted these risks. devCO is not responsible for errors or omissions
>     >in this message and denies any responsibility for any damage arising
>     >from the use of e-mail. Any opinion and other statement contained in
>     >this message and any attachment are solely those of the author and do
>     >not necessarily represent those of the company.
>     >
>     >2015-02-03 5:22 GMT-05:00 Gunnar Wagnergunnar.wagner@irisgermanica.com  <mailto:gunnar.wagner@irisgermanica.com>  [6] > :
>     >
>     >On 2/3/2015 6:12 PM, David Arnold
>     >wrote:
>     >
>     >
>
>         cite="mid:CAOLEt-F2Farq=
>          >9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com
>         <mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com> [7] "
>          >type="cite"> >Cristophe has a very stringent and knowledgable >line
>         of thought. I also see *all* counter arguments invalidated. >Unless
>         the local NAS which isn't served over VPN, but strictly >locally (by
>         ethernet). But if it's a local setup, why not just >have the Databse
>         local as well and reduce entropy in the >arguments. Here have been
>         apples mixed with banana... > >I'd >recommend Large Objects as the
>         default. > >I though that (storing attachemnets in the db) is the
>         default [v8] . >but following this thread so far puts me in doubts a
>         little >can someone clarify for dummies? Also if it is not the default
>         and >the recommendation would be to have attachments stored in the db
>         ... >how do you achieve that > > > > > > >
>
>             cite="mid:CAOLEt-F2Farq=
>              >9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com
>             <mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com> [8]
>             " >type="cite"> > >2015-02-03 3:42 GMT-05:00 Christophe >Combelles
>              : > >Le 3 février
>             2015 04:08:15 EET, Phui Hock < phuihock@codekaki.com
>             <mailto:phuihock@codekaki.com> [10] >> a écrit : >>I regret the
>             best theorical solution is rarely the default in Odoo.
>              >>Practicality >>is always dependent on the context and you
>             should rather let people >>choose what >>is practical for them.
>             Hopefully it's easy to migrate data from the >>default >>storage
>             to large objects. >>> >>> LOBs: >>> >>> - transcationals >>«
>             transactional » means much more than a single word >>LOBs also :
>              >>- are easier to setup >>- are more reliable >>- are replicable
>              >>- are consistent (and not "eventually" consistent...) >>- can
>             benefit from postgresql tools regarding backups, replication,
>              >>etc. >>> >>> but >>> >>> - bloat the replication stream >>You
>             need to transfer the file anyway. At least, there is only one
>              >>stream to >>manage. No need to set up an rsync or anything
>             else. Depending on the >>type of clients, saving binaries to
>             database may be not acceptable at >>all. I have a client who is a
>             media house and 90% of their files are >>several hundreds MB in
>             size. One of their primary requirements is to >>have these files
>             backup and accessible on their NAS-RAID. If the files >>are saved
>             in the database, >How do you keep a total sync between what is in
>             the NAS and the >ir.attachment metadata in the database? > >>it
>             would incur unnecessary database overhead >>every time the files
>             need to be accessed or manipulated. >With the default 7.0 storage
>             maybe, with the blob there are no more >overhead than just reading
>             a file from disk. > >>For instance, we needed to generate previews
>             and thumbnails for each >of >>the known types. We also need to add
>             watermark on-the-fly. These type >>of operations are easier and
>             faster when files are saved on local >>filesystem. >It seems
>             easier in the first place (but not faster). It's easier to >modify
>             a file in the filesystem with a filesystem tool indeed, but it's
>              >harder to do it while keeping real sync with what is in the
>             database >anyway. And eventually I would bet it's easier in the
>             long run to rely >on the database security and transactionality.
>              >It all depend on the importance you give to your data. >
>              >>Having these binaries saved on local filesystem also benefit
>             from >>quicker streaming when served via web server such as nginx.
>              >This is totally wrong, you can stream large blobs directly from
>             the >database do the browser, even PHP offers such features. And
>             regarding >small files they are supposed to be correctly cached
>             anyway so they end >up being never served by the filesystem nor
>             the database. >>> - esay to browse by a human >>I'm not sure
>             humans are good at browsing such SHA1 trees : >>│ ├── 57 >>│ │ └──
>             57d4a8c7b652d6928d791c458cb04a246e3fb1ba >>│ ├── dc >>│ │ └──
>             dcf00aacce882bbfd117c0277e514f829b4c5bf0 >>│ └── ef >>│ └──
>             ef2c882a36dbe90fc1e7e28d816ad1ac1464cfbb >>└── v8 >>├── 35 >>│ └──
>             358d08bafbecf102728c217bcc3eb0 a24647443c >>├── 36 >>While I agree
>             these are not human readable, I don't think it is hard >>too to
>             derive another filestore that overrides SHA1 filename with
>              >>actual file name. >I believe something similar was done in old
>             6.1 or 7.0 modules that >were abandonned due to their poor
>             quality. Repoducing that with the >filestore would make people
>             believe they can use and modify it as real >files which is wrong
>             unfortunately, unless you maintain the sha1 >filename and
>             directory manually... >
>              >>_______________________________________________ >>Mailing-List:
>             https://www.odoo.com/groups/community-59 [11] >>Post to: mailto:
>             community@mail.odoo.com <mailto:community@mail.odoo.com> [12]
>              >>Unsubscribe: https://www.odoo.com/groups?unsubscribe [13] > >
>              >_______________________________________________ >Mailing-List:
>             https://www.odoo.com/groups/community-59 [14] >Post to: mailto:
>             community@mail.odoo.com <mailto:community@mail.odoo.com> [15]
>              >Unsubscribe: https://www.odoo.com/groups?unsubscribe [16] > > >
>              > > > >_______________________________________________
>              >Mailing-List: https://www.odoo.com/groups/community-59 [17]
>              >Post to: mailto:community@mail.odoo.com
>             <mailto:community@mail.odoo.com> [18] >Unsubscribe:
>             https://www.odoo.com/groups?unsubscribe [19] > > > >-- > > >
>              >Gunnar >Wagner | Iris Germanica Ltd. | JinQian Gong Lu 385,
>             8-201 >| FengXian >Qu, 201404 Shanghai | P.R. CHINA >+86 >159 0094
>             1702 | +49 (0)176 7808 9090 [20] | skype: >professorgunrad | >None
>             [21] www.fashionsupermarket.net
>              > >
>              >_______________________________________________ >Mailing-List:
>             https://www.odoo.com/groups/community-59 [22] >Post to: mailto:
>             community@mail.odoo.com <mailto:community@mail.odoo.com> [23]
>              >Unsubscribe: https://www.odoo.com/groups?unsubscribe [24] > >
>              >_______________________________________________ >Mailing-List:
>             https://www.odoo.com/groups/community-59>Post to:
>             mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>              >Unsubscribe: https://www.odoo.com/groups?unsubscribe > > > >[1]
>             http://www.postgresql.org/docs/9.4/static/largeobjects.html >[2]
>             http://www.elaleman.co/ >[3] http://www.elaleman.co/ >[4]
>             mailto:dar@devco.co <mailto:dar@devco.co> >[5] http://www.devco.co
>              >[6] mailto:gunnar.wagner@irisgermanica.com
>             <mailto:gunnar.wagner@irisgermanica.com> >[7]
>             mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com
>             <mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com> >[8]
>             mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com
>             <mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com> >[9]
>             mailto:ccomb@anybox.fr <mailto:ccomb@anybox.fr> >[10]
>             mailto:phuihock@codekaki.com <mailto:phuihock@codekaki.com> >[11]
>             https://www.odoo.com/groups/community-59 >[12]
>             mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>              >[13] https://www.odoo.com/groups?unsubscribe >[14]
>             https://www.odoo.com/groups/community-59 >[15]
>             mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>              >[16] https://www.odoo.com/groups?unsubscribe >[17]
>             https://www.odoo.com/groups/community-59 >[18]
>             mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>              >[19] https://www.odoo.com/groups?unsubscribe >[20] tel:%2B49
>             %280%29176 7808 9090 >[21] http://www.fashionsupermarket.net >[22]
>             https://www.odoo.com/groups/community-59 >[23]
>             mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>              >[24] https://www.odoo.com/groups?unsubscribe
>
>     _______________________________________________
>     Mailing-List: https://www.odoo.com/groups/community-59
>     Post to: mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>     Unsubscribe: https://www.odoo.com/groups?unsubscribe
>
>
> _______________________________________________
> Mailing-List: https://www.odoo.com/groups/community-59
> Post to: mailto:community@mail.odoo.com
> Unsubscribe: https://www.odoo.com/groups?unsubscribe
>