Community mailing list archives

community@mail.odoo.com

Re: About Attachment File Location (again)

by
Anybox, Christophe Combelles
- 02/03/2015 19:24:42

Le 4 février 2015 01:15:18 EET, Antony Lesuisse <al@openerp.com> a écrit :
>To BLOB or Not To BLOB, or To LOB or Not To LOB...
>We decided to make fs attachment the default in v8. I think it's a
>sensible
>but I understand that there are pro and con and this choice is not
>universal...
>That's the reason we made it very easy to switch to blob or to

Mmh actually you're speaking of "bytea" not blob.
(Lob and blob was the same in my mind: large objects.)


>implement any
>other storage strategy as a module.
>It easy to switch to blob storing by setting ir.config.parameters
>'ir_attachment.location' to 'db'.
>Implementing LOB storage in a module base_lob should also be very easy
>- add a data_oid column to store the LOB id.
>- override the _data_get _data_set method of ir.attachment


yep that's what is already done in our module except it's called attachment_large_object instead of base_lob  https://anybox.fr/blog/postgresql-large-object-storage-for-odoo

but still there are huge improvements remaining to be done in the base/ir_attachment (memory consumption and base64)



>
>On 02/03/2015 11:18 PM, David Arnold wrote:
>> ​It seems we still need to deepen our common understanding about
>Large Object
>> Facilities​, it's benefits and it's pitfalls and how they differ from
>> traditional db filstorage. The discussion is somewhat apple and
>bananas, so
>> I'd like to ask Christophe to comprehensifly clarify and give
>structure.
>>
>> For example on the link referenced by Nhomar is actually an argument
>in favor
>> of SQL FILSTREAM facility, which seems at first sight to me the
>counterpart to
>> POSTGRESQL LOB.
>>
>> I wished to be able to contribute edge knowledge to keep this
>discussion
>> informative and usefull...
>>
>>
>>
>> *Saludos Cordiales*
>> David Arnold
>> ​
>>
>> David Arnold BA HSG / Analista
>> 315 304 13 68/ dar@devco.co <mailto:dar@devco.co>
>>
>> *devCO - empresa de consultoría de sistemas (en fundación)*
>> http://www.devco.co
>>
>> This e-mail message may contain confidential or legally privileged
>information
>> and is intended only for the use of the intended recipient(s). Any
>> unauthorized disclosure, dissemination, distribution, copying or the
>taking of
>> any action in reliance on the information herein is prohibited.
>E-mails are
>> not secure and cannot be guaranteed to be error free as they can be
>> intercepted, amended, or contain viruses. Anyone who communicates
>with us by
>> e-mail is deemed to have accepted these risks. devCO is not
>responsible for
>> errors or omissions in this message and denies any responsibility for
>any
>> damage arising from the use of e-mail. Any opinion and other
>statement
>> contained in this message and any attachment are solely those of the
>author
>> and do not necessarily represent those of the company.
>>
>>
>> 2015-02-03 7:47 GMT-05:00 Christophe Combelles <ccomb@anybox.fr
>> <mailto:ccomb@anybox.fr>>:
>>
>>   Le 3 février 2015 12:32:57 EET, David Arnold a écrit :
>>   >I though that (storing attachemnets in the db) is the default [v8]
>.
>>   >but following this thread so far puts me in doubts a little
>>   >can someone clarify for dummies? Also if it is not the default and
>the
>>   >recommendation would be to have attachments stored in the db ...
>how do
>>   >you achieve that
>>   >http://www.postgresql.org/docs/9.4/static/largeobjects.html [1]
>>   >The keyword in the introduction is "special", arge objects are a
>>   >special facility of Postgres just made for this use case. So far
>this
>>   >facility never has been touched by odoo. (apart from Christophes
>>   >implementation - kudos!) Stored in databse meant so far as base64
>>
>>   I personally didn't write a line of this module, it has been done
>by our team to achieve reliable attachment storage on a large project.
>But it still needs improvements, such as avoiding the base64
>encoding/decoding which is unuseful (and maybe comes from the old
>client/server xmlrpc architecture before 6.0) and also avoiding to
>raise the whole attachment in memory. For media oriented client data
>with +100MB attachments, that would be the very first thing to improve.
>>
>>
>>
>>   >encoded objects in normal data table fields. This indeed has been
>even
>>   >more madness :)   None [2]
>>   >*Saludos Cordiales*  David Arnold  ​ [3]  David Arnold BA HSG
>>   > / Analista
>>   >315 304 13 68/dar@devco.co <mailto:dar@devco.co> [4]
>>   >*devCO - empresa de consultoría de sistemas (en fundación)*
>>   >http://www.devco.co [5]
>>   >This e-mail message may contain confidential or legally privileged
>>   >information and is intended only for the use of the intended
>>   >recipient(s). Any unauthorized disclosure, dissemination,
>distribution,
>>   >copying or the taking of any action in reliance on the information
>>   >herein is prohibited. E-mails are not secure and cannot be
>guaranteed
>>   >to be error free as they can be intercepted, amended, or contain
>>   >viruses. Anyone who communicates with us by e-mail is deemed to
>have
>>   >accepted these risks. devCO is not responsible for errors or
>omissions
>>   >in this message and denies any responsibility for any damage
>arising
>>   >from the use of e-mail. Any opinion and other statement contained
>in
>>   >this message and any attachment are solely those of the author and
>do
>>   >not necessarily represent those of the company.
>>   >
>>   >2015-02-03 5:22 GMT-05:00 Gunnar
>Wagnergunnar.wagner@irisgermanica.com
><mailto:gunnar.wagner@irisgermanica.com> [6] > :
>>   >
>>   >On 2/3/2015 6:12 PM, David Arnold
>>   >wrote:
>>   >
>>   >
>>
>>     cite="mid:CAOLEt-F2Farq=
>>     >9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com
>>     <mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com> [7]
>"
>>     >type="cite"> >Cristophe has a very stringent and knowledgable
>>line
>>     of thought. I also see *all* counter arguments invalidated.
>>Unless
>>     the local NAS which isn't served over VPN, but strictly >locally
>(by
>>     ethernet). But if it's a local setup, why not just >have the
>Databse
>>     local as well and reduce entropy in the >arguments. Here have
>been
>>     apples mixed with banana... > >I'd >recommend Large Objects as
>the
>>     default. > >I though that (storing attachemnets in the db) is the
>>     default [v8] . >but following this thread so far puts me in
>doubts a
>>     little >can someone clarify for dummies? Also if it is not the
>default
>>     and >the recommendation would be to have attachments stored in
>the db
>>     ... >how do you achieve that > > > > > > >
>>
>>       cite="mid:CAOLEt-F2Farq=
>>       >9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com
>>       <mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com>
>[8]
>>       " >type="cite"> > >2015-02-03 3:42 GMT-05:00 Christophe
>>Combelles
>>       : > >Le 3 février
>>       2015 04:08:15 EET, Phui Hock < phuihock@codekaki.com
>>       <mailto:phuihock@codekaki.com> [10] >> a écrit : >>I regret the
>>       best theorical solution is rarely the default in Odoo.
>>       >>Practicality >>is always dependent on the context and you
>>       should rather let people >>choose what >>is practical for them.
>>       Hopefully it's easy to migrate data from the >>default
>>>storage
>>       to large objects. >>> >>> LOBs: >>> >>> - transcationals >>«
>>       transactional » means much more than a single word >>LOBs also
>:
>>       >>- are easier to setup >>- are more reliable >>- are
>replicable
>>       >>- are consistent (and not "eventually" consistent...) >>- can
>>       benefit from postgresql tools regarding backups, replication,
>>       >>etc. >>> >>> but >>> >>> - bloat the replication stream >>You
>>       need to transfer the file anyway. At least, there is only one
>>       >>stream to >>manage. No need to set up an rsync or anything
>>       else. Depending on the >>type of clients, saving binaries to
>>       database may be not acceptable at >>all. I have a client who is
>a
>>       media house and 90% of their files are >>several hundreds MB in
>>       size. One of their primary requirements is to >>have these
>files
>>       backup and accessible on their NAS-RAID. If the files >>are
>saved
>>       in the database, >How do you keep a total sync between what is
>in
>>       the NAS and the >ir.attachment metadata in the database? > >>it
>>       would incur unnecessary database overhead >>every time the
>files
>>       need to be accessed or manipulated. >With the default 7.0
>storage
>>       maybe, with the blob there are no more >overhead than just
>reading
>>       a file from disk. > >>For instance, we needed to generate
>previews
>>       and thumbnails for each >of >>the known types. We also need to
>add
>>       watermark on-the-fly. These type >>of operations are easier and
>>       faster when files are saved on local >>filesystem. >It seems
>>       easier in the first place (but not faster). It's easier to
>>modify
>>       a file in the filesystem with a filesystem tool indeed, but
>it's
>>       >harder to do it while keeping real sync with what is in the
>>       database >anyway. And eventually I would bet it's easier in the
>>       long run to rely >on the database security and
>transactionality.
>>       >It all depend on the importance you give to your data. >
>>       >>Having these binaries saved on local filesystem also benefit
>>       from >>quicker streaming when served via web server such as
>nginx.
>>       >This is totally wrong, you can stream large blobs directly
>from
>>       the >database do the browser, even PHP offers such features.
>And
>>       regarding >small files they are supposed to be correctly cached
>>       anyway so they end >up being never served by the filesystem nor
>>       the database. >>> - esay to browse by a human >>I'm not sure
>>       humans are good at browsing such SHA1 trees : >>│ ├── 57 >>│ │
>└──
>>       57d4a8c7b652d6928d791c458cb04a246e3fb1ba >>│ ├── dc >>│ │ └──
>>       dcf00aacce882bbfd117c0277e514f829b4c5bf0 >>│ └── ef >>│ └──
>>       ef2c882a36dbe90fc1e7e28d816ad1ac1464cfbb >>└── v8 >>├── 35 >>│
>└──
>>       358d08bafbecf102728c217bcc3eb0 a24647443c >>├── 36 >>While I
>agree
>>       these are not human readable, I don't think it is hard >>too to
>>       derive another filestore that overrides SHA1 filename with
>>       >>actual file name. >I believe something similar was done in
>old
>>       6.1 or 7.0 modules that >were abandonned due to their poor
>>       quality. Repoducing that with the >filestore would make people
>>       believe they can use and modify it as real >files which is
>wrong
>>       unfortunately, unless you maintain the sha1 >filename and
>>       directory manually... >
>>       >>_______________________________________________
>>>Mailing-List:
>>       https://www.odoo.com/groups/community-59 [11] >>Post to:
>mailto:
>>       community@mail.odoo.com <mailto:community@mail.odoo.com> [12]
>>       >>Unsubscribe: https://www.odoo.com/groups?unsubscribe [13] > >
>>       >_______________________________________________ >Mailing-List:
>>       https://www.odoo.com/groups/community-59 [14] >Post to: mailto:
>>       community@mail.odoo.com <mailto:community@mail.odoo.com> [15]
>>       >Unsubscribe: https://www.odoo.com/groups?unsubscribe [16] > >
>>
>>       > > > >_______________________________________________
>>       >Mailing-List: https://www.odoo.com/groups/community-59 [17]
>>       >Post to: mailto:community@mail.odoo.com
>>       <mailto:community@mail.odoo.com> [18] >Unsubscribe:
>>       https://www.odoo.com/groups?unsubscribe [19] > > > >-- > > >
>>       >Gunnar >Wagner | Iris Germanica Ltd. | JinQian Gong Lu 385,
>>       8-201 >| FengXian >Qu, 201404 Shanghai | P.R. CHINA >+86 >159
>0094
>>       1702 | +49 (0)176 7808 9090 [20] | skype: >professorgunrad |
>>None
>>       [21] www.fashionsupermarket.net
>>       > >
>>       >_______________________________________________ >Mailing-List:
>>       https://www.odoo.com/groups/community-59 [22] >Post to: mailto:
>>       community@mail.odoo.com <mailto:community@mail.odoo.com> [23]
>>       >Unsubscribe: https://www.odoo.com/groups?unsubscribe [24] > >
>>       >_______________________________________________ >Mailing-List:
>>       https://www.odoo.com/groups/community-59>Post to:
>>       mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>>       >Unsubscribe: https://www.odoo.com/groups?unsubscribe > > >
>>[1]
>>       http://www.postgresql.org/docs/9.4/static/largeobjects.html
>>[2]
>>       http://www.elaleman.co/ >[3] http://www.elaleman.co/ >[4]
>>       mailto:dar@devco.co <mailto:dar@devco.co> >[5]
>http://www.devco.co
>>       >[6] mailto:gunnar.wagner@irisgermanica.com
>>       <mailto:gunnar.wagner@irisgermanica.com> >[7]
>>       mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com
>>       <mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com>
>>[8]
>>       mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com
>>       <mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com>
>>[9]
>>       mailto:ccomb@anybox.fr <mailto:ccomb@anybox.fr> >[10]
>>       mailto:phuihock@codekaki.com <mailto:phuihock@codekaki.com>
>>[11]
>>       https://www.odoo.com/groups/community-59 >[12]
>>       mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>>       >[13] https://www.odoo.com/groups?unsubscribe >[14]
>>       https://www.odoo.com/groups/community-59 >[15]
>>       mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>>       >[16] https://www.odoo.com/groups?unsubscribe >[17]
>>       https://www.odoo.com/groups/community-59 >[18]
>>       mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>>       >[19] https://www.odoo.com/groups?unsubscribe >[20] tel:%2B49
>>       %280%29176 7808 9090 >[21] http://www.fashionsupermarket.net
>>[22]
>>       https://www.odoo.com/groups/community-59 >[23]
>>       mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>>       >[24] https://www.odoo.com/groups?unsubscribe
>>
>>   _______________________________________________
>>   Mailing-List: https://www.odoo.com/groups/community-59
>>   Post to: mailto:community@mail.odoo.com
><mailto:community@mail.odoo.com>
>>   Unsubscribe: https://www.odoo.com/groups?unsubscribe
>>
>>
>> _______________________________________________
>> Mailing-List: https://www.odoo.com/groups/community-59
>> Post to: mailto:community@mail.odoo.com
>> Unsubscribe: https://www.odoo.com/groups?unsubscribe
>>
>_______________________________________________
>Mailing-List: https://www.odoo.com/groups/community-59
>Post to: mailto:community@mail.odoo.com
>Unsubscribe: https://www.odoo.com/groups?unsubscribe

-- 
Envoyé de mon téléphone Android avec K-9 Mail. Excusez la brièveté.