Community mailing list archives

community@mail.odoo.com

Re: About Attachment File Location (again)

by
gunnar wagner
- 02/03/2015 22:21:28
@ Christophe

>> https://anybox.fr/blog/postgresql-large-object-storage-for-odoo

you mention a pgdump with --schema=public enables you to make dumps that are small. I get the logic, that you do a dump without the attachments and that the dump is small due to that. What I don't get is what such a dump is worth?
I assume there must be a value to it, otherwise you wouldn't recommend it. I guess restoring from such a 'mall' dump would read the missing attachments from a previous full dump?
Do you make 'full' dumps of your db's also frequently but just less frequent (taking the risk that you loose some attachments in case of disaster) then 'small' dumps?

would be great whether you can elaborate your procedure regarding dumps and restoring those a little.





On 2/4/2015 8:26 AM, Christophe Combelles wrote:
<blockquote cite="mid:3BB62E66-96BD-47CB-AEA2-F1CAC4BB6247@anybox.fr" type="cite">
Le 4 février 2015 01:15:18 EET, Antony Lesuisse <al@openerp.com> a écrit :
>To BLOB or Not To BLOB, or To LOB or Not To LOB...
>We decided to make fs attachment the default in v8. I think it's a
>sensible
>but I understand that there are pro and con and this choice is not
>universal...
>That's the reason we made it very easy to switch to blob or to

Mmh actually you're speaking of "bytea" not blob.
(Lob and blob was the same in my mind: large objects.)


>implement any
>other storage strategy as a module.
>It easy to switch to blob storing by setting ir.config.parameters
>'ir_attachment.location' to 'db'.
>Implementing LOB storage in a module base_lob should also be very easy
>- add a data_oid column to store the LOB id.
>- override the _data_get _data_set method of ir.attachment


yep that's what is already done in our module except it's called attachment_large_object instead of base_lob  https://anybox.fr/blog/postgresql-large-object-storage-for-odoo

but still there are huge improvements remaining to be done in the base/ir_attachment (memory consumption and base64)



>
>On 02/03/2015 11:18 PM, David Arnold wrote:
>> ​It seems we still need to deepen our common understanding about
>Large Object
>> Facilities​, it's benefits and it's pitfalls and how they differ from
>> traditional db filstorage. The discussion is somewhat apple and
>bananas, so
>> I'd like to ask Christophe to comprehensifly clarify and give
>structure.
>>
>> For example on the link referenced by Nhomar is actually an argument
>in favor
>> of SQL FILSTREAM facility, which seems at first sight to me the
>counterpart to
>> POSTGRESQL LOB.
>>
>> I wished to be able to contribute edge knowledge to keep this
>discussion
>> informative and usefull...
>>
>>
>>
>> *Saludos Cordiales*
>> David Arnold
>> ​
>>
>> David Arnold BA HSG / Analista
>> 315 304 13 68/ dar@devco.co <mailto:dar@devco.co>
>>
>> *devCO - empresa de consultoría de sistemas (en fundación)*
>> http://www.devco.co
>>
>> This e-mail message may contain confidential or legally privileged
>information
>> and is intended only for the use of the intended recipient(s). Any
>> unauthorized disclosure, dissemination, distribution, copying or the
>taking of
>> any action in reliance on the information herein is prohibited.
>E-mails are
>> not secure and cannot be guaranteed to be error free as they can be
>> intercepted, amended, or contain viruses. Anyone who communicates
>with us by
>> e-mail is deemed to have accepted these risks. devCO is not
>responsible for
>> errors or omissions in this message and denies any responsibility for
>any
>> damage arising from the use of e-mail. Any opinion and other
>statement
>> contained in this message and any attachment are solely those of the
>author
>> and do not necessarily represent those of the company.
>>
>>
>> 2015-02-03 7:47 GMT-05:00 Christophe Combelles <ccomb@anybox.fr
>> <mailto:ccomb@anybox.fr>>:
>>
>>   Le 3 février 2015 12:32:57 EET, David Arnold a écrit :
>>   >I though that (storing attachemnets in the db) is the default [v8]
>.
>>   >but following this thread so far puts me in doubts a little
>>   >can someone clarify for dummies? Also if it is not the default and
>the
>>   >recommendation would be to have attachments stored in the db ...
>how do
>>   >you achieve that
>>   >http://www.postgresql.org/docs/9.4/static/largeobjects.html [1]
>>   >The keyword in the introduction is "special", arge objects are a
>>   >special facility of Postgres just made for this use case. So far
>this
>>   >facility never has been touched by odoo. (apart from Christophes
>>   >implementation - kudos!) Stored in databse meant so far as base64
>>
>>   I personally didn't write a line of this module, it has been done
>by our team to achieve reliable attachment storage on a large project.
>But it still needs improvements, such as avoiding the base64
>encoding/decoding which is unuseful (and maybe comes from the old
>client/server xmlrpc architecture before 6.0) and also avoiding to
>raise the whole attachment in memory. For media oriented client data
>with +100MB attachments, that would be the very first thing to improve.
>>
>>
>>
>>   >encoded objects in normal data table fields. This indeed has been
>even
>>   >more madness :)   None [2]
>>   >*Saludos Cordiales*  David Arnold  ​ [3]  David Arnold BA HSG
>>   > / Analista
>>   >315 304 13 68/dar@devco.co <mailto:dar@devco.co> [4]
>>   >*devCO - empresa de consultoría de sistemas (en fundación)*
>>   >http://www.devco.co [5]
>>   >This e-mail message may contain confidential or legally privileged
>>   >information and is intended only for the use of the intended
>>   >recipient(s). Any unauthorized disclosure, dissemination,
>distribution,
>>   >copying or the taking of any action in reliance on the information
>>   >herein is prohibited. E-mails are not secure and cannot be
>guaranteed
>>   >to be error free as they can be intercepted, amended, or contain
>>   >viruses. Anyone who communicates with us by e-mail is deemed to
>have
>>   >accepted these risks. devCO is not responsible for errors or
>omissions
>>   >in this message and denies any responsibility for any damage
>arising
>>   >from the use of e-mail. Any opinion and other statement contained
>in
>>   >this message and any attachment are solely those of the author and
>do
>>   >not necessarily represent those of the company.
>>   >
>>   >2015-02-03 5:22 GMT-05:00 Gunnar
>Wagnergunnar.wagner@irisgermanica.com
><mailto:gunnar.wagner@irisgermanica.com> [6] > :
>>   >
>>   >On 2/3/2015 6:12 PM, David Arnold
>>   >wrote:
>>   >
>>   >
>>
>>     cite="mid:CAOLEt-F2Farq=
>>     >9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com
>>     <mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com> [7]
>"
>>     >type="cite"> >Cristophe has a very stringent and knowledgable
>>line
>>     of thought. I also see *all* counter arguments invalidated.
>>Unless
>>     the local NAS which isn't served over VPN, but strictly >locally
>(by
>>     ethernet). But if it's a local setup, why not just >have the
>Databse
>>     local as well and reduce entropy in the >arguments. Here have
>been
>>     apples mixed with banana... > >I'd >recommend Large Objects as
>the
>>     default. > >I though that (storing attachemnets in the db) is the
>>     default [v8] . >but following this thread so far puts me in
>doubts a
>>     little >can someone clarify for dummies? Also if it is not the
>default
>>     and >the recommendation would be to have attachments stored in
>the db
>>     ... >how do you achieve that > > > > > > >
>>
>>       cite="mid:CAOLEt-F2Farq=
>>       >9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com
>>       <mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com>
>[8]
>>       " >type="cite"> > >2015-02-03 3:42 GMT-05:00 Christophe
>>Combelles
>>       : > >Le 3 février
>>       2015 04:08:15 EET, Phui Hock < phuihock@codekaki.com
>>       <mailto:phuihock@codekaki.com> [10] >> a écrit : >>I regret the
>>       best theorical solution is rarely the default in Odoo.
>>       >>Practicality >>is always dependent on the context and you
>>       should rather let people >>choose what >>is practical for them.
>>       Hopefully it's easy to migrate data from the >>default
>>>storage
>>       to large objects. >>> >>> LOBs: >>> >>> - transcationals >>«
>>       transactional » means much more than a single word >>LOBs also
>:
>>       >>- are easier to setup >>- are more reliable >>- are
>replicable
>>       >>- are consistent (and not "eventually" consistent...) >>- can
>>       benefit from postgresql tools regarding backups, replication,
>>       >>etc. >>> >>> but >>> >>> - bloat the replication stream >>You
>>       need to transfer the file anyway. At least, there is only one
>>       >>stream to >>manage. No need to set up an rsync or anything
>>       else. Depending on the >>type of clients, saving binaries to
>>       database may be not acceptable at >>all. I have a client who is
>a
>>       media house and 90% of their files are >>several hundreds MB in
>>       size. One of their primary requirements is to >>have these
>files
>>       backup and accessible on their NAS-RAID. If the files >>are
>saved
>>       in the database, >How do you keep a total sync between what is
>in
>>       the NAS and the >ir.attachment metadata in the database? > >>it
>>       would incur unnecessary database overhead >>every time the
>files
>>       need to be accessed or manipulated. >With the default 7.0
>storage
>>       maybe, with the blob there are no more >overhead than just
>reading
>>       a file from disk. > >>For instance, we needed to generate
>previews
>>       and thumbnails for each >of >>the known types. We also need to
>add
>>       watermark on-the-fly. These type >>of operations are easier and
>>       faster when files are saved on local >>filesystem. >It seems
>>       easier in the first place (but not faster). It's easier to
>>modify
>>       a file in the filesystem with a filesystem tool indeed, but
>it's
>>       >harder to do it while keeping real sync with what is in the
>>       database >anyway. And eventually I would bet it's easier in the
>>       long run to rely >on the database security and
>transactionality.
>>       >It all depend on the importance you give to your data. >
>>       >>Having these binaries saved on local filesystem also benefit
>>       from >>quicker streaming when served via web server such as
>nginx.
>>       >This is totally wrong, you can stream large blobs directly
>from
>>       the >database do the browser, even PHP offers such features.
>And
>>       regarding >small files they are supposed to be correctly cached
>>       anyway so they end >up being never served by the filesystem nor
>>       the database. >>> - esay to browse by a human >>I'm not sure
>>       humans are good at browsing such SHA1 trees : >>│ ├── 57 >>│ │
>└──
>>       57d4a8c7b652d6928d791c458cb04a246e3fb1ba >>│ ├── dc >>│ │ └──
>>       dcf00aacce882bbfd117c0277e514f829b4c5bf0 >>│ └── ef >>│ └──
>>       ef2c882a36dbe90fc1e7e28d816ad1ac1464cfbb >>└── v8 >>├── 35 >>│
>└──
>>       358d08bafbecf102728c217bcc3eb0 a24647443c >>├── 36 >>While I
>agree
>>       these are not human readable, I don't think it is hard >>too to
>>       derive another filestore that overrides SHA1 filename with
>>       >>actual file name. >I believe something similar was done in
>old
>>       6.1 or 7.0 modules that >were abandonned due to their poor
>>       quality. Repoducing that with the >filestore would make people
>>       believe they can use and modify it as real >files which is
>wrong
>>       unfortunately, unless you maintain the sha1 >filename and
>>       directory manually... >
>>       >>_______________________________________________
>>>Mailing-List:
>>       https://www.odoo.com/groups/community-59 [11] >>Post to:
>mailto:
>>       community@mail.odoo.com <mailto:community@mail.odoo.com> [12]
>>       >>Unsubscribe: https://www.odoo.com/groups?unsubscribe [13] > >
>>       >_______________________________________________ >Mailing-List:
>>       https://www.odoo.com/groups/community-59 [14] >Post to: mailto:
>>       community@mail.odoo.com <mailto:community@mail.odoo.com> [15]
>>       >Unsubscribe: https://www.odoo.com/groups?unsubscribe [16] > >
>>
>>       > > > >_______________________________________________
>>       >Mailing-List: https://www.odoo.com/groups/community-59 [17]
>>       >Post to: mailto:community@mail.odoo.com
>>       <mailto:community@mail.odoo.com> [18] >Unsubscribe:
>>       https://www.odoo.com/groups?unsubscribe [19] > > > >-- > > >
>>       >Gunnar >Wagner | Iris Germanica Ltd. | JinQian Gong Lu 385,
>>       8-201 >| FengXian >Qu, 201404 Shanghai | P.R. CHINA >+86 >159
>0094
>>       1702 | +49 (0)176 7808 9090 [20] | skype: >professorgunrad |
>>None
>>       [21] www.fashionsupermarket.net
>>       > >
>>       >_______________________________________________ >Mailing-List:
>>       https://www.odoo.com/groups/community-59 [22] >Post to: mailto:
>>       community@mail.odoo.com <mailto:community@mail.odoo.com> [23]
>>       >Unsubscribe: https://www.odoo.com/groups?unsubscribe [24] > >
>>       >_______________________________________________ >Mailing-List:
>>       https://www.odoo.com/groups/community-59>Post to:
>>       mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>>       >Unsubscribe: https://www.odoo.com/groups?unsubscribe > > >
>>[1]
>>       http://www.postgresql.org/docs/9.4/static/largeobjects.html
>>[2]
>>       http://www.elaleman.co/ >[3] http://www.elaleman.co/ >[4]
>>       mailto:dar@devco.co <mailto:dar@devco.co> >[5]
>http://www.devco.co
>>       >[6] mailto:gunnar.wagner@irisgermanica.com
>>       <mailto:gunnar.wagner@irisgermanica.com> >[7]
>>       mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com
>>       <mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com>
>>[8]
>>       mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com
>>       <mailto:9d90nnzMUjvLBUfPv76_HO3NQDNhURJ-SEOGg@mail.gmail.com>
>>[9]
>>       mailto:ccomb@anybox.fr <mailto:ccomb@anybox.fr> >[10]
>>       mailto:phuihock@codekaki.com <mailto:phuihock@codekaki.com>
>>[11]
>>       https://www.odoo.com/groups/community-59 >[12]
>>       mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>>       >[13] https://www.odoo.com/groups?unsubscribe >[14]
>>       https://www.odoo.com/groups/community-59 >[15]
>>       mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>>       >[16] https://www.odoo.com/groups?unsubscribe >[17]
>>       https://www.odoo.com/groups/community-59 >[18]
>>       mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>>       >[19] https://www.odoo.com/groups?unsubscribe >[20] tel:%2B49
>>       %280%29176 7808 9090 >[21] http://www.fashionsupermarket.net
>>[22]
>>       https://www.odoo.com/groups/community-59 >[23]
>>       mailto:community@mail.odoo.com <mailto:community@mail.odoo.com>
>>       >[24] https://www.odoo.com/groups?unsubscribe
>>
>>   _______________________________________________
>>   Mailing-List: https://www.odoo.com/groups/community-59
>>   Post to: mailto:community@mail.odoo.com
><mailto:community@mail.odoo.com>
>>   Unsubscribe: https://www.odoo.com/groups?unsubscribe
>>
>>
>> _______________________________________________
>> Mailing-List: https://www.odoo.com/groups/community-59
>> Post to: mailto:community@mail.odoo.com
>> Unsubscribe: https://www.odoo.com/groups?unsubscribe
>>
>_______________________________________________
>Mailing-List: https://www.odoo.com/groups/community-59
>Post to: mailto:community@mail.odoo.com
>Unsubscribe: https://www.odoo.com/groups?unsubscribe

-- 
Envoyé de mon téléphone Android avec K-9 Mail. Excusez la brièveté.

_______________________________________________
Mailing-List: https://www.odoo.com/groups/community-59
Post to: mailto:community@mail.odoo.com
Unsubscribe: https://www.odoo.com/groups?unsubscribe


--

Gunnar Wagner | Iris Germanica Ltd. | JinQian Gong Lu 385, 8-201 | FengXian Qu, 201404 Shanghai | P.R. CHINA
+86 159 0094 1702 | +49 (0)176 7808 9090 | skype: professorgunrad | www.fashionsupermarket.net