Community mailing list archives

community@mail.odoo.com

Re: About Attachment File Location (again)

by
Anybox, Christophe Combelles
- 01/31/2015 15:04:07
Le 31/01/2015 14:37, Antony Lesuisse a écrit :
>   From a theorical point of view i would agree. But not from a pratical point
> of view.

I regret the best theorical solution is rarely the default in Odoo. Practicality 
is always dependent on the context and you should rather let people choose what 
is practical for them.  Hopefully it's easy to migrate data from the default 
storage to large objects.

>
> LOBs:
>
> - transcationals

« transactional » means much more than a single word

LOBs also :

- are easier to setup
- are more reliable
- are replicable
- are consistent  (and not "eventually" consistent...)
- can benefit from postgresql tools regarding backups, replication, etc.

>
> but
>
> - bloat the replication stream

You need to transfer the file anyway. At least, there is only one stream to 
manage. No need to set up an rsync or anything else.

> - bloat the sql dumps

You can choose to have a partial dump without attachments with : pg_dump 
--schema=public
I find it cleaner and easier to understand than the odoo-specific zip file.


> - break the unix file descriptor api open(2) read(2) lseek(2) you have to wrap
> db (pyscopg makes that easy under python with lobject)

Indeed, LOBs offer some facilities to seek, truncate, etc.  Nothing is broken, 
that's just an API.

> - impossible to manipulate with unix tools
>
> filestore as sha1 (git-like):
>
> - non transactionnal
>
> but
>
> - quickly rsyncable (file are never touched except for unlink)
> - space efficient rsnapshots

these points seem related to the practical details of the odoo saas infrastructure


> - esay to browse by a human

I'm not sure humans are good at browsing such SHA1 trees :

│   ├── 57
│   │   └── 57d4a8c7b652d6928d791c458cb04a246e3fb1ba
│   ├── dc
│   │   └── dcf00aacce882bbfd117c0277e514f829b4c5bf0
│   └── ef
│       └── ef2c882a36dbe90fc1e7e28d816ad1ac1464cfbb
└── v8
     ├── 35
     │   └── 358d08bafbecf102728c217bcc3eb0a24647443c
     ├── 36


> - easy to manipulate with unix tools for example you can easily do a:
>
> find filestore/db -type f -exec file -b '{}' ';' | sort | uniq -c | sort -n

You could achieve the same result with an SQL request on proper file metadata.  
It would be several orders of magnitude faster...
For typical files stored in Odoo, I mean PDF files, images, or attachments of 
several MB, there is no point storing directly in the filesystem.

That may be only relevant if Odoo wanted to turn into a huge video filestore 
with files larger than 4 TB, and you wanted the files to be directly read by 
some encoding tools (such as ffmpeg). I think we're not there yet.  And I 
suspect it wouldn't be a big deal to offer file-like access to large objects for 
these tools by mapping.


>
> At the moment odoo attachments are limited by the worker memory so the main
> raison behind that choice was rsyncablility/rsnapshotability and avoid
> bloating backups dumps.

That just confirms the wrong decision of the filestore:

- For the memory problem Odoo should never raise the whole file into memory but 
directly stream to and from the storage.
- For the dump you can still use the --schema=public, that's so easy
- For rsyncability, don't turn a consequence into an argument : rsync is not 
needed with Large Objects


>
> On 01/31/2015 12:13 PM, Christophe Combelles wrote:
> >
> > Le 31 janvier 2015 05:47:44 EET, Togar Hutabarat <to.garnet@gmail.com> a écrit :
> >>Hello Community!
> >>Maybe many question similar with my question has been asked by a many
> >>people in forums or Stackoverflow. But please allow me to ask again in
> >>this community forum. My question: is there any possibility for Odoo to
> >>save attachment directly into a file system directory (instead of
> >>database)?
> >>Another doubt about this: I know that we can configure FTP for Odoo
> >>(module document_ftp), so we can browse/copy/open attachment by using
> >>file explorer (Linux/Windows/iOS) or using any FTP client. Is that it
> >>really store the file into FTP server? Or is Odoo just use FTP as a
> >>emulation interface to access attachments? What we want is to make sure
> >>Odoo not store attachment data into database that may increase database
> >>size when we upload big files.
> >
> >
> >   Storing big files in the database really is not a problem, if you use the right storage with a way to dump without attachments if needed :
> > http://anybox.fr/blog/postgresql-large-object-storage-for-odoo
> > Not only it is no problem, but most of the time it's a far better solution to keep your data in a safe place.
> >
> > regards,
> > Christophe
> >
> >
> >>Really appreciate any answer.. Thanks,   *
> >>Togar Parnaehan Hutabarat*  *Databit - Odoo Partner Indonesia*
> >>Email    :  togarhutabarat@databit.co.id [1]  Mobile   :
> >>+62821-3079-8827  YM       :  sherlocx@ymail.com [2]   Skype
> >> : togar.hutabarat
> >>Twitter  : @togarhutabarat
> >>
> >>_______________________________________________
> >>Mailing-List: https://www.odoo.com/groups/community-59
> >>Post to: mailto:community@mail.odoo.com
> >>Unsubscribe: https://www.odoo.com/groups?unsubscribe
> >>
> >>
> >>
> >>[1] mailto:togarhutabarat@databit.co.id
> >>[2] mailto:sherlocx@ymail.com
> >
> > _______________________________________________
> > Mailing-List: https://www.odoo.com/groups/community-59
> > Post to: mailto:community@mail.odoo.com
> > Unsubscribe: https://www.odoo.com/groups?unsubscribe
> >
>
> _______________________________________________
> Mailing-List: https://www.odoo.com/groups/community-59
> Post to: mailto:community@mail.odoo.com
> Unsubscribe: https://www.odoo.com/groups?unsubscribe
>