[owncloud-devel] OwnCloud on WD Proposal

Jos Poortvliet jospoortvliet at gmail.com
Sun Dec 20 19:51:56 GMT 2015


On Sunday 20 December 2015 20:16:28 Stefan Bruens wrote:
> On Sunday 20 December 2015 17:49:37 Jos Poortvliet wrote:
> > You shouldn't run ownCloud on a SD card anyhow - these cards are heavily
> > optimized for sequential read and write performance, sporting block sizes
> > in the megabytes. Storing a database on that with its random write
> > patterns reads to massive write amplification: for every tiny write the
> > SD card writes a full block of several MB's, burning through the lifetime
> > write capacity of the card in months, weeks or even days if you use it
> > heavily.
> Jos, can you please stop these unwarranted claims.
> 
> While it is true that a small write may lead to a much larger write, this is
> in general not the case. NAND flashes today have pages of 2kByte to 8kByte,
> thats the smallest amount you can write. Everytime you "overwrite" a
> logical block of data, it is just marked as stale data.
> 
> The larger erase blocks only comes into play on garbage collection. Garbage
> collection happens if
> - there are no erase blocks with unused (neither used nor stale) pages, or
> - occasionaly, during idle periods
> Garbage collection happens more frequently if the media is filled up.
> 
> Although SD cards *are* actually much slower on random access than on
> sequential access, these are not worse than HDDs. HDDs have access times of
> ~10ms, so you have ~100 Ops/s. SD write access is about the same, random
> read access is about one order of magnitude faster.
> 
> The RPi is also limited by its small memory. As soon as you have to move
> dirty memory pages to storage or drop data from caches, the USB disk is a
> bottleneck.
> 
> For the matter of write amplification, you should differentiate between
> write amplification caused by:
> (1) the database
> (2) the file system
> (3) the flash translation layer (i.e. inside the storage media)
> 
> In fact (1) can alone lead to a write amplification by a factor of two.
> SQLite in its default journal mode writes everything to the journal *and*
> to the database file. SQLite since 3.7.0 knows a Write-Ahead-Log (WAL) mode
> where changes from the journal are only written back to the database during
> checkpointing, coalescing many small writes into larger ones. Other
> databases use similar schemes to SQLites WAL.
> 
> (2) may be problematic some filesystems not used properly. For e.g. BTRFS
> you should make sure COW is disabled for database files.
> 
> (3) may be an issue if you have old, bad, cheap SD cards. You can not rule
> this out in general, but if the card has a somewhat decent random write
> performance (~1MByte/s), the card needs to have a proper FTL.
> 
> IMHO is is a very good idea to put both the OS and the database on SD card.
> Even a small (8GByte) SD will be largely unused, which leaves plenty of
> space for flash wear leveling end ensures seldom run of the garbage
> collector.

The SD card which comes with the prototype is 4GB and I've found plenty of 
reports on the internet about breaking those in 1-3 months of usage with a Pi. 
I myself have lost a USB stick of 8GB in 2 days (admittedly of continuous data 
writing, I did use it for performance testing).

We could go for larger SD cards like 8 or 16 GB to give more room for wear 
leveling and we can use F2FS as filesystem perhaps. With good use of caching 
and no logging and read-only root filesystem, perhaps it is quite safe. I'd 
still want to have a backup of the database to the hard drive once a day or 
something in place, though... And an automated way to restore the operating 
system and DB to a new SD card if things break, then.

I still don't feel comfortable with it, but I'm no expert either. How sure are 
you that it's safe? Sure enough to commit your private data to it?

/J

> Kind regards,
> 
> Stefan

-- 
Disclaimer:
Everything I do and say is based on my view of the world today. I am not 
responsible for changes in the world, nor my view on it. Everything I say is 
meant in a positive and friendly way, unless explicitly stated otherwise.
find me on blog.jospoortvliet.com
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 181 bytes
Desc: This is a digitally signed message part.
URL: <http://mailman.owncloud.org/pipermail/devel/attachments/20151220/93cb335c/attachment.sig>


More information about the Devel mailing list