[owncloud-devel] OwnCloud on WD Proposal
jospoortvliet at gmail.com
Sun Dec 20 19:51:56 GMT 2015
On Sunday 20 December 2015 20:16:28 Stefan Bruens wrote:
> On Sunday 20 December 2015 17:49:37 Jos Poortvliet wrote:
> > You shouldn't run ownCloud on a SD card anyhow - these cards are heavily
> > optimized for sequential read and write performance, sporting block sizes
> > in the megabytes. Storing a database on that with its random write
> > patterns reads to massive write amplification: for every tiny write the
> > SD card writes a full block of several MB's, burning through the lifetime
> > write capacity of the card in months, weeks or even days if you use it
> > heavily.
> Jos, can you please stop these unwarranted claims.
> While it is true that a small write may lead to a much larger write, this is
> in general not the case. NAND flashes today have pages of 2kByte to 8kByte,
> thats the smallest amount you can write. Everytime you "overwrite" a
> logical block of data, it is just marked as stale data.
> The larger erase blocks only comes into play on garbage collection. Garbage
> collection happens if
> - there are no erase blocks with unused (neither used nor stale) pages, or
> - occasionaly, during idle periods
> Garbage collection happens more frequently if the media is filled up.
> Although SD cards *are* actually much slower on random access than on
> sequential access, these are not worse than HDDs. HDDs have access times of
> ~10ms, so you have ~100 Ops/s. SD write access is about the same, random
> read access is about one order of magnitude faster.
> The RPi is also limited by its small memory. As soon as you have to move
> dirty memory pages to storage or drop data from caches, the USB disk is a
> For the matter of write amplification, you should differentiate between
> write amplification caused by:
> (1) the database
> (2) the file system
> (3) the flash translation layer (i.e. inside the storage media)
> In fact (1) can alone lead to a write amplification by a factor of two.
> SQLite in its default journal mode writes everything to the journal *and*
> to the database file. SQLite since 3.7.0 knows a Write-Ahead-Log (WAL) mode
> where changes from the journal are only written back to the database during
> checkpointing, coalescing many small writes into larger ones. Other
> databases use similar schemes to SQLites WAL.
> (2) may be problematic some filesystems not used properly. For e.g. BTRFS
> you should make sure COW is disabled for database files.
> (3) may be an issue if you have old, bad, cheap SD cards. You can not rule
> this out in general, but if the card has a somewhat decent random write
> performance (~1MByte/s), the card needs to have a proper FTL.
> IMHO is is a very good idea to put both the OS and the database on SD card.
> Even a small (8GByte) SD will be largely unused, which leaves plenty of
> space for flash wear leveling end ensures seldom run of the garbage
The SD card which comes with the prototype is 4GB and I've found plenty of
reports on the internet about breaking those in 1-3 months of usage with a Pi.
I myself have lost a USB stick of 8GB in 2 days (admittedly of continuous data
writing, I did use it for performance testing).
We could go for larger SD cards like 8 or 16 GB to give more room for wear
leveling and we can use F2FS as filesystem perhaps. With good use of caching
and no logging and read-only root filesystem, perhaps it is quite safe. I'd
still want to have a backup of the database to the hard drive once a day or
something in place, though... And an automated way to restore the operating
system and DB to a new SD card if things break, then.
I still don't feel comfortable with it, but I'm no expert either. How sure are
you that it's safe? Sure enough to commit your private data to it?
> Kind regards,
Everything I do and say is based on my view of the world today. I am not
responsible for changes in the world, nor my view on it. Everything I say is
meant in a positive and friendly way, unless explicitly stated otherwise.
find me on blog.jospoortvliet.com
-------------- next part --------------
A non-text attachment was scrubbed...
Size: 181 bytes
Desc: This is a digitally signed message part.
More information about the Devel