[owncloud-devel] Experience with WD Chalenge like setup

Michal Hrusecky michal at hrusecky.net
Thu Dec 17 22:23:43 GMT 2015

Jos Poortvliet - 22:36 17.12.15 wrote:
> On Thursday 17 December 2015 22:07:34 Michal Hrusecky wrote:
> > Hi,
> > 
> > while discussion is going on, I have a few questions regarding everyones
> > experience with running similar setup to what is proposed in WD
> > Challenge. I have at home BananaPi with external USB harddrive running
> > ownCloud now and while installing it I did few decisions that might be
> > relevant to the challenge as well, so I was wondering whether you have
> > similar experience/setup or something else (will help us shape the image
> > and improve our home setups).
> > 
> > Decisions I made:
> > -----------------
> > 
> > 1) Database on SD card
> > 
> > I have whole system and even the ownCloud database on SD card. Reasoning
> > behind it is that even though it is slow, my externel harddrive is put
> > asleep most of the time and spinning it up would take quite some time.
> > So I prefer to get immediately something responding slowly than
> > something that needs to wake up and than performs. And I don't want to
> > have it spinning all the time as that is the point of going for ARM
> > board - saving up some energy.
> Yeah, but it will kill the SD card in a few weeks ;-)

Depends... Mine is running Tumbleweed fine for about a year. And yes,
for end users dying SD cards might be an issue. But as SD cards are
cheap, if it will backup itself once in a while on HDD and would be easy
to recover from the issue...

> These SD Cards SUCK at doing tiny writes - they have huge blocks
> (think 4MB or so) so every time you write, like 1KB of data, it writes
> 4 MB. Slow AND it will exhaust the number of writes the card can
> sustain in no time. SD cards are made for photo's, not databases :(

Yep, SQLite mimics pictures better than MySQL I would guess :-) But in
general, I would like to avoid spinning HDD as much as possible to save
energy and noise. But yes, performance is an issue with SD card.

> > 2) SQLite instead of MySQL
> > 
> > SQLite has plenty of issues, but I ended up using it instead of MySQL as
> > I have a feeling that on SD card it is faster. But didn't run any
> > benchmark and didn't tuned up MySQL much. On real drive, MySQL should
> > beat SQLite in performance without any problem, but I had the feeling
> > that writes/reads were much more scattered all around and thus slower
> > than SQLite when running on slow weird storage like SD card. Did anybody
> > did some benchmark? What is your experience? I actually migrated from
> > SQLite to MySQL and than back, but maybe I should have spend more time
> > tunning MySQL.
> Yeah, SQLite is cool - but not very good with ownCloud, really. We can
> try, yes, but then we might need to do some patching in ownCloud to
> make it more efficient with SQLite.
> That's the same as with NGNIX, btw, it works but there are rough areas
> so we might need to patch ownCloud. Then again, that is not a big
> problem, I have heard that ownCloud is open source so we can probably
> get those patches upstream :P
> So - don't listen too much to grumpy me, give it a try. But then a
> serious try: put the 1TB drive full of pictures and documents, then
> create 10 users, let 5 run the sync client and let 2 others view
> pictures.
> If that works fast, you're ready for release! ;-)

Yep, sounds like good stress test :-) In general, I think it doesn't
make sense to use SQLite on HDD, but we will see.

> > My performance boosting TODO:
> > -----------------------------
> > 
> > 1) Copy database to tmpfs on start and sync it back regularly to keep
> > persistence, gain speed but end up with possibility of loosing last X
> > minutes of history. Would require quite some scripting which I didn't
> > get to yet, but wondering whether somebody does have something like that
> > already and what people think about it :-)
> THAT is a great trick, yes, and that might work very well. Of course
> we need to think about losing data but if only the DB is updated and
> not the file system, it is probably fine.

Yep, the only issue might be stuff like calendar/contacts. Magic last
worlds, let's make it configurable and shift blame to the user :-D

But seriously, might be a way to avoid spinning disk while keeping up
the performance. With MySQL we can probably even split it per table and
some tables put directly on the disk (contacts/calender) and others in
memory (caches).

> > 2) Use squashfs + overlayfs for OS. Found out that reads from SD card
> > are quite slow and as I don't want to have my HDD spinning all the time,
> > sounds like some of the reads from OS can be speed up by using squashfs
> > and thus compressed filesystem. Would require special mounting on start
> > but could safe both space and time. Not for data, but should allow to
> > fit whole OS on small SD, maybe even multiple times and could improve
> > performance over direct SD card. Did some basic unfair testing and
> > looked like reading files from squashfs on top of SD card is faster than
> > reading from SD card directly.
> Our idea was to use the SD card exclusively as a boot loader, do
> everything from the hard drive... Because of the damage I talked about
> earlier.

I understand, not sure how much agree :-)

More information about the Devel mailing list