[owncloud-devel] Experience with WD Chalenge like setup

Johan Ouwerkerk jm.ouwerkerk at gmail.com
Thu Dec 17 23:16:55 GMT 2015


It would kinda depend on the SD card, no? You've got these endurance
SD cards, too, which are used for e.g. security cameras and they're
good for continuous 24x7 1080p video capture. But on the flipside your
average user is more likely to use whatever SD card they have on hand,
and so they might well have the cheapest binned product that would die
if you looked at it funnily.

... And then there's the tangential issue of knock-off SD cards that
conform to relevant SD* standards but don't do SPI. These might not
boot properly (some u-boot require SPI access for booting off mmc
devices, it's what the generic u-boot mmc driver uses).

On Thu, Dec 17, 2015 at 11:23 PM, Michal Hrusecky <michal at hrusecky.net> wrote:
> Jos Poortvliet - 22:36 17.12.15 wrote:
>> On Thursday 17 December 2015 22:07:34 Michal Hrusecky wrote:
>> > Hi,
>> >
>> > while discussion is going on, I have a few questions regarding everyones
>> > experience with running similar setup to what is proposed in WD
>> > Challenge. I have at home BananaPi with external USB harddrive running
>> > ownCloud now and while installing it I did few decisions that might be
>> > relevant to the challenge as well, so I was wondering whether you have
>> > similar experience/setup or something else (will help us shape the image
>> > and improve our home setups).
>> >
>> > Decisions I made:
>> > -----------------
>> >
>> > 1) Database on SD card
>> >
>> > I have whole system and even the ownCloud database on SD card. Reasoning
>> > behind it is that even though it is slow, my externel harddrive is put
>> > asleep most of the time and spinning it up would take quite some time.
>> > So I prefer to get immediately something responding slowly than
>> > something that needs to wake up and than performs. And I don't want to
>> > have it spinning all the time as that is the point of going for ARM
>> > board - saving up some energy.
>>
>> Yeah, but it will kill the SD card in a few weeks ;-)
>
> Depends... Mine is running Tumbleweed fine for about a year. And yes,
> for end users dying SD cards might be an issue. But as SD cards are
> cheap, if it will backup itself once in a while on HDD and would be easy
> to recover from the issue...
>
>> These SD Cards SUCK at doing tiny writes - they have huge blocks
>> (think 4MB or so) so every time you write, like 1KB of data, it writes
>> 4 MB. Slow AND it will exhaust the number of writes the card can
>> sustain in no time. SD cards are made for photo's, not databases :(
>
> Yep, SQLite mimics pictures better than MySQL I would guess :-) But in
> general, I would like to avoid spinning HDD as much as possible to save
> energy and noise. But yes, performance is an issue with SD card.
>
>> > 2) SQLite instead of MySQL
>> >
>> > SQLite has plenty of issues, but I ended up using it instead of MySQL as
>> > I have a feeling that on SD card it is faster. But didn't run any
>> > benchmark and didn't tuned up MySQL much. On real drive, MySQL should
>> > beat SQLite in performance without any problem, but I had the feeling
>> > that writes/reads were much more scattered all around and thus slower
>> > than SQLite when running on slow weird storage like SD card. Did anybody
>> > did some benchmark? What is your experience? I actually migrated from
>> > SQLite to MySQL and than back, but maybe I should have spend more time
>> > tunning MySQL.
>>
>> Yeah, SQLite is cool - but not very good with ownCloud, really. We can
>> try, yes, but then we might need to do some patching in ownCloud to
>> make it more efficient with SQLite.
>>
>> That's the same as with NGNIX, btw, it works but there are rough areas
>> so we might need to patch ownCloud. Then again, that is not a big
>> problem, I have heard that ownCloud is open source so we can probably
>> get those patches upstream :P
>>
>> So - don't listen too much to grumpy me, give it a try. But then a
>> serious try: put the 1TB drive full of pictures and documents, then
>> create 10 users, let 5 run the sync client and let 2 others view
>> pictures.
>>
>> If that works fast, you're ready for release! ;-)
>
> Yep, sounds like good stress test :-) In general, I think it doesn't
> make sense to use SQLite on HDD, but we will see.
>
>> > My performance boosting TODO:
>> > -----------------------------
>> >
>> > 1) Copy database to tmpfs on start and sync it back regularly to keep
>> > persistence, gain speed but end up with possibility of loosing last X
>> > minutes of history. Would require quite some scripting which I didn't
>> > get to yet, but wondering whether somebody does have something like that
>> > already and what people think about it :-)
>>
>> THAT is a great trick, yes, and that might work very well. Of course
>> we need to think about losing data but if only the DB is updated and
>> not the file system, it is probably fine.
>
> Yep, the only issue might be stuff like calendar/contacts. Magic last
> worlds, let's make it configurable and shift blame to the user :-D
>
> But seriously, might be a way to avoid spinning disk while keeping up
> the performance. With MySQL we can probably even split it per table and
> some tables put directly on the disk (contacts/calender) and others in
> memory (caches).
>
>> > 2) Use squashfs + overlayfs for OS. Found out that reads from SD card
>> > are quite slow and as I don't want to have my HDD spinning all the time,
>> > sounds like some of the reads from OS can be speed up by using squashfs
>> > and thus compressed filesystem. Would require special mounting on start
>> > but could safe both space and time. Not for data, but should allow to
>> > fit whole OS on small SD, maybe even multiple times and could improve
>> > performance over direct SD card. Did some basic unfair testing and
>> > looked like reading files from squashfs on top of SD card is faster than
>> > reading from SD card directly.
>>
>> Our idea was to use the SD card exclusively as a boot loader, do
>> everything from the hard drive... Because of the damage I talked about
>> earlier.
>
> I understand, not sure how much agree :-)
>
> _______________________________________________
> Devel mailing list
> Devel at owncloud.org
> http://mailman.owncloud.org/mailman/listinfo/devel


More information about the Devel mailing list