[Tux3] a few thoughts about REALLY outstanding features

[Daniel Phillips]

On Thursday 18 December 2008 05:09, myLC at gmx.net wrote:
> I hope I'm not bumping in here being a disturbance with my
> ideas, since I'm not much of a "FS affectionate".
> 
> First of all, I know about ZFS, btrfs, XFS and the like.
> While there are surely big differences they ALL lack
> functionality that I would like to see as a programmer and
> user.

It is good to see a reminder that there will still be filesystems 
research left for others to do, even after Tux3 is working 
perfectly :-)

> To me current filesystems are still based on the old C64
> Datasette (call it a streamer). Basically you are able to
> append/overwrite data to/from files and that's it.
>
> Let's take a real world example: Video editing.
> Suppose you have a 20 GB file (almost normal if the stream
> is in HD) and you want to cut it (i.e.: take out bits from
> the beginning, the middle and the end).
> Currently the way this is being done is by copying the file,
> leaving out the unwanted parts and then deleting the
> original. This brings up two problems:
> 1. If you're low on HD space you're screwed.
> 2. It is DEAD SLOW (especially on a DVB-receiver with 32 MB
>    of RAM, a 250 MHz CPU and a slow IDE interface (about 40
>    MB/s tops and in this case - with the drive's heads
>    jumping forth and back constantly - a lot less).

If you are writing a video editing application, you probably should
design an intermediate representation that uses multiple files and
keeps track of them with an index.

> Is it only me thinking that this is a silly way of handling
> such things on a block oriented filesystem (preferably
> supporting sparse blocks)? Shouldn't you be able to:
> - insert/prepend a chunk of data into a file
> - remove a chunk from a file
> - move a chunk from one file into another
> 
> As there is no such standard yet this would have to be
> implemented using (ugly) IOCTL calls (or such) currently, of
> course.
> 
> All of the above would be very useful when dealing with
> large data, such as DVB-recordings (i.e.: video) or virtual
> disk files (virtualization). This is also interesting for
> large databases. Currently they are implementing "their own"
> filesystems on top of other filesystems - which would then
> be superfluous.
> 
> Not having this kind of functionality seems even sillier
> with truly random access "flash drives" slowly invading the
> territory. While an operation as mentioned above could take
> an hour on a USB stick the very same task could be done in a
> few milliseconds with a "more intelligent" approach.
> 
> While the variances in performance between different
> filesystem (say ZFS, XFS and ext3) usually are around one
> digit in factor AT MOST (i.e. factor 2 or 3), being able to
> insert/remove from within a file could easily give you a
> performance plus of FACTOR 1000 and more (even on a fast
> harddrive). This would make a BIG difference and would thus
> be a truly outstanding feature easy to advertise (magazines
> would talk about it, manufacturers of embedded - video -
> devices would think about using it).
> 
> Another thing that bugs me is that filesystems do not
> distinguish between fast and slow zones on harddrives.
> The difference between reading/writing on/to the inner rim
> and the outer rim of a harddrive is about factor two. There
> should be a way to indicate a preference (priority) on where
> the file should land on the harddrive/partition.
> 
> I expressed both of these ideas on several discussion forums
> about filessystems before (for instance on the btrfs mailing
> list) - unfortunately without any luck.
> 
> So is your filesystem going to be "just another filesystem"
> as well? Would the above be so hard to implement?
> Either of these features would make your filesystem truly
> outstanding.

Just another filesystem I am afraid, with one big advance (versioning) 
and a number of incremental ones.  This time around, we aim to 
implement existing practice better, keep the code base tight, and add 
some features that are show stoppers if Linux doesn't have them.

My theory is, if we really focus on just a few things and do those as 
well as we can, and satisfy the really common use cases like desktop 
and NFS server, that puts us in a position to play, and for other 
people to play with our code.  And other people will be able to play a 
lot more easily if we continue our focus on small and tight and really 
convenient to work on.

Then just grab the code and start playing with pointers.  We will do 
this much for you:  we will take special care to make it easy to change 
block pointers around on the fly.  That's needed for base capabilities 
like volume shrink.  You can use it for your magic pointers project.

Regards,

Daniel

_______________________________________________
Tux3 mailing list
Tux3 at tux3.org
http://mailman.tux3.org/cgi-bin/mailman/listinfo/tux3