[Tux3] Deduplication?

[Daniel Phillips]

On Thursday 16 April 2009, Ray Van Dolson wrote:
> Potential end user here, hope you don't mind the intrusion. :-)
> 
> I've read a bit here in the past on deduplication coming to Tux3.  I'm
> thinking of this in terms of block-based deduplication, a la what we
> have on WAFL with NetApp.  Will Tux3 have something along these lines?
> Does it already?
> 
> We love dedupe for its application in virtualization environment (for
> storing VMDK files specifically, which are great candicates for
> deduplication).  However, I know of no other filesystem other than WAFL
> that has this functionality, and others like ZFS don't seem to be in a
> hurry to add it for whatever reason.  Maybe they don't see the value as
> anything more than a niche thing more for backup administrators?
> 
> Thanks!
> Ray

Our Pune Institute volunteers put in the work to demonstrate the
effectiveness of deduplication, and to get a respectable level of
functionality working.  This pretty much decides the question for me.
Block level deduplication will be supported by Tux3.

There are competing approaches to deduplication that should also be
investigated.  Deduplication can be implemented at three different
layers in the storage stack:

  1) Block level
  2) Filesystem level
  3) Stacked filesystem level

It is not immediately clear to me which is best.  At least, by
implementing at the filesystem level, there is no additional cost for
storing metadata blocks.  I suspect that the third approach, stacking
filesystem, will eventually prove superior, as it removes complexity
from the filesystem while not imposing much additional overhead.  But
that is for the future.  Currently, the only effective way to stack a
filesystem is by implementing it in user space under FUSE, and the FUSE
codebase still looks a little young to me and not well suited to high
volume applications.  That could change in time, however, the current
approach as part of the filesystem has the advantage of existing and
apparently functions pretty well.

Ongoing development of deduplication will be needed.  I do not intend
to put time into that myself, because other areas are more pressing.
Volunteers to continue the work initiated by the Pune students would be
welcome.  First, the code needs to be forward ported to the current
development tree, which would be an easy way for a volunteer or
volunteers to get started.  Then I would like to see the new "physical"
btree used by the Pune code to store the block database changed to a
logical form, where the new data is stored logically in a file.  As it
is, the physical form of the database would impose additional
requirements on the atomic commit logging/replay mechanism, adding
complexity.  It would be preferable that deduplication not add code to
the atomic commit path.

Regards,

Daniel

_______________________________________________
Tux3 mailing list
Tux3 at tux3.org
http://mailman.tux3.org/cgi-bin/mailman/listinfo/tux3