btrfs Deduplication on Synology with duperemove

Questions and mods regarding system management may go here
Forum rules
1) This is a user forum for Synology users to share experience/help out each other: if you need direct assistance from the Synology technical support team, please use the following form:

https://account.synology.com/support/support_form.php?lang=enu



2) To avoid putting users' DiskStation at risk, please don't paste links to any patches provided by our Support team as we will systematically remove them. Our Support team will provide the correct patch for your DiskStation model.
grintor
I'm New!
I'm New!
Posts: 1
Joined: Thu Dec 29, 2016 4:03 pm

btrfs Deduplication on Synology with duperemove

Postby grintor » Thu Dec 29, 2016 4:23 pm

After reading about btrfs block-level deduplication capabilities https://btrfs.wiki.kernel.org/index.php/Deduplication, I expected to find support for this in my Synology NAS. When I didn't I decided to make it so.

I thought I would share with the community my findings. I statically compiled http://markfasheh.github.io/duperemove/ for the Synology. I compiled both the stable v0.10 and the v0.11 beta 4 (latest)

I ran it on my synology and it works, but it's much too memory intensive to just point at the root of the filesystem, unfortunately (even using --hashfile)

I wish synology would integrate btrfs deduplication into the OS. It could be done in a less resource intensive way -- just a low priority process that's always running in the background and looking for duplicate blocks -- saving it's findings to a file rather than memory.

duplicate blocks -- that's the amazing thing to me. This isn't a file-level feature, it's block-level. So even if two files are 90% different, it can deduplicate away that 10%.

Anyway, here are the binaries and my notes on how I statically compiled and ran it:

https://drive.google.com/open?id=0B211Q ... XdWTTNsUWM
User avatar
Eideen
Enlightened
Enlightened
Posts: 416
Joined: Sat Jun 16, 2012 11:57 am
Location: Norway

Re: btrfs Deduplication on Synology with duperemove

Postby Eideen » Sun Jan 08, 2017 2:57 pm

Great work man.

I send synology support and ask if they support it.
DS412+ with DX510/ DSM 6.1b / 2xWD 4TB red+2(+1)xWD6TB (RAID5), 240GB SSD
DS215j / DSM 6.0 / 2xSG3TB (RAID 1) / remote backup
cyberpower BS650E, 1x2TB external drive for backup.
Christian72D
I'm New!
I'm New!
Posts: 1
Joined: Fri Mar 21, 2014 10:21 am
Location: Germany

Re: btrfs Deduplication on Synology with duperemove

Postby Christian72D » Mon Apr 10, 2017 5:15 am

Can you tell me how to use the tool?
Do i need to run it any x hours per cron?
How do i need to start ist?

THIS was THE killing feature of btrfs for ME...
xRoThx
Trainee
Trainee
Posts: 11
Joined: Tue Sep 12, 2017 9:34 pm

Re: btrfs Deduplication on Synology with duperemove

Postby xRoThx » Sat Sep 30, 2017 8:25 pm

Hello Synology

I would like to bring this topic to your attention again.

Please consider this feature as for me, and my customers indirectly.. This could be a huge money-saver.

Maybe, as the need for personal use would be pretty low, you can make it as a paid application available on the application store.
I would be very interested.

Thank you for considering this.

Thomas :)
User avatar
Eideen
Enlightened
Enlightened
Posts: 416
Joined: Sat Jun 16, 2012 11:57 am
Location: Norway

Re: btrfs Deduplication on Synology with duperemove

Postby Eideen » Sat Sep 30, 2017 9:11 pm

According to Synology (I can't find the statement), and the Deduplication info on Btrfs, [Deduplcation] typically requires large amounts of RAM to store the lookup table of known block hashes.
DS412+ with DX510/ DSM 6.1b / 2xWD 4TB red+2(+1)xWD6TB (RAID5), 240GB SSD
DS215j / DSM 6.0 / 2xSG3TB (RAID 1) / remote backup
cyberpower BS650E, 1x2TB external drive for backup.
xRoThx
Trainee
Trainee
Posts: 11
Joined: Tue Sep 12, 2017 9:34 pm

Re: btrfs Deduplication on Synology with duperemove

Postby xRoThx » Sun Oct 01, 2017 7:54 am

Hello

Than they could offer it to all Synology Plus models with 8gb ram?
As most of the home users won't need this feature, they could be aiming on the higher end of their products.

Grtz
T
ByteSizedAlex
I'm New!
I'm New!
Posts: 1
Joined: Tue Oct 24, 2017 8:50 pm

Re: btrfs Deduplication on Synology with duperemove

Postby ByteSizedAlex » Tue Oct 24, 2017 8:56 pm

Eideen wrote:According to Synology (I can't find the statement), and the Deduplication info on Btrfs, [Deduplcation] typically requires large amounts of RAM to store the lookup table of known block hashes.


In-line dedupe certainly would require a large memory set to minimise the performance impact of hash lookup. If it was implemented as a background/scheduled process the memory requirement could be avoided with the acceptance the task takes longer to complete - it really just comes down to how accepting you are of a write impact. Would certainly be nice to have the option to run dedupe on a Synology, perhaps as a hidden/advanced option with caveats and disclaimers to try and avoid people implementing without realising the consequences.

Return to “System Managment Mods”

Who is online

Users browsing this forum: No registered users and 2 guests