Cloud Station using too much disk space!

Discussion room for Synology Cloud Station package
Forum rules
This is a user forum for Synology users to share experience/help out each other: if you need direct assistance from the Synology technical support team, please use the following form:
https://myds.synology.com/support/suppo ... p?lang=enu

Re: Cloud Station using too much disk space!

Postby psyfer9983 » Sat Jun 01, 2013 8:10 pm

I gave up on Cloud Station. It still starts to grow every so often. Its like there is a file audit that runs from time to time and it makes new copies. I'm done with using Cloud Station until the next update thou I may go back to the previous version as I didn't have this problem on it.

Synology, PLEASE FIX THIS ASAP!
DS412+ with 4 x 2TB HDDs Hybird RAID -- 4 bay USB enclosure with 4 x 2TB HDDs non RAID for backup
psyfer9983
Student
Student
 
Posts: 73
Joined: Wed Jan 23, 2013 7:16 am

Re: Cloud Station using too much disk space!

Postby SpacedMonkey » Sat Jul 20, 2013 2:13 am

I also have a serious problem with disk usage. I have about 200Gb of videos on my DS. The disk usage was reported as 810Gb. 4 x what I have stored. Could find no obvious reason for this. After reading a few of the posts here I tried turning off the Cloud Station sharing on all my folders. 2 minutes later the DS was reporting 221Gb of disk storage.

This would kind of indicate that the Cloud Station is making 3 extra copies of all my files. Surely this can't be correct. It kind of make the Cloud Station almost useless. This effectively turns my 2Tb DS into a 500Gb DS - not very impressed.
SpacedMonkey
I'm New!
I'm New!
 
Posts: 1
Joined: Sat Jul 20, 2013 2:01 am

Re: Cloud Station using too much disk space!

Postby Bambam5 » Sat Aug 17, 2013 9:21 pm

Unfortunately I am experiencing the same problem!

Less than a week ago I bought my first Synology, a DS413j, mainly because of the CloudStation feature. Now it turns out I pretty much threw away 400 euro! :-(
2 days ago I set up CloudStation on my Windows 7 laptop, syncing with my homes/<username> folder of about 13 GB. That seemed to work OK, except for a problem with Thunderbird and Firefox lockfiles (I wrote a separate post on this).
So I decided to copy my music and video collection to a Shared folder, so that my family can access it (this is why I bought the DS413j in the first place).
The Shared folder was configured for use via CloudStation. The data (188GB) was already on the DS (in a non-shared folder) so I only had to move it to the Shared folder.
Shortly thereafter I notice that the CPU is continuously at 100%, and that 2 processes syncd (75%) and client-linux (25%) are the culprits. Also I notice a lot of continuous disk activity (15 - 20 MB/s read and write) without any network data transfer. Next I notice the amount of Used data on my DS volume steadily increasing.
So I start googling. By the time Used data is at 600GB I find this thread and decide to take the Shared Folder out of the CloudStation configuration.
Immediately the CPU load drops to 30% and the amount of Used data starts decreasing instead of increasing.

Now I'm back at a stable 375GB in use (as reported by Storage Manager). The amount of data I put on the DS is 350GB, so still a difference of 25GB.
Although I can live with that, I still don't understand why DS needs it.
Is there a way (preferably easy) to find out where these 25GB have gone? I'm running DSM 4.2-3211 (latest released version).


Sure hope this will be fixed soon in DSM 4.3! I bought Synology because of reputation and features, but CloudStation is a huge disappointment!

Any comments or suggestions for alternative solutions are welcome as I really need this feature!
DS413j / DSM 5.0-4458
Bambam5
Beginner
Beginner
 
Posts: 21
Joined: Fri Aug 16, 2013 11:21 am

Re: Cloud Station using too much disk space!

Postby psyfer9983 » Sat Aug 17, 2013 11:48 pm

I went back to an older version of CS. Same problem. I'm thinking of trying ownCloud. If it has the same features as CS, and if it works, I will stick with it. I'm still wandering why Synology hasn't addressed this problem or at least posted a response. This has been know for a couple of months now.
DS412+ with 4 x 2TB HDDs Hybird RAID -- 4 bay USB enclosure with 4 x 2TB HDDs non RAID for backup
psyfer9983
Student
Student
 
Posts: 73
Joined: Wed Jan 23, 2013 7:16 am

Re: Cloud Station using too much disk space!

Postby fatman01923 » Sun Aug 18, 2013 1:55 am

I gave up on cloudstation a long time ago. I highly recommended using Windows, Mac, or a Linux workstation to manage your backups and syncs. I use Dirsyncpro and it works well for me check it out. :D
Need help making Photostation, Videostation, or AudioStation properly index your files go here for this tutorial.
http://forum.synology.com/enu/viewtopic.php?f=19&t=66999
fatman01923
I'm New!
I'm New!
 
Posts: 9
Joined: Sat Mar 30, 2013 12:54 am

Re: Cloud Station using too much disk space!

Postby foopydog » Sun Aug 25, 2013 7:26 pm

Why don't we hear from Synology? This is the voice of your users. Are you listening?

Users quickly discovered that enabling shares on CloudStation creates a repository at least as large as the underlying share. This information should be published, so that users understand the storage requirements before enabling shares on CloudStation willy-nilly. That way, system administrators can plan for storage. I believe that all packages and services should publish these requirements.

For CloudStation, it should explain that when you "enable sharing" of a share with Could station, it will build a repository at least as large as the share itself. There will be heavy disk activity during the building of the repo, and the Diskstation might be slow during that time.

Here are some examples from practical experience:
1. Cloudstation on the DS211. During this repo-building stage, the NAS was crippled. CPU was at 90%, and Disk Utilization north of 90% too. Accessing shares with Samba was so slow that it would time-out. At a rate of about 70GB/hr, the DS211 would have taken 3 days to complete building the repo.
2. Cloudstation on the DS713. During this repo-building stage, CPU was at 23%, and storage was at 90% utilization, but the NAS didn't break a sweat, and was perfectly usable. It took 10 hours to create 2TB of repositories, at a rate of about 200GB/hour.

My guess is that this 'repository' is a Subversion Repo, and that's how it is able to maintain multiple versions of files, and tracking whether the client is in sync with the file. That is, when you sync a file on your cloud, you are "checking out" a file from the Subversion repo. This is a good system if you are dealing with text files, but photos and movies are binary files, and not suitable for this type of revision-control. Storing 16 versions of the file requires up to 16x as much storage for binaries, but a tiny fraction of that for text files.

Other problems: you can't just share a folder in the Cloud, you have to share the entire Shared Folder and everything in it. The repo and the file itself are duplicates.

In principle, CloudStation is about making files available on the Cloud, but in reality, it's a WAN-based multi-user revision-control system. That's nice is that if two people want to use a file, they are protected from clobbering each other's work. On the downside, it will eat storage like there's no tomorrow, building a repository of a billion files, even if you only wanted to share a handful of them, or share them read-only.

What's wrong with an optimistic strategy? The last modification wins. It works with Samba. Then we don't need the stupid repository. Yes, you could just set up a VPN. The DiskStation already supports that. But for me, the problem is that i don't have a publicly routable IP address. Yes, my ISP has a NAT router to share IP addresses. So a server-based cloud, such as CloudStation is good, because i don't need port-forwarding to access files outside my firewall.

So i guess for now, the CloudStation is a unanimous flop. I'll be disabling the service to reclaim my 2TB of lost storage. Here is what it requires:
* The option to choose which folders to share (like you do with a Backup job)
* The option to choose whether each of those folders supports revision history or not.
* Folders without any revision history use an optmistic strategy for any modificaitons.

I suspect that if Synology adopted a Versioning File System, like the ReadyNAS does with the btrfs, that would enable them to do a better Cloud implementation, with multiple-versions and snapshots, and without needing a repo which is nothing but a duplicate of the file itself.

I still love my Synology DS713+ and i wouldn't trade it for a ReadyNAS, and i'll never go back to anything less than a busiess-grade (+) device, as the lesser NAS devices have so little CPU power, they are not good for doing anything other than file-sharing.
foopydog
I'm New!
I'm New!
 
Posts: 9
Joined: Mon May 31, 2010 10:07 pm

Re: Cloud Station using too much disk space!

Postby Bambam5 » Thu Aug 29, 2013 1:46 pm

Did some testing with DSM 4.3 and the new version of CS and Windows client, to see if anything changed. Although the new features are great (ability to select which folders to sync, syncing process no longer blocks on lock files) the disk usage does not seem to have improved.

As a test I added 21.9 GB of data (24720 files in 1556 folders) to a new CS shared folder. Syncing took 3 hours over a 1 Gbps LAN connection. Peak transfer rate observed was 150 Mbps.
After syncing, the amount of used storage on the volume had increased by 43.3 GB (there was no other activity on the DS during syncing). So that's almost a factor of two! This is inline with earlier reports on disk usage.

So CS still has room for improvement regarding this topic...
DS413j / DSM 5.0-4458
Bambam5
Beginner
Beginner
 
Posts: 21
Joined: Fri Aug 16, 2013 11:21 am

Re: Cloud Station using too much disk space!

Postby psyfer9983 » Thu Aug 29, 2013 7:22 pm

I just installed the new CS version but haven't tested it yet. I would like to see an option to disable file/history versions and use the CS as a live folder sync only. I don't need multiple versions of files/folders, I just need one version that is synced across all my PCs. If file/history versions must be on, at least do what Dropbox does. They delete old versions of 30 days or so but but for me, I need one copy of a file that doesn't use any more space than the file's size itself. I would just use a network share for this but I need LAN and WAN access. I have tried using a VPN and got it work but the connection drops out when doing file transfers. Right now I have a 5Gb Dropbox account that I'm using and it getting close to max. I really need CS to work or an alternative to it that I can host on my DS412+.

So to Synology, I request an option to remove or disable the file/history versions and use the CS as a live file/folder sync. This would solve my problems altogether.
DS412+ with 4 x 2TB HDDs Hybird RAID -- 4 bay USB enclosure with 4 x 2TB HDDs non RAID for backup
psyfer9983
Student
Student
 
Posts: 73
Joined: Wed Jan 23, 2013 7:16 am

Re: Cloud Station using too much disk space!

Postby Bambam5 » Thu Aug 29, 2013 10:16 pm

Did you try to set the Maximum Versions Settings to zero in the CloudStation Settings? I would expect that to disable the file versions history. From the information in CS help system I gather that the Maximum version number settings are applied only the next time a file is modified though... However I have not tried this myself.
DS413j / DSM 5.0-4458
Bambam5
Beginner
Beginner
 
Posts: 21
Joined: Fri Aug 16, 2013 11:21 am

Re: Cloud Station using too much disk space!

Postby psyfer9983 » Fri Aug 30, 2013 8:58 pm

Bambam5 wrote:Did you try to set the Maximum Versions Settings to zero in the CloudStation Settings? I would expect that to disable the file versions history. From the information in CS help system I gather that the Maximum version number settings are applied only the next time a file is modified though... However I have not tried this myself.


Yes I did try 0 but CS says the lowest setting is 1. Setting it to 1 on older versions (I still haven't tested the new version) makes 2 copies of the files/folders.
DS412+ with 4 x 2TB HDDs Hybird RAID -- 4 bay USB enclosure with 4 x 2TB HDDs non RAID for backup
psyfer9983
Student
Student
 
Posts: 73
Joined: Wed Jan 23, 2013 7:16 am

Re: Cloud Station using too much disk space!

Postby WessexUnderwater » Sun Sep 01, 2013 6:18 pm

I have the same issue with DSM 4.3 and CS 2.1-2561.

Cloudstation seems to be wrong for managing digital media for the home, for at least two reasons
1) It's version retention overhead is OTT for binary files with a few K of metadata per gig of static, usually replaceable, digital media.
2) Synology's individual media apps, such as videostation, assume files are physically stored according to a breakdown - Movies, TV Programmes, etc. It makes syncing between autonomous devices unlikely to work well unless you are going to make sure the same directory regime applies on all those devices, which is in conflict with the way many of the media devices on those apps order their data.

With retrospect, a setup where the master store of data is on the NAS, accessed by all devices, and with a backup on a different device seems a better way to go.

This is a great thread for confirming its not just my ham-fistedness that caused the problem, but, like Foopydogs masterly post suggested, I'd rather Synology were telling me beforehand what wouldn't work. I think there is a conflict between Synology's marketing to home users, which is essentially 'It just works' and the technical reality that you have to think quite hard about how to get it right. I suspect Synology aren't addressing the technical issues users have for fear of blurring the marketed image. I wouldn't have bought mine if I had known how hard it was going to be to set up - proving their decision to be commercially the right one in the short term. However I won't buy or recommend if I can't get the best out of it, so in that respect they are wrong.

They could do a lot better if they oriented their documentation (and probably apps) around how to achieve typical end goals like 'Manage, Share and Secure your Digital Media' instead of how to enable technical components.
WessexUnderwater
I'm New!
I'm New!
 
Posts: 2
Joined: Sun Aug 11, 2013 9:41 am

Re: Cloud Station using too much disk space!

Postby psyfer9983 » Sun Sep 01, 2013 10:37 pm

WessexUnderwater wrote:I have the same issue with DSM 4.3 and CS 2.1-2561.

Cloudstation seems to be wrong for managing digital media for the home, for at least two reasons
1) It's version retention overhead is OTT for binary files with a few K of metadata per gig of static, usually replaceable, digital media.
2) Synology's individual media apps, such as videostation, assume files are physically stored according to a breakdown - Movies, TV Programmes, etc. It makes syncing between autonomous devices unlikely to work well unless you are going to make sure the same directory regime applies on all those devices, which is in conflict with the way many of the media devices on those apps order their data.

With retrospect, a setup where the master store of data is on the NAS, accessed by all devices, and with a backup on a different device seems a better way to go.

This is a great thread for confirming its not just my ham-fistedness that caused the problem, but, like Foopydogs masterly post suggested, I'd rather Synology were telling me beforehand what wouldn't work. I think there is a conflict between Synology's marketing to home users, which is essentially 'It just works' and the technical reality that you have to think quite hard about how to get it right. I suspect Synology aren't addressing the technical issues users have for fear of blurring the marketed image. I wouldn't have bought mine if I had known how hard it was going to be to set up - proving their decision to be commercially the right one in the short term. However I won't buy or recommend if I can't get the best out of it, so in that respect they are wrong.

They could do a lot better if they oriented their documentation (and probably apps) around how to achieve typical end goals like 'Manage, Share and Secure your Digital Media' instead of how to enable technical components.

I agree with you to a point. That point is I would still buy my DS412+ again. It does what I had a PC doing but saves me on overhead. I have retired that old PC since buying a DS212j, then I bought the DS412+ due to needing more drives, faster CPU and RAM. Over all, I'm very happy with my DS412+ though like others, Synology could improve on it. But this is with everything you get today. I still haven't had the time to test CS for myself and it was the only draw back I had.
DS412+ with 4 x 2TB HDDs Hybird RAID -- 4 bay USB enclosure with 4 x 2TB HDDs non RAID for backup
psyfer9983
Student
Student
 
Posts: 73
Joined: Wed Jan 23, 2013 7:16 am

Re: Cloud Station using too much disk space!

Postby cago » Wed Sep 04, 2013 12:16 pm

I have the same problem and just wrote the support. I hope, there ist a solution. As soon as I know more I let you know. Or does someone already have a solution?
cago
I'm New!
I'm New!
 
Posts: 1
Joined: Wed Sep 04, 2013 12:00 pm

Re: Cloud Station using too much disk space!

Postby Daluxe » Fri Sep 06, 2013 4:35 pm

The problem seems to be the repository's settings.

By default the value seems to be 32 versions. So Cloudstation will save up to 32 versions, each time you sync your folders. (So if you sync 10GB, it will eat up to 320GB... that's plain crazy)

Here's how I solved it:

1) Deactivate CloudStation
2) Reactivate it
3) Go to the Synology settings and lower the max. versions. (I set it to 2)
4) Optional: Let Cloudstation rebuild the database, if it fails (because there won't be enouzgh free space first).
5) You might have to activate the users and shared folders again.
6) ENJOY your CloudStation again ;)

If you guys need further assistance, feel free to PM me :)

cheers from Austria and have a nice weekend.
Daluxe
I'm New!
I'm New!
 
Posts: 2
Joined: Mon Jul 15, 2013 10:27 am

Re: Cloud Station using too much disk space!

Postby refriend » Sun Oct 13, 2013 9:57 pm

Just did a test.
Fresh install of CloudStation (2.1-2561) on my DS411 (DSM 4.3-3776).
Fresh install of CloudStation (v2561) on Win7 PC.
Sync'd only my 15.8 GB photos folder from Win7 PC.
The DS Photo folder already had a copy of all photo files.
Set CloudStation versioning to keep only one version.
Synology Available space went from 1399.5 GB to 1315.81 GB
So 15.8GB of data consumed 83.69GB of space on the NAS. Sounds like a problem to me.
refriend
I'm New!
I'm New!
 
Posts: 1
Joined: Mon Dec 26, 2011 4:15 pm

PreviousNext

Return to Cloud Station

Who is online

Users browsing this forum: No registered users and 2 guests