Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Discuss with the community any ideas you'd love to see in future DiskStations and DSM updates! We do our best to monitor and forward all of them, but we recommend to also use this form as our team will systematically see your suggestion:
https://www.synology.com/form/inquiry/feature
Forum rules
1) This is a user forum for Synology users to share experience/help out each other: if you need direct assistance from the Synology technical support team, please use the following form:

https://account.synology.com/support/su ... p?lang=enu



2) To avoid putting users' DiskStation at risk, please don't paste links to any patches provided by our Support team as we will systematically remove them. Our Support team will provide the correct patch for your DiskStation model.

Poll: directory filtering and support for '.ignore' file in Cloud Station?

Being able to filter/ignore folders from the client application,
23
37%
Being able to filter/ignore folders from the server application,
19
31%
Being able to filer files and folders via a local .ignore file.
20
32%
 
Total votes: 62

frqnck
I'm New!
I'm New!
Posts: 3
Joined: Wed Feb 03, 2016 1:37 pm

Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Unread post by frqnck » Wed Feb 03, 2016 2:33 pm

At present, Cloud Station does not support filtering/ignoring folders which is very limiting to say the least... only file can be filtered. It is also missing regex like filetring.

Another feature that would be greatly appreciated would be option to use a local '.ignore' file of some kind just like .gitignore (see //git-scm.com/docs/gitignore).

These features are serious omissions.

Are these on the Cloud Station product roadmap? If so, is there any ETA?

hayyatlost
I'm New!
I'm New!
Posts: 2
Joined: Sat Apr 02, 2016 11:39 am
Contact:

Re: Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Unread post by hayyatlost » Sat Apr 02, 2016 11:48 am

Being able to filter/ignore folders from the client application

lexar
I'm New!
I'm New!
Posts: 1
Joined: Sun Apr 17, 2016 9:05 am

Re: Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Unread post by lexar » Sun Apr 17, 2016 9:18 am

The current filter in the Cloud Station seems not to be applied to the directory names -- why not?!?

As a result I have tons of empty folders, because the LightRoom preview files are filtered out (*.lrprev) but the whole directory tree is backuped :evil:

What is the problem with applying the filter rule to the directory name as well? This is probably just two lines of code...

joah
I'm New!
I'm New!
Posts: 1
Joined: Tue Nov 08, 2016 9:45 am

Re: Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Unread post by joah » Tue Nov 08, 2016 10:00 am

There is a way to blacklist folders globally in the Cloud Station client. Well, it is not working in the GUI.

Cloud Station has of course the ability to manage blacklisting folders. So please put that in GUI. You are actually frustrate dev users - really noone wants to sync Git folders or node_modules (typically 20-30k files) or bower_components or cache directories.

You have to edit the file:
Windows: C:/Users/joah/AppData/Local/CloudStation/CloudStation.app/conf/filter-v4150
Ubuntu like OS: ~/.CloudStation/CloudStation.app/conf/filter-v4150
Mac: don't know

Just add the folders that you not want to get synced right below [Directory]. But be carefull, once the Client is updated the config file can get overwritten.

Code: Select all

[Version]
major = 1
minor = 1

[Common]
black_char = "\\/"
black_name = "@tmp", "@eaDir", ".SynologyWorkingDirectory", "#recycle", "desktop.ini", ".ds_store", "Icon\r", "thumbs.db", "$Recycle.Bin", "@sharebin", "System Volume Information", "Program Files", "Program Files (x86)", "ProgramData", "#snapshot"
max_length = 255
max_path = 2048

[File]
black_name = "@tmp", "@eaDir", ".SynologyWorkingDirectory", "#recycle", "desktop.ini", ".ds_store", "Icon\r", "thumbs.db", "$Recycle.Bin", "@sharebin", "tmp", "temp", "System Volume Information", "Program Files", "Program Files (x86)", "ProgramData", "#snapshot"
black_prefix = "~"
max_size = 0

[Directory]
black_name = ".git", ".stversions", "bower_components", "node_modules", "@tmp", "@eaDir", ".SynologyWorkingDirectory", "#recycle", "desktop.ini", ".ds_store", "Icon\r", "thumbs.db", "$Recycle.Bin", "@sharebin", "System Volume Information", "Program Files", "Program Files (x86)", "ProgramData", "#snapshot"

[EA]

jhbriese
I'm New!
I'm New!
Posts: 1
Joined: Thu Nov 17, 2016 2:30 pm

Re: Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Unread post by jhbriese » Thu Nov 17, 2016 2:33 pm

Thank you joah,

this solution works great for me. I had massive problems with node_modules and bower_components in the past.
BTW the path for Mac: ~/.CloudStation/CloudStation.app/Contents/Resources/conf/filter-v4150

Thanks
Jan

jwktje
I'm New!
I'm New!
Posts: 2
Joined: Tue Jan 10, 2017 2:05 pm

Re: Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Unread post by jwktje » Tue Jan 10, 2017 2:10 pm

Joah and jhbriese you are heroes!
This has stopped me from replacing my Synology NAS.
I use Cloudsync to backup all my webdevelopment projects and (mostly) because of node_modules taking ages to index, the daemon service was eating up my CPU almost daily.

After changing the config it now works great! However it would be way better if you could change this in the GUI.
Because as pointed out by joah, any update might overwrite the file.

jwktje
I'm New!
I'm New!
Posts: 2
Joined: Tue Jan 10, 2017 2:05 pm

Re: Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Unread post by jwktje » Wed Jan 11, 2017 12:42 am

Exactly what I didn't hope happened within a matter of hours.
Cloud station was updated and the config file was overwritten.
Hope they update this soon so I don't have to manually update this file after every update to prevent my CPU from going into 100% usage from indexing.

jeastbur
I'm New!
I'm New!
Posts: 7
Joined: Sat Oct 03, 2015 5:40 pm

Re: Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Unread post by jeastbur » Wed Jan 18, 2017 4:04 pm

I've had a Synology NAS for several years but just started using Cloud Station Backup from my Windows machine and the very first thing I tried to do was add a list of folder names that should be ignored. I really wish this was available as a setting in the client application.
Being a software dev, every time I do a build, the backup service kicks off backing up 100s or 1000s of files in my /bin and /obj folders that I don't need backed up. At this point, over 25% of my backed up files don't really need to be backed up!

markko1
I'm New!
I'm New!
Posts: 3
Joined: Sun Jan 29, 2017 10:00 pm

Re: Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Unread post by markko1 » Sun Jan 29, 2017 10:08 pm

It is indeed very surprising that Synology's own backup software is so poor! Has anyone tried something else that is free and works? I had my eye on UrBackup, that seemed very promising to be run via Docker on my NAS, but it seems that UrBackup does not play well with Docker :-(
PS: thanks for the "hack" that allows to ignore folders! I also needed to ignore some jspm and npm stuff.

icarus
I'm New!
I'm New!
Posts: 5
Joined: Mon Sep 02, 2013 10:48 pm

Re: Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Unread post by icarus » Sat Feb 04, 2017 2:54 pm

I would love to see a better ignore/filter mechanism as well.

Does anybody know whether joahs solution supports wildcards or even regurlar expressions? I have huge cache directories I want to exclude (spread over multiple folders; do not have exactly the same name, but common parts).

Unfortunately, I currently have no time to conduct experiments in order to find the answer to my question.

skywalker5446
I'm New!
I'm New!
Posts: 3
Joined: Sun Nov 13, 2016 12:40 am

Re: Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Unread post by skywalker5446 » Fri Jan 05, 2018 7:28 am

Currently on macOS, Cloud Station Backup version 4.2.5-4396, this is NOT working for me.

I even shut down the app (exit from the context menu), then edited the file mentioned above, unlinked and relinked my account in the app to see if it was copying and then caching the filters, but it's still uploading files matching the pattern: <dir>/.git/<giant files> even though I put ".git" in the Directory section.

Can anyone on macOS 10.13.2 running Backup version 4.2.5-4396 confirm this method works?

RichT
I'm New!
I'm New!
Posts: 4
Joined: Wed Oct 04, 2017 6:48 am

Re: Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Unread post by RichT » Sat Jan 06, 2018 11:35 pm

Just adding my support for this. I use my Synology to backup a bunch of remote computers for my family that I don't have easy access to. Right now one of them is constantly streaming Chrome cache files, which is a huge waste of space and bandwidth. I should be able to push a directory ignore from the Cloud Station Server just as I can push a file ignore or a file extension ignore today.

yodus
I'm New!
I'm New!
Posts: 2
Joined: Wed Jan 10, 2018 4:22 pm

Re: Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Unread post by yodus » Wed Jan 10, 2018 4:30 pm

Ignore file would be very handy. Now I have to remember to "disable" synchronization on every machine i'm using.
Ideally if it would follow git format.

dom0000
I'm New!
I'm New!
Posts: 1
Joined: Sun Feb 04, 2018 2:38 pm

Re: Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Unread post by dom0000 » Sun Feb 04, 2018 2:49 pm

Thought I’d sign up just to add a vote for server-side folder ignore patterns. I installed Cloud Station a few days ago with the hope of using it to sync work files to a laptop, but even though the initial sync is complete the CPU usage is pinned at 100% of 1 core, even 12+ hours later with no actual activity taking place. Presumably this is a result of the fact I’ve synced lots of Git repos and the client daemon can’t work efficiently with that number of small files.

I’ve tried the helpful suggestion of adding “.git” to the directory patterns in the filter-v4150 and filter files and that does seem to prevent new .git dirs from being indexed but that doesn’t help with the already-indexed folders and is not a long term solution as a client update will overwrite the changes, re-index the files and cause the problem to recur.

I wonder - since there is undocumented client-side folder filtering, maybe there is an equivalent filter config on the server? I will have a poke around but I’m not sure where to start.

razvanioan
Trainee
Trainee
Posts: 14
Joined: Mon Mar 30, 2009 8:51 am

Re: Poll: Should Cloud Station provides directory filtering and support for '.ignore' like file?

Unread post by razvanioan » Wed Feb 14, 2018 11:10 pm

Hello,

Being as well a developer who is using Drive (ex: CloudStation) to sync local dev server with the "web" folder of Synology's server, I also have issues with the quantity of the files synced.

I had found this thread and was happy about @joah solution, but then quickly read that this is overridden by client update.

Then I've started to search on my own, trying to find where keeps that app the configuration data (the one adjustable from the UI)

I've quickly found it in /data/db/ (root folder in the same paths you've mentioned, not in the .app)
There are in there 4 sqlite DB (files), one called filter.sqlite, so I was excited :)

I've opened it, and there's a nice table called filter_table
I've studied it a little bit, and understood that sess_id it's the ID of the sync task (I have 3)

The only thing is that I didn't knew what the value should be for filter_type (if is there one in the code) for folder (name, prefix or suffix)
  • -2 is for file name / extension
  • -13 is for exact path (excludable from the tree with checkboxes)
  • -15 is the file size filter (0 MB)
  • -4 is added with value . (just a dot) if Sync files and folders with the prefix "." is unchecked (checking means NOT to exclude, thus not present in this table for that specific session_id = task for web server)
I said that I've nailed it, by manually changing this table (closing the app before):
  • tried the -4 filter_type with a value node_modules
  • tried other different filter_types (not all: -1, -3)
But I don't know (yet) how, when I start the app, all my manual changes in this .sqlite file are reverted back, so I don't know if they would work
Trying to manually add "node_modules*" in the file filter exclusion list, I've seen that assigns filter_type = -1

So if somebody else finds out why or what overrides this .sqlite on app start, we might have a better chance, as DB shouldn't be overwritten on app update !

Then started to look ahead on the other folders of the app, and found a /data/session/ID/conf/ where ID is the ID number from the DataBase (session ID = task)

Here there are 3 files:
  • blacklist.filter
  • whitelist.filter
  • force-hash-check-list.filter
Opened blacklist.filter and added black_prefix = ".git", ".stversions", "bower_components", "node_modules" for Directory configuration which was blank.

In here there are also present the rules for files suffix & prefix, and custom folder path exclusion list (from the UI)

I'll just monitor if this file will be overridden upon app update or not, if yes, we're back to square one.
I still bet on the DB file, but have to find out why it's not keeping the values I manually change.

Post Reply

Return to “Feature Requests & Product Improvement Suggestions”