Thumbnail generation issue workaround

Questions about the Synology Photo Station can be placed here.

Thumbnail generation issue workaround

Postby APBilbo » Tue May 29, 2012 11:18 am

This workarround doesn't work on versions of DSM greatter than 4.0


I've a work arround for the thumbnail generation process time problem. The issue is that it takes forever to generate thumbnails (mine was spending arround a minute per picture, wich for my 25000 pictures means 18 days of thumbnail building).

With my workaround I've been able to build the thumbnails in 4 seconds per picture :).

The work arround does the following:
- Generate symbolic links to the original picture for 1280x1280 and 800x800 thumbnails (this also does save a lot of storage space while keeping photo station to work correctly).
- Build the 640x640 thumbnail using the main convert process.
- Build the 320x320 and 120x120 thumbnail using the 640x640 thumbnail as source instead of the original picture (this saves a lot of processing time and I/O).
- For all the thumbnails generated (the 3 smaller sizes) It uses -sample parameter instead of the -size one as sugested by benob http://forum.synology.com/enu/viewtopic.php?f=17&t=24083#p102578.

To install the patch do the following:
- READ THIS INSTRUCTIONS CAREFULLY
- IF YOU DON'T KNOW WHAT YOU'RE DOING, CONSIDER CLOSING THIS WINDOW
- Understand this code is provided AS IS and without any waranty. Use it at your own risk. Allways make a backup of the files before changing them.
- open an administration console (i.e. using Putty, note the user name is "root" and not "admin". The password is the same one).
- create a file somewhere called convert-patch.pl (i.e. /usr/syno/bin/convert-patch.pl).
- Paste the following code inside (i.e. execute "vi /usr/syno/bin/convert-patch.pl", then hit 'a' key, then seccond click the putty screen while having the code in the clipboard. When code is pasted hit "ESC" and type ":wq")

Code: Select all
#!/usr/bin/perl -w
@new_args = (); # Where we are going to copy the arguments
$doLnk = 0;     # Flag meaning that we can generate a symbolic link to the original file
$chgSource = 0; # Flag meaning that we have to change the source to the medium sized picture
$copyPath = ""; # The path where we are generation the thumbnails. It comes in the  registry:temporary-path parameter

for $arg(@ARGV) {
  #Detect the folder of the thumbnails
  if($arg =~ m/^registry.temporary.path./) {
    $copyPath = $arg;
    $copyPath =~ s/^registry.temporary.path.//;
  }
  elsif($arg eq "800x800"){ $doLnk = 1; }
  elsif($arg eq "1280x1280"){ $doLnk = 1; }                                                                       
  elsif($arg eq "120x120"){ $chgSource = 1; }                                                               
  elsif($arg eq "320x320"){ $chgSource = 2; }
  elsif($arg eq "-size") { $arg = "-sample"; }
 
  push @new_args, $arg;
}

#We change the source to the medium sized picture to save reading time.
if($chgSource){
  $new_args[$#new_args-1] = $copyPath . "/SYNOPHOTO:THUMB_B.jpg";
}

#We make a symbolic link when ever it applies
if($doLnk){

  exec "ln", ("-sf", $ARGV[$#ARGV-1], $ARGV[$#ARGV]);
 
}else{
  # Nothing more to do... lets get the thumbnail!
  exec "/lib/hddapp/usr/syno/bin/convert", @new_args;

}


- Now run the following commands (change the paths to point to the fiel you have created if it applyes).

Code: Select all
chmod +x /usr/syno/bin/convert-patch.pl
ln -sf /usr/syno/bin/convert-patch.pl /usr/syno/bin/convert


- Now edit the files /usr/syno/etc.defaults/thumb.conf and /usr/syno/etc.defaults/thumb_high.conf exchanging the 120 and 640 thumbnail sizes and filenames (this step is needed for the 640x640 thumbnail to be generated before the 120x120 and 320x320).

Code: Select all
[thumb 120]
size=640
quality=80
filename=SYNOPHOTO:THUMB_B.jpg

[thumb 320]
size=320
quality=90
filename=SYNOPHOTO:THUMB_M.jpg

[thumb 640]
size=120
quality=90
filename=SYNOPHOTO:THUMB_S.jpg

[thumb 800]
size=800
quality=90
filename=SYNOPHOTO:THUMB_L.jpg

[thumb 1280]
size=800
quality=90
filename=SYNOPHOTO:THUMB_XL.jpg


- Now reload the service:

Code: Select all
/usr/syno/etc/rc.d/S77synomkthumbd.sh restart


- Now your thumbnails should be generating lightning fast now!! :)


To uninstall the patch just:

- Restore the backups you should have made of the thumb.conf and thumb-high.conf.
- Set the convert-patch.pl file contents to this:
Code: Select all
#!/usr/bin/perl -w
exec "/lib/hddapp/usr/syno/bin/convert", @ARGV;


Hope this helps :).
Last edited by APBilbo on Wed Nov 07, 2012 11:49 am, edited 4 times in total.
APBilbo
Trainee
Trainee
 
Posts: 12
Joined: Sun May 27, 2012 9:53 pm

Re: Thumbnail generation issue workarround

Postby myCloud » Tue May 29, 2012 11:55 am

Very interesting. Thanks for posting your process!
DS 1512+ w/3GB, 5 x 3TB Seagate ST3000DM001 8.2TB RAID 6, half files/half Time Machine.
Icy Dock MB559U3S-1SB enclosure w/4TB Hitachi UltraStar via USB 3 for files backup
UVERSE to AirPort Extreme + 2 AirPort Express w/speakers. TRENDnet TV-IP312WN camera
CyberPower CP1500PFCLCD Sine Wave UPS
DSM 4.1-2661 w/SSH + SFTP, VPN Server, Syslog Server, Media Server, Mail Server, Mail Station,
Audio Station, Surveillance Station, Photo Station, Web Station - DS Apps on iPad & iPod Touch.
User avatar
myCloud
Skilled
Skilled
 
Posts: 648
Joined: Fri Mar 23, 2012 11:28 am

Re: Thumbnail generation issue workaround

Postby pollewops » Tue May 29, 2012 2:36 pm

I hope Synology will implement such a fix asap.....

I tried to implement the described fix myself but get stucked to create the pl file..access denied although i am logged on using putty and admin account. I need to wait for a fix from Synology.

Until now I stopped the Photo Station service since it put my cpu to 100% and does 1 photo in 10 minutes or so...so very very slow.....
pollewops
Trainee
Trainee
 
Posts: 14
Joined: Sat May 26, 2012 10:40 am
Location: Netherlands, Almere

Re: Thumbnail generation issue workaround

Postby APBilbo » Tue May 29, 2012 10:33 pm

pollewops wrote:I hope Synology will implement such a fix asap.....

I tried to implement the described fix myself but get stucked to create the pl file..access denied although i am logged on using putty and admin account. I need to wait for a fix from Synology.

Until now I stopped the Photo Station service since it put my cpu to 100% and does 1 photo in 10 minutes or so...so very very slow.....


Hi,

you have to login with the "root" user and not admin. The password is the same one. Give it a try and let me know how it works.

BTW if you feel more conftable you can put the pl file inside the /volume1/photo/convert-patch.pl and point the symbolic link there. That way you can edit it using a shared folder wich is eassiest than 'vi' :-).
APBilbo
Trainee
Trainee
 
Posts: 12
Joined: Sun May 27, 2012 9:53 pm

Re: Thumbnail generation issue workaround

Postby Goner » Wed May 30, 2012 10:31 am

Or use the Photo Uploader in the Synology Assistant, then your PC does all the thumbnailing ...

NAS : DS212j with 2 ST2000DL003 in SHR / DSM 4.2-3211
LAN : Fritz!Box 7170, 4 Devolo, 1 Icidu 200Mbps AV homeplugs, 2 5-port switches
HW : Conceptronic CHD3NET, ACRyan Playon!HD, Eminent EM7075dts, Wii U, PS2
User avatar
Goner
Enlightened
Enlightened
 
Posts: 443
Joined: Tue Mar 06, 2012 2:27 pm
Location: Rotterdam, Netherlands

Re: Thumbnail generation issue workaround

Postby pollewops » Wed May 30, 2012 4:15 pm

I was able to modify now and thumbnails are generated.
I have 35.000 photo's to be scanned and now after 3 hours I am halfway....

Tnx !

find . -name *:THUMB_M* | wc -l; sleep 250; find . -name *:THUMB_M* | wc -l
Results in about 100 photo's a minute :-)
pollewops
Trainee
Trainee
 
Posts: 14
Joined: Sat May 26, 2012 10:40 am
Location: Netherlands, Almere

Re: Thumbnail generation issue workaround

Postby APBilbo » Wed May 30, 2012 6:23 pm

pollewops wrote:I was able to modify now and thumbnails are generated.
I have 35.000 photo's to be scanned and now after 3 hours I am halfway....

Tnx !

find . -name *:THUMB_M* | wc -l; sleep 250; find . -name *:THUMB_M* | wc -l
Results in about 100 photo's a minute :-)


Great! I'm glad to know it worked for you :).

Goner wrote:Or use the Photo Uploader in the Synology Assistant, then your PC does all the thumbnailing ...


I think my method is even faster and it doesn't imply two machines working :). Also I prefer copying data directly from an USB device than sending it though network (mine is wifi G). But you're right, that might be a good alternative :).
APBilbo
Trainee
Trainee
 
Posts: 12
Joined: Sun May 27, 2012 9:53 pm

Re: Thumbnail generation issue workaround

Postby rcblackwell » Wed May 30, 2012 8:12 pm

Goner wrote:Or use the Photo Uploader in the Synology Assistant, then your PC does all the thumbnailing ...


When I first set up my DS I found that adding a few thousand photo's using the Photo Uploader was a slow and arduous process. Now whenever I have a large number of images to upload I copy the images to the correct directories then, using the above noted process, re-index the media files. There's currently over five thousand pictures on the DS. When I ran the script yesterday it took very little time to create the thumbs.
Robert Blackwell
Pickering, Ontario
User avatar
rcblackwell
Student
Student
 
Posts: 79
Joined: Sun May 16, 2010 12:20 pm
Location: Pickering, Ontario

Re: Thumbnail generation issue workaround

Postby pollewops » Wed May 30, 2012 9:27 pm

I need to come back to my positive answer. The thumbs are generated fast, but when i start ds photo+ on mu ipad no thumbs are shown except the ones generated earlierr using the slo method...

Any ideas why this happens ?
pollewops
Trainee
Trainee
 
Posts: 14
Joined: Sat May 26, 2012 10:40 am
Location: Netherlands, Almere

Re: Thumbnail generation issue workaround

Postby APBilbo » Wed May 30, 2012 10:24 pm

pollewops wrote:I need to come back to my positive answer. The thumbs are generated fast, but when i start ds photo+ on mu ipad no thumbs are shown except the ones generated earlierr using the slo method...

Any ideas why this happens ?


Mmmm.... ¿Could you open the online photo station from your browser? ¿Do you see the thumbnails there? Also please post your thumb.conf and thumb_high.conf configuration file contents.
APBilbo
Trainee
Trainee
 
Posts: 12
Joined: Sun May 27, 2012 9:53 pm

Re: Thumbnail generation issue workaround

Postby pollewops » Thu May 31, 2012 10:33 am

Yes I can open the photo station from a browser and then the thumbs are also not there..so something i did wrong probably..but what ?

Is there a way to re-trigger the creating of the thumbs ?

Here by my config files:

/usr/syno/etc.defaults/thumb.conf
Code: Select all
[thumb 120]
size=640
quality=80
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_B.jpg

[thumb 320]
size=320
quality=90
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_M.jpg

[thumb 640]
size=640
quality=80
filename=SYNOPHOTO:THUMB_S.jpg

[thumb 800]
size=800
quality=90
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_L.jpg

[thumb 1280]
size=1280
quality=90
filename=SYNOPHOTO:THUMB_XL.jpg


/usr/syno/etc.defaults/thumb_high.conf
Code: Select all
[thumb 120]
Size=640
quality=90
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_B.jpg

[thumb 320]
size=320
quality=90
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_M.jpg

[thumb 640]
size=120
quality=80
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_S.jpg

[thumb 800]
size=800
quality=90
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_L.jpg

[thumb 1280]
size=1280
quality=90
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_XL.jpg
pollewops
Trainee
Trainee
 
Posts: 14
Joined: Sat May 26, 2012 10:40 am
Location: Netherlands, Almere

Re: Thumbnail generation issue workaround

Postby APBilbo » Thu May 31, 2012 11:16 am

pollewops wrote:Yes I can open the photo station from a browser and then the thumbs are also not there..so something i did wrong probably..but what ?

Is there a way to re-trigger the creating of the thumbs ?

Here by my config files:

/usr/syno/etc.defaults/thumb.conf
Code: Select all
[thumb 120]
size=640
quality=80
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_B.jpg

[thumb 320]
size=320
quality=90
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_M.jpg

[thumb 640]
size=640
quality=80
filename=SYNOPHOTO:THUMB_S.jpg

[thumb 800]
size=800
quality=90
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_L.jpg

[thumb 1280]
size=1280
quality=90
filename=SYNOPHOTO:THUMB_XL.jpg


/usr/syno/etc.defaults/thumb_high.conf
Code: Select all
[thumb 120]
Size=640
quality=90
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_B.jpg

[thumb 320]
size=320
quality=90
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_M.jpg

[thumb 640]
size=120
quality=80
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_S.jpg

[thumb 800]
size=800
quality=90
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_L.jpg

[thumb 1280]
size=1280
quality=90
;unsharp=0.5x0.5+1.25+0.0
filename=SYNOPHOTO:THUMB_XL.jpg


The conf files seem to be ok... ¿Could you check if a folder called @ear (or something similar) exist inside some of your album folders? So for example if you have a picture at lets say:

/volume1/photo/album/myFoto.jpg

Then search for this folder:

/volume1/photo/album/@ear/myFoto.jpg/

Look at the contents and tell me if you have any files ending in "FAIL". If so, my guess is that you may have not run the reload process at the end of the installation (/usr/syno/etc/rc.d/S77synomkthumbd.sh restart) so the order what still 120 320 640 instead of 640 320 120, and the smaller thumbs may have not been generated. I think you can trigger the thumbnail ing just by:
- running the reindex process
- adding a new picture somewhere

If this doesn't work you can always move the albums from one place to another :) (like moving them inside a folder), this will trigger the generation. Please give it a try and let me know.
APBilbo
Trainee
Trainee
 
Posts: 12
Joined: Sun May 27, 2012 9:53 pm

Re: Thumbnail generation issue workaround

Postby pollewops » Thu May 31, 2012 11:41 am

There is no @ear folder.....there is however a @eaDir folder.
No files with FAIL at the end....

Image


I think I put everything back to default and will upload all 35.000 photo's again using the assistant...
pollewops
Trainee
Trainee
 
Posts: 14
Joined: Sat May 26, 2012 10:40 am
Location: Netherlands, Almere

Re: Thumbnail generation issue workaround

Postby APBilbo » Thu May 31, 2012 12:40 pm

pollewops wrote:There is no @ear folder.....there is however a @eaDir folder.
No files with FAIL at the end....

Image


I think I put everything back to default and will upload all 35.000 photo's again using the assistant...



Those in blue are folder them selfs

From that same place Please do

"cd DSC00184.jpg"

and then "ls"

and check if there are fail files there....
APBilbo
Trainee
Trainee
 
Posts: 12
Joined: Sun May 27, 2012 9:53 pm

Re: Thumbnail generation issue workaround

Postby pollewops » Thu May 31, 2012 12:49 pm

Ai...

PolNAS> ls
SYNOPHOTO:THUMB_B.fail SYNOPHOTO:THUMB_M.fail SYNOPHOTO:THUMB_XL.fail
SYNOPHOTO:THUMB_L.fail SYNOPHOTO:THUMB_S.fail
PolNAS>

And i did the /usr/syno/etc/rc.d/S77synomkthumbd.sh restart command.
I did it now again and still that problem....
Do I need to delete all those *.fail files ?
pollewops
Trainee
Trainee
 
Posts: 14
Joined: Sat May 26, 2012 10:40 am
Location: Netherlands, Almere

Next

Return to Photo Station

Who is online

Users browsing this forum: No registered users and 3 guests