Periscope on Synology: automatic subtitle download

[sverzijl]

I recently heard about the python tool Periscope, which is able to scan various Subtitle sites (like TheSubDB, Bierdopje, Addic7ed) and download subs in SRT format of the languages you prefer (if they are available of course). Documentation on this tool is very limited, but with a little trial and error I managed to get it working on Synology (411+ in this case, but I can't see any reason why it wouldn't work on other Synology boxes).
Since I know quite some people are looking for something like this, I decided to write a quick HOW-TO.

Please NOTE: Since Periscope does its searches based on the exact filename of the series (e.g House.S07E08.720p.HDTV.X264-DIMENSION.mkv) this solution will not work if you are using the post-processing/renaming features of SabNZBd or Sickbeard. So if you want this to work, you will have to disable renaming for now (until this tool is fully integrated into Sickbeard).

The steps:

In order to download Periscope onto Synology it's easiest to do so via SVN. In order to be able to install SVN, you first have to install ipkg if not already done so.

- Log on to your synology box (as root)
If you don't know how to do this, see here

- Install ipkg
See here

- Install svn
Once ipkg is installed, you can easily install SVN by running :

Code: Select all

ipkg install svn

- Download periscope
cd to the directory you want to install periscope in. This can be anywhere, but for now I will assume /volume1/@appstore (The SVN checkout command will create a subdirectory below this directory)

Code: Select all

cd /volume1/@appstore

now download periscope by running :

Code: Select all

svn checkout http://periscope.googlecode.com/svn/trunk/ periscope

- Install python 2.6

Code: Select all

ipkg install python26

- Install BeautifulSoup
BeautifulSoup is an HTML/XML parser that is used by periscope. To download it :

Code: Select all

cd /volume1/@appstore/periscope
wget http://launchpad.net/beautifulsoup/3.2/3.2.0/+download/BeautifulSoup-3.2.0.tar.gz

unzip/untar it :

Code: Select all

gunzip BeautifulSoup-3.2.0.tar.gz
tar -xvf BeautifulSoup-3.2.0.tar

remove the tar file as not needed anymore:

Code: Select all

rm BeautifulSoup-3.2.0.tar

Now copy BeautifulSoup.py to the plugins directory of periscope

Code: Select all

cp BeautifulSoup-3.2.0/BeautifulSoup.py periscope/plugins/

- Create simple python download command file
in /volume1/@appstore/periscope/periscope, create a file called downloadSub.py with following contents :

Code: Select all

#!/opt/bin/python2.6
# -*- coding: utf-8 -*-

import periscope
import sys
import logging
logging.basicConfig(level=logging.DEBUG)


subdl = periscope.Periscope()
filepath = sys.argv[1]
subtitle = subdl.downloadSubtitle(filepath, ['nl','en']) # Dutch, and if not available try English
if subtitle :
    print "Found a sub from %s in language %s, downloaded to %s" % ( subtitle['plugin'], subtitle['lang'], subtitle['subtitlepath'])

Either do this using vi if you're familiar with it, or use notepad, place it on a share and then copy it to the right location:

Code: Select all

cp /volume1/public/downloadSub.py /volume1/@appstore/periscope/periscope/

also make sure downloadSub.py is executable :

Code: Select all

chmod +x downloadSub.py

Note : the ['nl','en'] can be changed to the language(s) you want. In this case it will first try to obtain a Dutch subtitle. If not available it will try to download english subtitle.

- Test the download command by providing one of your existing series :

Code: Select all

 ./downloadSub.py /volume1/video/Series/House/House.S07E08.720p.HDTV.X264-DIMENSION.mkv

As debugging is on, you should see all actions that are being performed to get the (dutch or english) subs. If succesfull, last line should be something like:

Found a sub from <plugins.BierDopje.BierDopje object at 0xf6f912ac> in language nl, downloaded to /volume1/video/Series/House/House.S07E08.720p.HDTV.X264-DIMENSION.srt

- Scan for subs for all videofiles
Obviously you don't want to manually run the downloadSub.py for every new download. You can probably add python code to scan all directories, but since I didn't want to spend too much time on it I just used something I'm familiar with.

Code: Select all

cd /volume1/@appstore/periscope/periscope
find /volume1/video/Series/ \( -name *.avi -o -name *.mkv \) -mtime -7 -type f -exec ./downloadSub.py {} \;

(change /video/Series to the path you have stored your series)

This will scan the Series folder for all MKV and AVI files that were added in the last 7 days. You could also run the command only for series that do not have a SRT downloaded yet, but since I want the Dutch subs if they are available this method wouldn't work for me (i.e. when English sub already downloaded it would not try to get the Dutch one anymore).

Note: for 1st time run, you can just leave out the '-mtime -7', so it will scan all files instead of just the ones added in last 7 days.

Now to finish things off you can put these 2 lines in a scriptfile, say /volume1/@appstore/periscope/periscope/scanSeries.sh and put it in cron to run it every day.

To add the script to the cron scheduler, just add this line to /etc/crontab :

Code: Select all

#minute hour    mday    month   wday    who     command
0       19       *       *       *       root    /volume1/@appstore/periscope/periscope/scanSeries.sh >>/volume1/@appstore/periscope/periscope/scanSeries.log 2>&1

(command goes on 1 line)

make sure scanSeries.sh is executable :

Code: Select all

chmod +x scanSeries.sh

now restart cron daemon:

Code: Select all

/usr/syno/etc.defaults/rc.d/S04crond.sh stop
/usr/syno/etc.defaults/rc.d/S04crond.sh start

Every day at 19:00 the subs will now be downloaded.



Have fun:)

[Edit]
If you want to exclude some of the subtitle sources (i.e. because you know it will never find anything, or quality is always bad), you can do so by editing periscope/plugins/__init__.py and comment out the ones you don't want :
i.e. I only want to check Bierdopje and Addic7ed :

Code: Select all

#from OpenSubtitles import OpenSubtitles
#from SubtitleSource import SubtitleSource
#from SubScene import SubScene
#from Subtitulos import Subtitulos
from Addic7ed import Addic7ed
#from Podnapisi import Podnapisi
#from TheSubDB import TheSubDB
from BierDopje import BierDopje

#from TvSubtitles import TvSubtitles
#from Podnapisi2 import Podnapisi

[edit 14-01-2011]
If you are using SickBeard and you want to include periscope into the sabToSickBeard.py post-processing, you can do the following :
(Note: this will probably not be so usefull when downloading the most recent episodes of a series since subs will not be available yet for most of them until 1-2 days after airdate)

- Create a file called 'sabToPeriscopeToSickBeard.py' in the same directory as sabToSickBeard.py with following contents :

Code: Select all

#!/opt/bin/python2.6

import sys
sys.path.append("/volume1/@appstore/periscope/periscope")
import autoProcessTV
import periscope
import os
import glob
import logging
logging.basicConfig(level=logging.ERROR)

if len(sys.argv) < 2:
    print "No folder supplied - is this being called from SABnzbd?"
    sys.exit()


DOWNLOADPATH=sys.argv[1]

filelist=glob.glob(DOWNLOADPATH+"/*.avi")+glob.glob(DOWNLOADPATH+"/*.mkv")
for file in filelist:
  print "Trying to find subs for %s" % (file)
  subdl = periscope.Periscope()
  subtitle = subdl.downloadSubtitle(file, ['nl','en']) # Dutch, and if not available try English
  if subtitle :
    print "----------------------------------------------------------------------------------------------------------------"
    print "Found a sub in language %s, downloaded to %s" % ( subtitle['lang'], subtitle['subtitlepath'])
    print "----------------------------------------------------------------------------------------------------------------"


print "Now handing over postprocessing to SickBeard.."
if len(sys.argv) >= 3:
    autoProcessTV.processEpisode(sys.argv[1], sys.argv[2])
else:
    autoProcessTV.processEpisode(sys.argv[1])


Note: update the path to periscope in this script to the location you have installed it to.

- Make script executable :

Code: Select all

chmod +x sabToPeriscopeToSickBeard.py


- Make sure you have enabled 'Move Associated Files' in SickBeard (Config -> Episode Downloads)
- in SAB, change the script of the user defined category that is used by SickBeard (usually 'TV') from 'sabToSickBeard.py' to 'SabToPeriscopeToSickBeard.py' via the dropdown menu

That's it, from now on for any new download periscope will try to get the NL/EN sub.

[mrwolf]

Thanks for the tutorial, it works fine! Except for the cron-part...

When executing cron the log file says:

Code: Select all

/volumeUSB1/usbshare/periscope/periscope/searchSub.sh: cd: line 1: can't cd to /volumeUSB1/usbshare/periscope/periscope
find: ./downloadSub.py: No such file or directory
find: ./downloadSub.py: No such file or directory
find: ./downloadSub.py: No such file or directory
find: ./downloadSub.py: No such file or directory

searchSub.sh looks like:

Code: Select all

cd /volumeUSB1/usbshare/periscope/periscope
find /volume1/series/ \( -name *.avi -o -name *.mkv \) -mtime -7 -type f -exec ./downloadSub.py {} \;

When manually executing both lines in Putty, they work fine. So there seems to be no error in the text or path, and periscope is definitely installed on /volumeUSB1/usbshare/periscope/periscope, I triple-checked it.

What can I do to make this working via cron?

[sverzijl]

Thanks for the tutorial, it works fine! Except for the cron-part...
When manually executing both lines in Putty, they work fine. So there seems to be no error in the text or path, and periscope is definitely installed on /volumeUSB1/usbshare/periscope/periscope, I triple-checked it.

What can I do to make this working via cron?

Hm, I can't think of any reason why this wouldn't work from cron (assuming you have set the run-user to root as in my example).
What happens if you (just as a test) copy the periscope directory to a directory that is not on your USB drive but on the internal disk (/volume1/...) and schedule cron to run the job from there ?

Or put some additional cd commands in Searchsub.sh to see which directory 'cron' still does have access to :
e.g. :
cd /volumeUSB1
cd usbshare
cd periscope
cd periscope

[mrwolf]

It seems that indeed the USBshare is the reason. I've copied the dir to /volume1/@appstore, and it works. Thanks!

[jmurre]

This is perfect. I use this in combination with SickBeard, which automatically downloads the series, when they are available: http://forum.synology.com/enu/viewtopic.php?f=45&t=27170. This is an excellent addition to that.

You know by any chance if this works for movies as well or is there anything else available for that?

[sverzijl]

This is perfect. I use this in combination with SickBeard, which automatically downloads the series, when they are available: http://forum.synology.com/enu/viewtopic.php?f=45&t=27170. This is an excellent addition to that.

You know by any chance if this works for movies as well or is there anything else available for that?

Have you tried ?

You should be able to use it for movies as well. Most of the supported subtitle-sites are 'TV-series' based, but at least http://www.opensubtitles.org is supported which does have quite a collection.
Or you could start using Couchpotato, which is an excellent Movie finder (also works together with SABnzbd) that is now getting subtitle support as well.

[abibouba]

Hello,

Thanks a lot for your tutorial. This is a great added feature for our NAS.

Everything works perfectly except when I do :

find /volume1/video/Series/ \( -name *.avi -o -name *.mkv \) -mtime -7 -type f -exec ./downloadSub.py {} \;

It only finds the subtitle for one file while there are 20 .avi files in the directory. Do you have any idea ?

[Brickman]

Hello,

Thanks a lot for your tutorial. This is a great added feature for our NAS.

Everything works perfectly except when I do :

find /volume1/video/Series/ \( -name *.avi -o -name *.mkv \) -mtime -7 -type f -exec ./downloadSub.py {} \;

It only finds the subtitle for one file while there are 20 .avi files in the directory. Do you have any idea ?

With the following code you will find all the files:

Code: Select all

find /volume1/video/Series/ \( -name *.avi -o -name *.mkv \) -type f -exec ./downloadSub.py {} \;

The -mtime -7 argument returns only shows that are modified within the past 7 days.

[abibouba]

I've found the solution. It is because mtime is not necessarily the time when the file was copied on the hard drive.

Also a little improvement for people who like to have subtitle in 2 or more languages with the extension .en.srt for example.

Use the following DownloadSub.py and change it according to your language :

#!/opt/bin/python2.6
# -*- coding: utf-8 -*-

import periscope
import sys
import logging
import os
import string
logging.basicConfig(level=logging.DEBUG)


subdl = periscope.Periscope()
filepath = sys.argv[1]
print "\nSearch for English subtitle:"
print "=============================="
subtitle1 = subdl.downloadSubtitle(filepath, ['en']) # English
if subtitle1 :
print "Found a sub from %s in language %s, downloaded to %s" % ( subtitle1['plugin'], subtitle1['lang'], subtitle1['subtitlepath'])
os.rename(subtitle1['subtitlepath'], string.join(string.split(subtitle1['subtitlepath'], ".srt"), ".en.srt"))

print "\nSearch for Dutch subtitle:"
print "=============================="
subtitle2 = subdl.downloadSubtitle(filepath, ['nl']) # Dutch
if subtitle2 :
print "Found a sub from %s in language %s, downloaded to %s" % ( subtitle2['plugin'], subtitle2['lang'], subtitle2['subtitlepath'])
os.rename(subtitle2['subtitlepath'], string.join(string.split(subtitle2['subtitlepath'], ".srt"), ".nl.srt"))

[abibouba]

Sorry I have another question

You could also run the command only for series that do not have a SRT downloaded yet

How can we do that ?

[sverzijl]

Sorry I have another question

You could also run the command only for series that do not have a SRT downloaded yet

How can we do that ?

You can use this little script for that :

Code: Select all

find /volume1/video/Series/ \( -name *.avi -o -name *.mkv \) -type f|while read fname; do
srtname=${fname%.*}.srt
if [ ! -f "$srtname" ]; then
  echo "SRT not available yet, trying to find now ($fname)"
  ./downloadSub.py "$fname"
fi
done


[jmurre]

I've found the solution. It is because mtime is not necessarily the time when the file was copied on the hard drive.

Also a little improvement for people who like to have subtitle in 2 or more languages with the extension .en.srt for example.

Use the following DownloadSub.py and change it according to your language :

#!/opt/bin/python2.6
# -*- coding: utf-8 -*-

import periscope
import sys
import logging
import os
import string
logging.basicConfig(level=logging.DEBUG)


subdl = periscope.Periscope()
filepath = sys.argv[1]
print "\nSearch for English subtitle:"
print "=============================="
subtitle1 = subdl.downloadSubtitle(filepath, ['en']) # English
if subtitle1 :
print "Found a sub from %s in language %s, downloaded to %s" % ( subtitle1['plugin'], subtitle1['lang'], subtitle1['subtitlepath'])
os.rename(subtitle1['subtitlepath'], string.join(string.split(subtitle1['subtitlepath'], ".srt"), ".en.srt"))

print "\nSearch for Dutch subtitle:"
print "=============================="
subtitle2 = subdl.downloadSubtitle(filepath, ['nl']) # Dutch
if subtitle2 :
print "Found a sub from %s in language %s, downloaded to %s" % ( subtitle2['plugin'], subtitle2['lang'], subtitle2['subtitlepath'])
os.rename(subtitle2['subtitlepath'], string.join(string.split(subtitle2['subtitlepath'], ".srt"), ".nl.srt"))

If I try to run this I am getting the following error:

File "./DownSub.py", line 21
print "Found a sub from %s in language %s, downloaded to %s" % ( subtitle1['plugin'],
subtitle1['lang'], subtitle1['subtitlepath'])
^
IndentationError: expected an indented block

The print line is actually 1 line, but else I could not place the ^ correctly. Any idea why this happens and how I can solve this?

[sverzijl]

Python code blocks have no explicit begin or end, and no curly braces to mark where the code blocks starts and stops, but use indentation of the code.
So in this case the print command below the if-statement has to be idented (add say 4 spaces at beginning of the line. Every next command you want to be part of the if-block will have to be idented the same amount of spaces)

e.g. :

Code: Select all

if subtitle2 :
    print "Found"
    print "blah"
if whateversomething :
    print "yep"


[jmurre]

Python code blocks have no explicit begin or end, and no curly braces to mark where the code blocks starts and stops, but use indentation of the code.
So in this case the print command below the if-statement has to be idented (add say 4 spaces at beginning of the line. Every next command you want to be part of the if-block will have to be idented the same amount of spaces)

e.g. :

Code: Select all

if subtitle2 :
    print "Found"
    print "blah"
if whateversomething :
    print "yep"


Thanks, that's it. It's running now.

[jmurre]

Ok, I have got it running, but I am getting no hits on any of my TV series. All I get is:

Code: Select all

Search for English subtitle:
==============================
INFO:root:Searching subtitles for /volume1/tv/Greys Anatomy/Season 7/Greys Anatomy S07E03.mkv with langs ['en']
INFO:root:Searching on OpenSubtitles
INFO:root:Searching on Addic7ed
INFO:root:Searching on BierDopje
INFO:root:Searching on TvSubtitles
DEBUG:root:greys anatomy s07e03
INFO:root:BierDopje writing 0 items to queue
INFO:root:Addic7ed writing 0 items to queue
INFO:root:TvSubtitles writing 0 items to queue
ERROR:root:Open subtitles could not be contacted for login
INFO:root:OpenSubtitles writing 0 items to queue
SRT not available yet, trying to find now (/volume1/tv/Greys Anatomy/Season 5/Greys Anatomy S05E03.mkv)


The format I use for my TV shows is 'Show's name SxxExx.ext'. Is my format wrong or is there another issue here?