SynoCatch - Synology RSS Broadcatching Script

[GNaschenweng]

It would be great if someone could incorporate the TV-rename script as the one listed here: http://www.robmeerman.co.uk/coding/file_renamer

[John_P]

I broke the script when I "cleaned" it up!

Manual runs worked but crontab runs failed.

First problem is the tr command doesn't work reliably fine on manual runs, fails from crontab. Gone back to the sed way
I grouped two of the sed commands on to one line and that caused a problem.

It's been running for a little longer this time and seem OK with the feeds I'm using.

Code: Select all

#!/opt/bin/bash
# SynoFeedCatch - Torrent downloading by RSS based on a script by h0me5k1n)
#
# Script based on http://linc.homeunix.org:8080/scripts/bashpodder
# discussion on http://forum.synology.com/enu/viewtopic.php?f=38&t=18039
#need bash and xsltproc installed by ipkg
#Install toolchains via ipkg install 'ipkg install optware-devel'
# iconv see http://www.gnu.org/software/libiconv/
#download  ftp://ftp.gnu.org/gnu/libiconv/libiconv-1.13.tar.gz
# tar xvfz libiconv-1.13.tar.gz
# ./configure --prefix=/usr/local / make / make install
###CONFIGURATION PARAMETERS
## directory to put the downloaded torrents,with trailing slash
torrentdir="/volume1/torrents/rss/"
# User Vars
# Feed.conf contain lines with 6 strings seperated by an 'title regex'@'RSS feed url'#'name for cookie ie sitename'@'login name'@'userpassword'@login url / script
# lines can be comented out by placing a # as the first character
PRIVATEFEED="feed.conf"
#   Path to store cookies in
COOKIEPATH="/volume1/torrents/rss/cookies/"
rm -f DEBUG #seperate debug file I used for xsltproc
# Debug Log (set to /dev/null to turn off)
#DEBUG="/dev/null"
DEBUG=$torrentdir"debug.log"
echo "Debug log set to $DEBUG"
# Make script crontab friendly:
cd $(dirname $0)
echo -e "\nExecuting $0 on $(date)" >> $DEBUG
# feed dump reset
rm -f rssdata
# Read the feed.conf file and wget any url not already in the catch.log file:
#    subscription is the regex@url of rss feed from the bp.conf line by line.
while read torrent_private_feed
do
   if test $(echo "$torrent_private_feed" | sed 's/\(^#\).*$/\1/' ) = "#"
      then
# line is hashed out skip processing
      continue
   fi
#<title regex>@<rss feed urL>#<name for cookie>@<user>@<password>@<login url>
   subscription=$(echo "$torrent_private_feed" | sed 's/\(^[^@]\+@[^@#]\+\)#\+\([^#@]\+@[^@]\+@[^@]\+@[^@]\+\)/\1/' )
   private_feed=$(echo "$torrent_private_feed" | sed 's/\(^[^@]\+@[^@#]\+\)#\+\([^#@]\+@[^@]\+@[^@]\+@[^@]\+\)/\2/' )
# can't get the command to work right so will have to try a work around should look at this again!
# in the conf file 'private_feed' data might be empty I want to use this as a flag not to process
# it as a private torrent site needing a login but the regex fails?? and returns the full string when empy???
   if [ "$subscription" = "$private_feed" ]
      then
      private_feed=
   fi
   expression=$(echo "$subscription" | sed 's/\([^@]*\)@.*/\1/' )
#   and check if the subscription contain a site that needs a login??
   if ! [ -z "$private_feed" ]
      then
#   Private site requiring login found
#   Extract the torrent site name data from the line using sed
      private_site=$(echo "$private_feed" | sed 's/^\([^@]*\)@\([^@]*\)@\([^@]*\)@\([^@]*\)$/\1/' )
      private_user=$(echo "$private_feed" | sed 's/^\([^@]*\)@\([^@]*\)@\([^@]*\)@\([^@]*\)$/\2/' )
      private_password=$(echo "$private_feed" | sed 's/^\([^@]*\)@\([^@]*\)@\([^@]*\)@\([^@]*\)$/\3/' )
      private_login=$(echo "$private_feed" | sed 's/^\([^@]*\)@\([^@]*\)@\([^@]*\)@\([^@]*\)$/\4/' )
#   Do we have a cookie?
      cookie_file="$COOKIEPATH$private_site.txt"
      if ! [ -f $cookie_file ]
         then
#   Check for directory / file path and create as necessary
         if ! [ -d $COOKIEPATH ]
            then
            mkdir $COOKIEPATH 2>> $DEBUG
         fi
#   Get the cookie from site
# Demonoid Exception posted data username is nickname!!
# wget --save-cookies=$cookie_file --keep-session-cookies --post-data="nickname=$private_user&password=$private_password" "http://www.demonoid.com/account_handler.php"
         if echo "$subscription" | egrep -i "demonoid" &> /dev/null #gotyou for demonoid
            then
            post_data="nickname=$private_user&password=$private_password"
#            echo demonoid
         else
            post_data="username=$private_user&password=$private_password"
#            echo not demonoid
         fi
         wget --save-cookies=$cookie_file --keep-session-cookies --post-data="$post_data" $private_login
      fi
         xmldata=$(wget --load-cookies="$cookie_file" $(echo "$subscription" | sed 's/[^@]*@\(.*\)/\1/') -q -O -)
   else
         xmldata=$(wget $(echo "$subscription" | sed 's/[^@]*@\(.*\)/\1/') -q -O -)
   fi
   if  [ -z "$xmldata" ]
      then
      echo -e "$LINENO Failed to get data from "$(echo "$subscription" | sed 's/[^@]*@\(.*\)/\1/') >> $DEBUG
   else
# 1st line contains version and encoding delete the rest
      version=$(echo "$xmldata" | sed '2,$d') #>> $DEBUG
# extract encoding <?xml version="1.0" encoding="utf-8"?>
      encoding=$(echo "$version" | sed 's/^.*<?xml version="[^"]*".*encoding="\([^"]*\)".*?>.*$/\1/' )
      version=$(echo "$version" | sed 's/^.*<?xml version="\([^"]*\)".*$/\1/' )
# convert to upper case
      encoding=$(echo "$encoding" | sed -e 'y/qwertyuiopasdfghjklzxcvbnm/QWERTYUIOPASDFGHJKLZXCVBNM/' )
#      encoding=$(echo "$encoding" | tr '[:lower:]' '[:upper:]') #>> $DEBUG # works from bash but not from crontab
      if [ "$encoding" = "UTF-8" ] || [ "$encoding" = "ISO-8859-1" ] #converting 8859-1 breaks the paser
         then
         echo -e "$LINENO Xml version is $version RSS Encoding is $encoding" >> $DEBUG
      else
#         echo -e "$LINENO Reencoding from $encoding to UTF-8" >> $DEBUG
#   xsltproc throws a wobbly if you don't change the file declaration of the encoding
# Delete the first line and   Insert new line at the start of the file
         xmldata=$(echo "$xmldata" | sed '1,1d' )
         xmldata1=$(echo "$xmldata" | sed -e '1i\<?xml version="'$version'" encoding="UTF-8"?>') #>> $DEBUG
# Re encode file to UTF-8
         xmldata=$(echo "$xmldata1" | iconv -c -s -f "$encoding" -t UTF-8) >> $DEBUG
      fi
# If $expression is blank or is the same as the source then use a wildcard
      if [ "$expression" = "$subscription" ] || [ -z "$expression" ]
         then
         expression="."
      fi
# Parsing xml depending on where the torrent url is located inside <link> tags or as the value for the attribute enclosure
      if  fgrep -iq enclosure <<< "$xmldata"
         then
          file=$(echo "$xmldata" | xsltproc -v parse_enclosure.xsl - 2>> DEBUG )
     else
        file=$(echo "$xmldata" | xsltproc -v parse_link.xsl - 2>> DEBUG )
     fi
      file=${file//\ /_}
     file=${file//\\/}
     for url in $file
     do
         if  echo "$url" | egrep -i $expression &> /dev/null
           then
           torrent=$(sed 's/\([^#]*\)#.*/\1/' <<< "$url")
# Another case to cope with demonoid
           if echo "$subscription" | egrep -i "demonoid" &> /dev/null #should beable to do this without nameing an exception
              then
# the parsed page has no download link just a link to a html page with an embedded download link and login .
# got a match download the html page
               downloadpage=$(wget $(echo "$url" | sed 's/\([^#]*\)#.*/\1/') -q -O -)
# the line we need is   example <a href="/files/download/2208330/775222">Click here to download the torrent</a>
                                          #   <a href="/files/download/2208289/6976998">Click here to download the torrent</a>
# had problems here sed seemed to only return the entire webpage so had to break it down into a smaler chunk firs
               downloadurl=$(echo "$downloadpage" | grep -i '<a href="/files/download/[0-9]*/[0-9]*">Click here to download the torrent</a>')
               torrenturl=$(echo "$downloadurl" | sed 's/.*href="\(\/files\/download\/[^"]*\)">Click here to download the torrent<\/a>.*/\1/' )
               torrent="http://www.demonoid.com$torrenturl"
# Put the new torrent file url back into the $url catch file entry will be correct.
               torrentname=$( echo "$url" | sed -e 's/^.*\#//' -e :a -e 's/[\/]/-/;ta' -e 's/[\_]//;ta' -e 's/[]]//;ta' -e 's/[[]//;ta')
               url="$torrent#$torrentname"
            fi
# if not in the catch.log get the torrent
          if ! fgrep -i "$torrent" catch.log > /dev/null
             then
# URL and Mininova fixer
               torrent=$(echo "$torrent" | sed -e "s/mininova.org\/tor/mininova.org\/get/g")
# parse the filename from the end of the $url variable (after the #) and get rid of some of the crap
             torrentname=$( echo "$url" | sed -e 's/^.*\#//' -e :a -e 's/[\/]/-/;ta' -e 's/[\_]//;ta' -e 's/[]]//;ta' -e 's/[[]//;ta')
# append .torrent on the end
             torrentname=$torrentname.torrent
# File may not be in Catch.log but still remain on disk
               if [ -f "$torrentdir$torrentname" ]
                  then
                  echo "File with name $torrentname exists NOT DOWNLOADING!!" >> $DEBUG
               else
# Get the torrent, name it correctly and put it in the right directory
# First check if we need to load the cookie
                  if ! [ -z "$private_feed" ]
                     then
                     echo "$LINENO loading cookie for wget $torrent file" >> $DEBUG
                     if wget --connect-timeout=10 --tries=2 -qncH --load-cookies="$cookie_file" -O $torrentdir$torrentname $torrent
                        then
                        echo -e "$torrentname downloaded using $cookie_file from $torrent" >> $DEBUG
                     else
                        echo -e "Failed to get $torrentname using $cookie_file from $torrent" >> $DEBUG
                   fi
                  else
                     if wget --connect-timeout=10 --tries=2 -qncH -O $torrentdir$torrentname $torrent
                        then
                        echo -e "$torrentname downloaded from $torrent" >> $DEBUG
                     else
                        echo -e "Failed to get $torrentname from $torrent" >> $DEBUG
                  fi
               fi
            fi
# Delete the torrent file if it's an empty file
            if [ -s "$torrentdir$torrentname" ]
                then
               echo "$url" >> catch.log
            else
                echo -e "\n$torrentname is 0kb... deleting..." >> $DEBUG
                rm $torrentdir$torrentname
               fi
            fi
         fi
         echo "$url" >> rssdata
      done
   fi
done < $PRIVATEFEED
echo -e "Run Completed Check for errors above $(date)" >> $DEBUG

[phw198]

Now I just need the final piece to my evil plans: some way to get the files automatically moved to a configured folder on download completion. ie. move from /volume1/download to /volume1/video/TV/<show-name> based on feed.

I have a perl script that does similar - not based on feed, but just the downloaded file name. Just configure $downloadDir and $tvDir directories. The @shows is an array list where the 1st entry is the filename regular expression, and the 2nd entry is the subdirectory name of the $tvDir. Files then get places into further subdirectories eg. "Series 2".

Code: Select all

my $downloadDir = '/volume1/media';
my $tvDir = '/volume1/media/video/tv';
my @shows = (
             ["Good.*Wife", "The Good Wife"],
             ["Lost","Lost"],
             ["Big.*Bang","The Big Bang Theory"],
             ["Flash.*Forward","FlashForward"]
            );

opendir DH, $downloadDir;
FILE: foreach my $file (sort readdir(DH)) {
  if ($file =~ /^(.*s[0-9]{2}E[0-9]{2}).*\.avi$/i ) {
    #The.Good.Wife.S01E05.Crash.HDTV.XviD-FQM.[VTV].avi
    #print "$file\n$1\n";

    my $title = $1;       #The.Good.Wife.S01E05
    my $movedFile = 0;
    SHOW: foreach my $show (@shows) {
      my $show_regex = @$show[0];
      my $show_path = @$show[1];

      if ($title =~ /$show_regex.*(s[0-9]{2})/i) {
        my $series = $1;
        $series =~ s/s/Series/i;
        $series =~ s/0/ /i;

        my $destination = "$tvDir/$show_path/$series/";
        if (-e $destination) {
          system ("chmod 750 \"$downloadDir/$file\" ");
          my $cmd = "mv \"$downloadDir/$file\" \"$tvDir/$show_path/$series/\"";
          print "$cmd\n";
          system ($cmd);
        } else {
          print "Destination $destination does not exist for $file\n";
        }
        $movedFile = 1;
        last SHOW;
      }
    } #show

    if (!$movedFile) {
      print "No rule set for $file\n";
    }

  } #AVI file
} #file

closedir DH;


Then just cron the perl script.

[ingensin]

Hello great script I have been looking for something like this, but I cant seem to get it to run.

I have turn on the DEBUG, but that only outputs the line:

Code: Select all

Executing /volume1/script/synocatch.bsh on Tue Sep 21 08:29:39 CEST 2010

And I have tried to do some debugging my self by echoing some text from varies of places and outputting some variables.

It seems to stop here

Code: Select all

# Protect against the white space gotchas
# file=$(tr ' \\\t\r' '_/__' <<< "$file") # doesn't work on Synology DS207
file=${file//\ /_}
file=${file//\\/}


Code: Select all

echo $file

outputs nothing

I have a Synology 106e, and it has the same processor as the DS207, Is this the reason it will not work for me?


And can someone help me with some examples of this bp.conf.
I want to get series from a rss feed.

[GNaschenweng]

I had similar issues and especially since DSM 3.0 I noticed that the script stopped working altogether. I then changed all references to optware executables (such as sed, tr, xsltproc, fgrep etc) to full paths - i.e. instead of "sed xxxx" to "/opt/bin/sed" this did the trick and the scripts are fully functional again. Not sure why this is, as the paths are set properly in the environment.

[ingensin]

I had similar issues and especially since DSM 3.0 I noticed that the script stopped working altogether. I then changed all references to optware executables (such as sed, tr, xsltproc, fgrep etc) to full paths - i.e. instead of "sed xxxx" to "/opt/bin/sed" this did the trick and the scripts are fully functional again. Not sure why this is, as the paths are set properly in the environment.

That didn't do it...

Is this the right way to do it?

Code: Select all

xmldata=$(/opt/bin/wget $(echo "$subscription" | /opt/bin/sed 's/[^@]*@\(.*\)/\1/') -q -O -)

or this ?

Code: Select all

xmldata=$("/opt/bin/wget" $(echo "$subscription" | "/opt/bin/sed" 's/[^@]*@\(.*\)/\1/') -q -O -)

[GNaschenweng]

I did it without the quotes (your first option). I pretty much replaced everything. If you PM me your email, I can mail you the scripts.

[ingensin]

Her is my quick and easy solution

Code: Select all

  # Parsing xml depending on where the torrent url is located inside <link> tags or as the value for the attribute enclosure
   replvar=${xmldata//windows-1251/UTF-8};
   if  fgrep -iq enclosure <<< "$xmldata"
      then
         file=$(echo "$replvar" | xsltproc parse_enclosure.xsl - 2> /dev/null)
      else
         file=$(echo "$replvar" | xsltproc parse_link.xsl - 2> $DEBUG)
   fi


it was the encoding of the rss feed that crashed my script.

[ingensin]

Now all I need is to figure out how to write that regex and filter out what I want.

like CSI, NCIS, Num3rs, Dexter

Can someone help me, I can't seem to twist my head around that regex thing.

[deadelvis]

This means it doesn't matter whether the script puts the torrent files in the folder or if I do it myself - they'll automatically be picked up and downloaded.

This script will look at the contents of a folder and if it finds any torrent files it will load them... I have a version which does nzb files at the same time too!

Can you please provide the script that handles nzb files too? As far as I've read Downloadstation CLI only supports torrent but no nzb files? So I am curious how you handle those.

I am interested in setting up a "watch folder" for both torrent and nzb... and some of you scripting together with Downloadstation CLI might be the way to go?

[h0me5k1n]

Here's the script - with nzb handling:

Code: Select all

#!/bin/sh
# Script to start .torrent and .nzb files saved in a predefined folder

###CONFIGURATION PARAMETERS
## directory to put the downloaded torrents,with trailing slash
TORRENTDIR="/volume1/public/torrentfiles"
NZBDIR="/volume1/public/torrentfiles"
SCRIPTNAME=`basename $0`

# Make script crontab friendly:
cd $(dirname $0)

# Check for file presence
if ls $TORRENTDIR | grep torrent ;
 then
  find $TORRENTDIR -type f -name "*.torrent" | while read EACHFILE
  do

  # Remove spaces and square brackets from filename
  TORRENT=`echo $EACHFILE | sed 's/\([ ]\)/\\\ /g' | sed 's/\([[]\)/\\\[/g' |sed 's/\([]]\)/\\\]/g'`
  NEWFILENAME=`echo $EACHFILE | sed 's/[ ]*//g' | sed 's/[[]*//g' | sed 's/[]]*//g'`

  # Echo the variables (for debugging)
  #echo TORRENT is $TORRENT
  #echo NEWFILENAME is $NEWFILENAME
  mv "$EACHFILE" "$NEWFILENAME"

  # load into the downloadstation
  if downloadstation torrent "$NEWFILENAME"
   then
    rm "$NEWFILENAME"
  fi 
  done
 else
  echo "No .torrent files found"
fi


#----NZB
# Check for file presence
if ls $NZBDIR | grep nzb ;
 then
  find $TORRENTDIR -type f -name "*.nzb" | while read EACHFILE
  do

  # Remove spaces and square brackets from filename
  NZBFILE=`echo $EACHFILE | sed 's/\([ ]\)/\\\ /g' | sed 's/\([[]\)/\\\[/g' |sed 's/\([]]\)/\\\]/g'`
  NEWFILENAME=`echo $EACHFILE | sed 's/[ ]*//g' | sed 's/[[]*//g' | sed 's/[]]*//g'`

  # Echo the variables (for debugging)
  echo NZBFILE is $NZBFILE
  echo NEWFILENAME is $NEWFILENAME
  mv "$EACHFILE" "$NEWFILENAME"
 
  # load into the downloadstation
  if downloadstation torrent "$NEWFILENAME"
   then
    rm "$NEWFILENAME"
  fi 
  done
 else
  echo "No .nzb files found"
fi


I had to remove some lines related to "logging" as I was saving the output so it was viewable on a webpage but the above should work (I'm sure it did but correct me if I'm wrong). I'm not actually using the Download Station application to download "nzbs" anymore... I'm using SABnzbD for that - but I still use torrents for RSS.

The shebang might be wrong too!

[miracle2k]

Does anyone have experience with how using this (or any cron script really) affects disk spin downs? Say I'm running this every half hour: Will my disks be forced to wake up every time the script runs, due to the cron daemon needing to read the script file? I'd assume so - or is the Synology cron daemon optimized as to cache the scripts. The wiki page on dealing with disk spin downs seems to indicate a difference between the builtin cron and the optware cron, but it doesn't way what it is exactly.