Reboot problem...

[bingo1105]

Hello, all! I'm running a DS209 and when I issue the command 'reboot' from a root shell, I get the following items logged in /var/log/messages:

Sep 21 19:28:45 kernel: [16881.290000] Unable to handle kernel NULL pointer dereference at virtual address 00000000
Sep 21 19:28:45 kernel: [16881.290000] pgd = c0004000
Sep 21 19:28:45 kernel: [16881.290000] [00000000] *pgd=00000000
Sep 21 19:28:45 kernel: [16881.290000] Internal error: Oops: 817 [#1]
Sep 21 19:28:45 kernel: [16881.290000] last sysfs file: /sys/block/md2/md/dev-sdb3/state
Sep 21 19:28:45 kernel: [16881.290000] Modules linked in: usbhid hid usblp usb_storage ohci_hcd ehci_hcd ds209_synobios(P) sky2 ipv6 synoacl_ext4(P) synoacl_vfs(P) fuse vfat fat cryptosoft ecryptfs sha512_generic sha256_generic sha1_generic ecb cesa_o
Sep 21 19:28:45 kernel: [16881.290000] CPU: 0 Tainted: P (2.6.32.12 #1337)
Sep 21 19:28:45 kernel: [16881.290000] PC is at prio_tree_replace+0x2c/0x80
Sep 21 19:28:45 kernel: [16881.290000] LR is at prio_tree_remove+0xe4/0x100
Sep 21 19:28:45 kernel: [16881.290000] pc : [<c023bd4c>] lr : [<c023be84>] psr: 00000093
Sep 21 19:28:45 kernel: [16881.290000] sp : c7dfbd58 ip : c7dfbd88 fp : c7dfbd84
Sep 21 19:28:45 kernel: [16881.290000] r10: 00001000 r9 : c03aa3c0 r8 : 00000000
Sep 21 19:28:45 kernel: [16881.290000] r7 : c7e5eca4 r6 : c7dbf5a0 r5 : cf478f5c r4 : c7dbf5c4
Sep 21 19:28:45 kernel: [16881.290000] r3 : 00000000 r2 : c7dbf5c4 r1 : c7e5eca4 r0 : cf478f5c
Sep 21 19:28:45 kernel: [16881.290000] Flags: nzcv IRQs off FIQs on Mode SVC_32 ISA ARM Segment user
Sep 21 19:28:45 kernel: [16881.290000] Control: 0005397f Table: 07e90000 DAC: 00000015
Sep 21 19:28:45 kernel: [16881.290000] Process sshd (pid: 2248, stack limit = 0xc7dfa270)
Sep 21 19:28:45 kernel: [16881.290000] Stack: (0xc7dfbd58 to 0xc7dfc000)
Sep 21 19:28:45 kernel: [16881.290000] bd40: c014b3b4 c013b584
Sep 21 19:28:45 kernel: [16881.290000] bd60: ffffffff 00000000 00000be9 c7e5ec80 c7dbf5a0 00008000 c7dfbd94 c7dfbd88
Sep 21 19:28:45 kernel: [16881.290000] bd80: c014e0b8 c023bdb0 c7dfbdcc c7dfbd98 c014bf64 c014e030 bef6c000 00000000
Sep 21 19:28:45 kernel: [16881.290000] bda0: ffffffff 00000be9 00000000 00000000 c7e5ec80 cf0ba960 c7e79c00 00000001
Sep 21 19:28:45 kernel: [16881.290000] bdc0: c7dfbe04 c7dfbdd0 c014dde0 c014bf38 c7dfbdd8 00000000 00000097 c03aa3c0
Sep 21 19:28:45 kernel: [16881.290000] bde0: c7dfbfb0 cf0ba960 00000000 cf0ba994 ce8c6cc0 c7dfa000 c7dfbe1c c7dfbe08
Sep 21 19:28:45 kernel: [16881.290000] be00: c010a580 c014dd0c cf0ba960 00000000 c7dfbe44 c7dfbe20 c010dbb8 c010a558
Sep 21 19:28:45 kernel: [16881.290000] be20: c7dfbe44 c7dfbe30 c7dfa000 ce8c6cc0 00000087 c7dfbee0 c7dfbe74 c7dfbe48
Sep 21 19:28:45 kernel: [16881.290000] be40: c010ef38 c010dacc c7dfbe44 00000000 00000087 c7e94ab0 0000008c c7dfbee0
Sep 21 19:28:45 kernel: [16881.290000] be60: c7dfa000 c7dfbf60 c7dfbe8c c7dfbe78 c010f414 c010ede4 00000007 c7e94ab0
Sep 21 19:28:45 kernel: [16881.290000] be80: c7dfbec4 c7dfbe90 c011866c c010f390 c7dfbfb0 c7e94ac0 00000008 c7dfa000
Sep 21 19:28:45 kernel: [16881.290000] bea0: 00000000 c7dfbfb0 00000000 00000000 c7dfa000 0006cbf0 c7dfbfac c7dfbec8
Sep 21 19:28:45 kernel: [16881.290000] bec0: c00e2c7c c0118390 00030002 c7dfbfb0 c011f0f0 00000001 c03aa14c 00000005
Sep 21 19:28:45 kernel: [16881.290000] bee0: 00000007 00000000 00030002 0000d44c 0000d44c 00000000 c7e5ec80 0000d000
Sep 21 19:28:45 kernel: [16881.290000] bf00: c03acd40 00000000 00000000 00000000 00000002 cf437848 c7dfbf4c cb049b60
Sep 21 19:28:45 kernel: [16881.290000] bf20: c7dfbe20 c7dfbe24 c7dfbe28 c7dfbe2c c7dfbe30 00000000 0006ac10 00000000
Sep 21 19:28:45 kernel: [16881.290000] bf40: 00075248 00000000 00000007 00000000 c7dfbfa4 c7dfbf60 c016e30c c016cdbc
Sep 21 19:28:45 kernel: [16881.290000] bf60: 0000d44c 04000000 40228a00 00000000 00000000 c7d6f9c0 00000001 00000000
Sep 21 19:28:45 kernel: [16881.290000] bf80: 0006ac10 ffffffff 0006ac10 00000002 0000008e 00000000 c7dfa000 0006cbf0
Sep 21 19:28:45 kernel: [16881.290000] bfa0: 00000000 c7dfbfb0 c002a248 c00e2c28 00000011 00075248 00000000 00000000
Sep 21 19:28:45 kernel: [16881.290000] bfc0: 00000000 0006ac10 00000002 0000008e 00075248 0006ac10 0006cbf0 00000000
Sep 21 19:28:45 kernel: [16881.290000] bfe0: 00000008 bef6b038 40228a00 0000d44c 00000010 ffffffff 00000000 00000000
Sep 21 19:28:45 kernel: [16881.290000] [<c023bd4c>] (prio_tree_replace+0x2c/0x80) from [<c023be84>] (prio_tree_remove+0xe4/0x100)
Sep 21 19:28:45 kernel: [16881.290000] [<c023be84>] (prio_tree_remove+0xe4/0x100) from [<c014e0b8>] (__remove_shared_vm_struct+0x98/0xa8)
Sep 21 19:28:45 kernel: [16881.290000] [<c014e0b8>] (__remove_shared_vm_struct+0x98/0xa8) from [<c014bf64>] (free_pgtables+0x3c/0xac)
Sep 21 19:28:45 kernel: [16881.290000] [<c014bf64>] (free_pgtables+0x3c/0xac) from [<c014dde0>] (exit_mmap+0xe4/0x14c)
Sep 21 19:28:45 kernel: [16881.290000] [<c014dde0>] (exit_mmap+0xe4/0x14c) from [<c010a580>] (mmput+0x38/0xd8)
Sep 21 19:28:45 kernel: [16881.290000] [<c010a580>] (mmput+0x38/0xd8) from [<c010dbb8>] (exit_mm+0xfc/0x104)
Sep 21 19:28:45 kernel: [16881.290000] [<c010dbb8>] (exit_mm+0xfc/0x104) from [<c010ef38>] (do_exit+0x164/0x5ac)
Sep 21 19:28:45 kernel: [16881.290000] [<c010ef38>] (do_exit+0x164/0x5ac) from [<c010f414>] (do_group_exit+0x94/0xc0)
Sep 21 19:28:45 kernel: [16881.290000] [<c010f414>] (do_group_exit+0x94/0xc0) from [<c011866c>] (get_signal_to_deliver+0x2ec/0x328)
Sep 21 19:28:45 kernel: [16881.290000] [<c011866c>] (get_signal_to_deliver+0x2ec/0x328) from [<c00e2c7c>] (do_notify_resume+0x64/0x5f4)
Sep 21 19:28:45 kernel: [16881.290000] [<c00e2c7c>] (do_notify_resume+0x64/0x5f4) from [<c002a248>] (work_pending+0x1c/0x20)
Sep 21 19:28:45 kernel: [16881.290000] Code: e1530001 05822008 0a000008 e3a03000 (e5833000)
Sep 21 19:28:45 kernel: [16881.300000] ---[ end trace a1643618f8e44a82 ]---
Sep 21 19:28:45 kernel: [16881.300000] Fixing recursive fault but reboot is needed!

The NAS does not reboot; it simply kills the SSH session and hangs. My only option at that point is to shut it down by pulling the power...

This wasn't a problem in the 1285 beta, but it's a problem for both the 1334 and 1337 code releases. I am running OpenSSH installed via ipkg, but I don't think that's what's causing the problem... could be wrong about that, though. Can someone else confirm the issue?

Thanks,
-- Ed

[shimh]

I am in the same situation. I posted it in ipkg forum, and no reply so far. Synology supports suggest reset to default and reinstall the firmware. When I have time, I would disable ipkg in /etc/rc.local to see if the box reboots normally. If it does, a possible solution is in here: http://forum.synology.com/enu/viewtopic.php?f=158&t=25154. One concern is that the problem, which the solution is for, is the key, why it did not happen before until now? Has any one tried the solution?

EDIT: the problem is caused by optware. When it is disabled, the box reboots normally.

[bingo1105]

Agreed... I've found other issues since I posted this, and it looks like there are a variety of changes in 1337 that upset optware. Smartctl core dumps, and the init script for the smart daemon references incorrect paths. Looks like optware is going to have to create a new set of binaries for DSM 3.0...

For the moment, I'm back to a vanilla install. It's not the answer I wanted, but at least it's stable...

[shimh]

bingo1105, what is the vanilla install you refered to?

I tried the solution. It does not solve the problem, which mean it is caused by mount issue. New version of optware is needed I think. Where should we report the issue in case the devs are not aware of?

[bingo1105]

Sorry to be confusing -- I meant that I reinstalled the Synology firmware and removed optware altogether. I imagine this needs to be reported to the optware team, and their website suggests that the best way to do this is to send a message to their mailinglist.

[shimh]

I found the problematic pieces are the installed opt software, especially asterisk. If I do killall asterisk, and maybe other pieces in /opt/etc/init.d, such as ldap, before restart, the restart will be successful. So a simple workaround for now could be to run a script to kill asterisk when shutdown is initiated. Is it possible, guys?

[maxxfi]

Usually at shutdown time most of the startup scripts are run again with a 'stop' parameter, instead of a 'start' one.
That's vital e.g. for databases and filesystems that could get corrupted.
Was it at least a startup script provided?

[shimh]

Yes. In /etc/rc.optware, there are start, stop switches. When the system boots up, a very long /etc/rc script is run. At a point, it calls /etc/rc.local if it exists (I guess it is implemented by Synology), then /etc/rc.local calls "/etc/rc.optware start". rc.optware then calls /opt/etc/rc.optware to run the Sxx scripts in /opt/etc/init.d/Sxx scripts.

In /etc/rc.optware, the "stop" section is

Code: Select all

 stop)
     echo "Shutting down Optware."
     true

It really does nothing. But I can add killall commands here. The real problem is that the "stop" is probably never called. I tested it by adding

Code: Select all

echo "test..." >> /volume1/backup/test

in the stop section. Then I rebooted the box after kill all optware daemon processes. The box rebooted normally, but I don't see the "test" file in the backup share. If I manually run sh rc.optwware stop, the "test" file does appear. Any idea how to make it work? Thanks.

[glowaq]

I have the same problem. A new 1010+ with new harddrives. I use SMB/iSCSI with the device (3.0). If I try to reboot the websession will be disabled and the device blinks with the powerled.
Only pressing the power 10 sec will restart the device. Then I get an error "powerfail".

[maxxfi]

The command to shutdown the Synology, poweroff, seems to contain just a call to
/usr/syno/bin/syno_poweroff_task
This is a ELF executable, but it is possible to read the following inside the code:

Code: Select all

killall -STOP scemd > /dev/null 2>&1
echo "t" > /dev/ttyS1
%s/S*.sh
/usr/syno/etc/rc.d
/usr/syno/etc/rc.d/S95sshd.sh
%s stop > /dev/null 2>&1
touch /var/.NormalShutdown


Seeing that, I don't know if Synology goes through all the directories with startup scripts
but I tried now to make a simple script in /usr/syno/etc/rc.d/
that reacts to parameters 'start' and 'stop' and just write something to a log file
and it works.
So perhaps if you place it as /usr/syno/etc/rc.d/S01stopasterisk.sh you can stop your service
before anything else.

Another possibility is to intercept the 'poweroff' command and replace it with a script
that first do what you want, and then continue with the original 'poweroff' command.

[shimh]

Thanks a lot for the information, maxxfi. I've made all Sxx in /opt/etc/init.d respond to stop switch, and created the Sxx script in /usr/syno/etc/rc.d/ to call /etc/rc.optware stop, which in turn calls /opt/etc/rc.stop_optware, which will then stop all Sxx in /opt/etc/init.d. I will try to reboot the box tonight to see if it works.

EDIT:

I just tested the solution. Working!

[goetz]

Hi,
same problem here on my DS-106 with DSM-3.0 firmware from DS107. Running ipkg mysql4 prevent shutdown. Changing ipkg installation from mount to symlink (done for mounths) does not help. System cant umout /volume1 properly if ipkg processes are running. So I delete all references to optware in /etc/rc.local and generate a new script optware.sh in /usr/local/etc/rc.d

Code: Select all

#!/bin/sh

# Optware setup

case $1 in
start)
        for i in /opt/etc/init.d/S??* ;do

                # Ignore dangling symlinks (if any).
                [ ! -f "$i" ] && continue

                case "$i" in
                   *.sh)
                        # Source shell script for speed.
                        (
                                trap - INT QUIT TSTP
                                set start
                                . $i
                        )
                        ;;
                   *)
                        # No sh extension, so fork subprocess.
                        $i start
                        ;;
                esac
        done
        ;;

stop)

        for i in /opt/etc/init.d/S??* ;do

                # Ignore dangling symlinks (if any).
                [ ! -f "$i" ] && continue

                case "$i" in
                   *.sh)
                        # Source shell script for speed.
                        (
                                trap - INT QUIT TSTP
                                set stop
                               . $i
                        )
                        ;;
                   *)
                        # No sh extension, so fork subprocess.
                        $i stop                       ;;
                esac
          done
          ;;

*)
          echo "Usage: $0 [start|stop]"
          ;;
esac

Regards Goetz

[shimh]

goetz offered a cleaner solution. However, if you don't use symbolic link for opt, a modified optware.sh is required. The following is what you need to do.

1. Remove /etc/rc.local and, optionally /etc/rc.optware and /opt/etc/rc.optware
2. Create optware.sh in /usr/local/etc/rc.d/ and make it executable (chmod +x)

optware.sh

Code: Select all

#!/bin/sh

# Optware setup

if test -z "${REAL_OPT_DIR}"; then
# next line to be replaced according to OPTWARE_TARGET
    REAL_OPT_DIR=/volume1/@optware
fi

case $1 in
start)
        echo "Starting Optware."
        if test -n "${REAL_OPT_DIR}"; then
            if ! grep ' /opt ' /proc/mounts >/dev/null 2>&1 ; then
                mkdir -p /opt
                mount -o bind ${REAL_OPT_DIR} /opt
            fi
        fi
       
        for i in /opt/etc/init.d/S??* ;do

                # Ignore dangling symlinks (if any).
                [ ! -f "$i" ] && continue

                case "$i" in
                   *.sh)
                        # Source shell script for speed.
                        (
                                trap - INT QUIT TSTP
                                set start
                                . $i
                        )
                        ;;
                   *)
                        # No sh extension, so fork subprocess.
                        $i start
                        ;;
                esac
        done
        ;;

stop)

        for i in /opt/etc/init.d/S??* ;do

                # Ignore dangling symlinks (if any).
                [ ! -f "$i" ] && continue

                case "$i" in
                   *.sh)
                        # Source shell script for speed.
                        (
                                trap - INT QUIT TSTP
                                set stop
                               . $i
                        )
                        ;;
                   *)
                        # No sh extension, so fork subprocess.
                        $i stop                       ;;
                esac
          done
          ;;

*)
          echo "Usage: $0 [start|stop]"
          ;;
esac

exit 0


3. Make all Sxx script /opt/etc/init.d aware of start/stop switch. A sample code is the following:

S58slapd

Code: Select all

#!/bin/sh

case "$1" in

    start)
        echo "(Re)start service slapd"
        if [ -n "`pidof slapd`" ]; then
            killall slapd 2>/dev/null
        fi

        /opt/libexec/slapd

    ;;
    stop)
        echo "Stopping service slapd"
        if [ -n "`pidof slapd`" ]; then
            killall slapd 2>/dev/null
        fi

esac

exit 0

[FakeDave]

Hello,

I seem to have the same problem, even with the latest FW.
I'll try tomorrow your suggestions if I understand enough to reproduce them on my DS1010+
Do you know if this problem has been fed back to Synology and if there's any chance of having that fixed in a near future?
Thanx a lot

[shimh]

Technically, it is not Synology's problem. I sent the issue to Synology before. But basically they said the same thing. The key of the fixes is to properly implement start and stop functions in Sxx files and let Synology box to issue start and stop to optware properly (Create optware.sh in /usr/local/etc/rc.d/).