Power Failure DS2411+ caused Volume Size Mismatch

[SusiBiker]

I installed a new UPS on my 2411+/1211 system today.
Now when I boot, I get a single beep just as the system comes online and somehow I now have 23TB Volume, with 23TB free, and er, 17TB of accessible files on it.
Storage Manager bugs out after about five minutes saying that the connection has been lost.
You can still get to the volume via CIFS.

HOWEVER

After about 10 minutes, CPU utilization hits 99% and a quick look at the resource monitor says that there are 10 copies of resize2fs sharing out the processor between them.
I am trying to copy off what I can, but after half an hour the system will grind to a halt.

I've sent a request to Support, but heard nothing and am starting to get very desperate.
Any helpful ideas anyone?

Thanks,
Susi xx

[danielp]

Dear Susi,

I see Alex is assisting you in your support ticket.
Please be patient as we will respond to your ticket in our queue, as long as there are no problems with your email.

Thank you.

[melliott716]

Note I am experiencing a very similar situation on my DS2411+. After a power failure, the DiskStation restarted and appeared to be running normally, but when trying to access Storage Manager, it stops with a connection failure. I also have multiple instances of resize2fs running, taking all the CPU cycles.

I have not experienced the "grind to a halt" referenced below.

Thanks,
Mike

[melliott716]

Note that the number of running resize2fs processes seems to be tied to the number of times you've tried to start Storage Manager. I've restarted (power-off/power-on) my DS2411+, and there is only a single instance running now, using about 25% of the cpu (in addition to md2_raid5 & md2_resync, using between 10-25% cpu each). So trying to start Storage Manager multiple times exacerbates the situation. I have a support incident in, waiting for a response. In the mean time, avoiding trying to start Storage Manager (until I hear back from support or the Storage tab of the monitor no longer indicates that volume is undergoing a parity consistency check).

Mike

[SusiBiker]

Hi Mike, and thanks for the reply.

I had raised a query with TechSupp and they were attempting to help me out. However, I had to go away for a few days and we needed to get the 2411/1211 up and running again. The decision was taken to nuke the volume and OS installation and rebuild from scratch.
This was done mainly because our system had sort of grown organically and needed a bit of serious reorganisation. We were very lucky in that the damaged volume was still available in that we could get off new files that were waiting to be backed up.

Current Situation:
Rebuilt volume and reinstalled OS.
New UPS.
A second Synology server setup consisting of another DS2411+ and a DX1211 as a backup, offsite, with a decent UPS too.
A new respect for Synology TechSup (who were great; take a bow Alexander), and a sadness that the Synology cut-down Busybox Linux implementation does not appear to have the basic tools for manually working on EXT4 drives. That or the other functions that have been added, are undocumented.

Thanks all.
Much respect,
Susi.
xx

[melliott716]

You are fortunate that you did not have a significant quantity of data. There is apparently a serious bug in the Synology DSM that results in file system damage when a power failure occurs, where their tools are unable to fix the damage. The recourse seems to be to back up your data, and then delete and recreate the volume. Not exactly what I would call a user-friendly solution, and definitely not one that's appropriate for business use.

I have 17TB of data. I expect this process will take me more than 2 weeks to accomplish.

On the plus side, at least I did not lose any data.

[SusiBiker]

Hi Mike.
Thanks for the Heads Up regarding the DSM bug. That's NASTY!

I think you might have misunderstood what I wrote earlier on. The system was *reporting* 23TB available, on a 23TB sized array, but there was actually a similar amount of data present to that of yourself - *17TB*. This was why I was more than a bit concerned!
We were in the process of putting new data onto the server before doing a full backup when the incident occurred.
Luckily, we had all the data, it was just scattered all over the place and therefore needed consolidating, again, and backing up.

And yes, you are right, it does take a wee while to get everything back together. Thankfully, it actually took about a week to get everything off, delete the volume, and rebuild the array. Thank Synology for supporting bonded Gb Ethernet!
BTW, things REALLY slow up if you start writing to the array before it has finished expanding. But then, you already know that. In the rush I somehow "forgot."

I really hope you get your system sorted out.
If you get time, could you please let me know how you get on?
Thanks.

Best Wishes for a speedy recovery,
Susi

[SusiBiker]

The damn thing has happened AGAIN!!!
The 2411 failed to shutdown. After two hours of no activity other than that sodding blue flashing light, we pulled the power.
Once again, we have a corrupted file system showing 17TB of data with 17TB free on a 24TB volume, BUT the graphic and the warnings that we have run out of space. So again, it is totalled!
TWICE we have shut this system down, and TWICE, it has died!!
Am I really supposed to believe that the two thousand pounds we spent on the Synology hardware alone is money well spent?
FFS!

[SusiBiker]

Update... only a little while later....
Now have multiple copies of resize2fs running. I only started Storage Manager ONCE to see if it would connect.
The first ten listed items in the CPU usage ranking are? Yes, a prize to you, and all poor souls suffering from this apparently "known" fault! You are correct! All are resize2fs AND CPU utilisation is at its happy 100% mark!

All this, because the server room is being decorated and I wanted to protect my precious server. I'm too damn scared to turn off the off-site backup now unless that suffers the same fate!

In tears. Heartbroken. Sadly disappointed. All trust in the system lost. Distraught.

I wonder if Amazon will do a refund?


EDIT 19June12
A new has dawned. I have slept, a little.
TechSup have been in touch around 3am today (in the UK) and have requested my debug.dat file. Duly sent off with a more detailed description of what occurred and am now waiting patiently, but somewhat resignedly, for a reply.
I have been told that the TechSup folks are good, so fingers crossed.


EDIT 21June12
Sent TechSup the information they requested, now two days or thereabouts ago, and... no reply.
Giving them the benefit of the doubt. They may be extremely busy.

However, the problems remain -
1. We have a Server that is reporting a 23,785.65GB ARRAY, that has *17,179,868,899.69GB USED* and *-17,179,845,114.04GB AVAILABLE*.
I have named this phenomenon the: Synology TARDIS Drive, or STD for short!!

2. We are terrified to switch off our almost identical backup system in case that screws up as well.

3. This server is in Limbo until TechSup get back to us. We dare not use it. That's a lot of money sitting idle.

4 and most important Can we actually rely on our Synology storage solutions if they randomly corrupt the volume making it unusable and require a complete rebuild and reinstall of the DSM every time you need to shut it down and restart it?

So, what are we doing now? Apart from sulking that is?
We are having to make another backup, of our backup server, onto lots of external USB drives connected to Windows 7 machines. I'm beginning to wonder whether a home-built Windows 7 "server" with a few decent SATA array cards and some SuperMicro multiple-drive-bay-units might not have been a better choice?
Much cheaper to service and easier to repair volumes.

DAMMIT!! I still LOVE my Synology beasties! They look so cute
I just wish they'd work reliably...

[Ion23]

I have a similar problem. If you do get a workable solution to this, please let me know. I woke up one day and the storage manager was reporting 34TB of free space, out of my total of 10TB. Tried to reinstall the DSM using the reset button method, but it didn't solve the problem. Also tried to "force" the storage manager to degraded mode by removing one of the HDDs, also to no avail. I just cannot seem to access the storage manager, always encountering a connection failure. All other functions and data are intact.

[SusiBiker]

Dear Ion23,

Before you do anything else, please open a support ticket. We need to get this seriously basic fault in system protection sorted! Hell, even NTFS protects itself from power-off faults to some degree!

Anyway, I hate to be the bringer of bad news, but if you have the same problem, the volume is going to be shot-through with errors.
Do your system logs show an unscheduled power-off reboot? If not, you may be in luck. It could just be something simple.
If everything is showing (Normal) in the System Health window, things are not looking so good. Also, if when you start Storage Manager, it fails to connect, and the number of copies of resize2fs then eventually fills the CPU Usage Ranking list after a few minutes, then sadly, welcome to the club

The good news, what little there is, is that I have had this super-nice guy (all praise to Persey!) from Synology TechSup helping me for the past 24 hours or so. So Synology are aware of what is going on now. However, according to him for me, things are not looking good.
He said to me yesterday that this improper shutdown caused the files from RAM and/or the SWAP file being lost as they weren't written to the HDD before the manual power-off. This obviously wrecks things.

Just as a curiosity and a 'heads-up', did you by any chance install the Plex Media Server software from the Plex website?
Apparently, It can cause a power-off hang on shutdown that eventually wrecks the system when you manually power off. After about two hours of no disk activity, I manually powered off. My guess is that is what caused my woes this time around. The first power-off crash was due to a UPS failure.
Here is the Plex thread that I referred to - http://tiny.cc/80kegw

This awfully nice engineer said he is going to look at my disk logs in an attempt to finger exactly what caused the hang before I have to nuke the system later today.
He did say yesterday that there was an extremely thin chance that it "might be fixable", but the damage to the files on the storage volume would be great. Better to get any files I need off while they are intact then start again from scratch.

**This is STUPID** and should not be happening with "Business-Grade Hardware!"

Thinking about it, this will actually be the third time I've rebuilt the system from scratch. 24TB volume with ~17TB of data <sigh>

I'll let you know what he finds as soon as I have the information.

Regards,
Susi xx

PS Sorry 'bout the verbose answers, but I'm seriously disabled and don't get out much

PPS I still think Synology are the best. They just need to make things *More Robust*

[Ion23]

Hi Susi,

My problem is not that serious. Suspect there is something wrong with the storage manager itself. I can still access my data etc. I am of course unable to access the storage manager itself to make any changes. Already logged in a ticket. I had a similar thing happen to me when I had my 1010+, where there was a power failure, and my partition tables went crazy. System showed normal, but space was incorrect. No matter how many times I expanded the volume etc, it still showed wrong space. In the end, Synology support had to remotely correct the error. I just feel that they should make some of these options available to the end user, instead of just calling them to solve all these problems. I suspect they have to do the same thing again to remotely solve my storage manager problem.

I wouldn't worry so much about your data. I think they have a way to recover it.

[SusiBiker]

Hiya,

Thankfully, I can also get to my data no problems. Like you say, it is the underlying partition tables that are apparently wrecked.
I have been told that it would cause more problems than it would fix by correcting the table in my case.

I don't mind reinitialising the system again too much. Not now that I've had a chance to calm down from the the original crash incident at the beginning of the week.
I just really don't want to be going through this again the next time we come to shut the server down. And the next time... And the next time.......

I bought at the "Small and Medium Business" level in the hope that I would be getting a "Professional" piece of kit that is reliable and more stable than "Home-spec" systems. I did this because I didn't want lots of silly problems with system management that requires a lot of work at root level. I wanted, as near as possible, a Business-Grade Plug-And-Go system.
What I actually got was something that has great specs, looks awesome, and works great - until you need to shut it down or there is a power problem. Oh, and apparently, it sometimes requires a lot of rummaging around under the bonnet - *at root level*.

Every time a reboot is required, however occasionally, it feels like another 24TB game of Russian Roulette has begun -
Will it crash this time...? Or, will it live for another day? Will it...? Won't it...?

I am just scared of the day when both the 2411+ and the 1211 are fully populated with 3TB drives. Terrified in fact.

Hope you have better luck.
Susi.

[tobysmurf]

My DS2411+ has done the same thing. I have a base 2411+ filled with 3TB HDDs, and I show 26.82TB capacity, 0 used and 17 exabytes free. If I purposely fill up a bit beyond that it changes to 26.82TB capacity, 260GB used and 26.6TB free. This array actually has over 15TB of data on it. It was fine until our first Synology crash - ever since then this has been the case. And now I too have an instance of resize2fs running on my machine sucking up 25% CPU.

[SusiBiker]

My DS2411+ has done the same thing. I have a base 2411+ filled with 3TB HDDs, and I show 26.82TB capacity, 0 used and 17 exabytes free. If I purposely fill up a bit beyond that it changes to 26.82TB capacity, 260GB used and 26.6TB free.

Hi and welcome to the forum,
I'm just sorry to hear that this is the reason you have visited.

First and foremost, from your DSM4.1 webman web page, click on the "Report Bugs" tab at the bottom and report this because, the sooner you flag your problem, the sooner you will get help.
Be aware that *IF* they reply, it might take two or more days for them to get back to you. More at the weekends.

If you do get a reply from Tech Support, they will probably ask permission for you to set them up with temporary admin account password, and a couple of ports to be opened in your firewall and DiskStation.

*BEFORE* you let Tech Support near your system, if you do not have a backup of your data, make this your Number One Priority and get it done, NOW!

I hope I am wrong, but after some messing around, they will more than likely tell you that your file system is too badly damaged and that you will have to reinitialise the system and install from scratch: DSM, data, packages, the lot.
I do hope I am wrong.

I thought the EXT4 file system was pretty much bullet-proof and that is why it is being used. Perhaps not. Also, I have been led to believe, that the utilities we need to fix this problem ourselves have been excised from the DSM/Linux system.

Can anyone *in the know* comment on this?


I hope your problem can be fixed.
If not, I'm afraid it's time to bite the bullet: Reinstall

How did your 2411 come to be in this position? Was it a power-failure? Or what?
If it was power-fail related- Time to get a UPS, and quick!

Best of luck,
Susi