Why does TOS lack resiliance in Power Fail situations?

[Gremlin]

Yesterday we suffered a power outage. I have 2 NAS (see Sig) both of which are connected to UPS devices. TOS 5.1 unit lives in an out-building with a large UPS giving 20100 (seconds?) of backup time (according to TOS) and TOS is set to shutdown after 20 minutes without mains power. The second (TOS6.0) unit is lodged in my bedroom using a slightly smaller UPS.

So, when the power failed at 3:30AM I was woken by 2 factors. Firstly the UPS alarrm sounded and then my diabetes 'low glucose' alarm went off - some coincidence! Given the noise possible waking the household and my need to do something "medical", I shut off the UPS at the physical control panel. No harm, no foul but TOS6.0 failed to recover later. That I sort of accept.

Now, as far as TOS5.1 is concerned, the outcome was far from acceptable. When the mains power was restored some 6 hours later, this machine was also crippled and the UPS totally drained. After much head scratching and restoration of the 'correct' FIXED IP addresses (reinstallation of TOS5.1 and configuration)I was confronted with Degraded RAID (Traid) But 3, apparently, good disks. No errors on face value. Because the disks were "good" (presumed), there seemed to be no facility available to repair the raid. Remember, this machine should have shut down after 20 minutes and remained "sane", but obviously something odd happened. (After over 100 days uptime and no concerns.)

Investigation revealed that 2 out of 3 Traid disks had 2 Volume partions albeit with odd sizes showing (roughly 50%). The third disk showed one Volume partition (roughly the correct size for one). [Note: using cfdisk via command line]. I pulled all the disks and confirmed their status on a (Windows) desktop using a 3rd party partition manager tool. My solution was to "guess" that the disk showing single volume was, potentially, the point of failure so I deleted all the partitions on that disk. I was then able, luckily, to successfully, repair the Traid via the control panel.

TOS6.0 remains broken mainly because there is no installation package to rebuild the existing system and it will have to be done from scratch {AGAIN}! Investigation continues.

What I find totally unacceptable in all this is the failure of TOS to shutdown smoothly, conforming to the setting in TOS, while maintaining a coherent raid configuration. I have seen far too many similar incidents (on this forum) of raid failure due to power issues, whether due to 'outside forces' or 'self induced'. Whatever the reason(s) I believe TOS should be much more resilient because we all know these things are going to happen. {Question: ?Developers living in a vacuum where bad, real world events don't happen?}

[Gremlin]

Back to the TOS6 system issue. This has been fixed, I think.

As before a good portion of time was spent retrieving/setting a good known IP address. Why, when there is an issue, does TOS default to DHCP and not use the fixed IP settings that are still in the system? It would be understandable if the OS and/or settings were lost, but this does not appear to be the case.

Now this is a 2xhdd Traid. That means there is a 50/50 chance of using the correct disk. Examined both in the partition manager. Nothing obvious showed up and at first glance both disks were configured in the same way. However, I happened to notice that the disk/partition array of one disk had a slightly malformed graphical display in that the free/used space bars of the first partition {UTOSDISK} were straying outside of "assigned space". I did not notice anything unusual in 'cfdisk' but I checked both raid hdd several times and the display was consistent. So 1 disk "good", 1 disk "odd" I put the "good" disk back in NAS and it did boot up to TOS6 (with degraded raid).

Deleted all partitions on disk2 (the odd one), formatted with a single ext4 partition (to make sure it worked) then deleted that partition so that the rebuild had a clean disk (relatively) to work on. Inserted in NAS and rebuilt pool array via control panel. On checking this disk via cli, it shows no label (ie UTOSDISK) on first ext4 partition {as I think I have reported previously}. The other partitions (os/swap/volumes) appear to be labelled 'in sync'.

An unrelated issue seemed to become evident durin the course of this event. The TNAS-PC app appeared to default to Chinese for button texts when working on (what the app perceived to be) "uninitiallised" drives. {Thought: Maybe this explains the default to DHCP}

So, back in business but less than happy with, what I believe to be, poor performance of the UPS power handling routines. These need to work flawlessly and consistently especially when being used for remote installations. I don't trust TOS to do this at the moment.

[DigitalAnalog]

Sorry to hear about that experience. It's always nerve-wracking when you're dealing with these events even though you've made reasonable precautions to ensure your system and data's integrity. Unfortunately though, I think you already know what the official TM response will be.

[Gremlin]

One lives in hope

[TMzethar]

We will test to reproduce and verify the situation with reference to your usage experience.

[Gremlin]

I suspect (after some thought) this situation relates to something that has been discussed before. When the power fails and the TOS registers the UPS going to battery, it never really goes into "power off" ('shutdown -P {now}'?) but does a switch to a level level that shuts out users, etc but leaves the OS running.

I know that there are reasons to do this from the 'system' point of view but I feel we need options. ie 'Halt the system after n minutes' THEN 'Power off in nn minutes' BUT 'cancel shutdown if power restored'.

FYI I use APC ups systems and have opted to use 'apcupsd' on my windows machines (for some years now). I don't recall ever having a system corruption issue with these systems. The UPS can be told to wait for a period of time after mains power is restored before supplying power to devices (for example). I am sure NUT based systems can be configured in a more refined way but I don't have experience with them. (NUT for windows was mothballed years ago I believe).

[TMroy]

In future TOS 6 releases, you will have a power-off option for the UPS support.