System Instability

cinjun · Jul 10, 2014

I recently added a second vdev to my system, 4x2TB disks to the machine. Unfortunately the system now will randomly freeze, usually after 3 or 4 days. It was running fine for about a year now. I'm not really sure where the issue could be. Anyway here are my specs:

Code:

CPU: Core i7-3770
MOBO:  Asus P8Z77-V LK
MEMORY: 32GB Corsair 4x8GB
HBA: IBM M1015 (IT Mode)
POWER: Corsair CX600W
OS: Ubuntu Server 12.04.4
OS Drive: Samsung 840 Pro SSD
VDEV1: 4x3TB Seagate
VDEV2: 3x2TB WD + 1x2TB Seagate

I've run memtest for 3+ days without problems, even swapping for 4 fresh sticks with no change.

My current guess is either the HBA or the power supply. Are there anyway for me to check these devices through the OS (logs or something) or another way?

drescherjm · Jul 10, 2014

Even though your 3 day memtest passed and you completely swapped your ram I would check if the problem still happens with only 1 dimm installed instead of 4.

cinjun · Jul 10, 2014

I've done tests as well with just a single DIMM. I'm leaning away from RAM because it would have caused problems before I added the new VDEV

drescherjm · Jul 10, 2014

Have you looked at the SMART on the disks in the new VDEV perhaps one or more of them are about to die.

fixedmy7970 · Jul 10, 2014

drescherjm said:
Have you looked at the SMART on the disks in the new VDEV perhaps one or more of them are about to die.

this^. if ur on linux, open terminal and run smartctl

cinjun · Jul 10, 2014

All the drive are marked as PASSED, I'm running an extended test on the oldest drive.

drescherjm · Jul 10, 2014

By experience the full drive SMART PASSED / FAILED status is pretty meaningless. What about the raw values of Reallocated_Sector_Ct, Current_Pending_Sector and UDMA_CRC_Error_Count?

Here is a script that looks at all drives and prints this info.

Code:

#!/bin/bash

function process_device()
{
        echo -n ${device}
        hdparm -I /dev/${device} | grep "al Number"
        smartctl --all /dev/${device} | grep -e "Reallocated_Sector_Ct" -e "Current_Pending_Sector" -e "Offline_Uncorrectable" -e "UDMA_CRC_Error_Count" -e "Hardware_ECC_Recovered"

        smartctl --all /dev/${device} | grep FIRMWARE -C 10
        echo
}

date
                                                                                                                                                                       
for a in /dev/sd?;                                                                                                                                                     
do                                                                                                                                                                     
        device=${a/\/dev\//}                                                                                                                                           
        process_device                                                                                                                                                 
done

Here is an example of the output:

Code:

jmd0 shell-scripts # sh examine_smart.sh 
Thu Jul 10 10:43:11 EDT 2014
sda     Serial Number:      CVEM848601V3080DGN  
  5 Reallocated_Sector_Ct   0x0002   100   100   000    Old_age   Always       -       0

sdb     Serial Number:      W300FC1Z
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

sdc     Serial Number:      WD-WCAVY0212354
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

sdd     Serial Number:      PL1321LAGA2W7H
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       7

sde     Serial Number:      PL2311LAG2UNAJ
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       0

sdf     Serial Number:      S2HGJDWZ806051      
  5 Reallocated_Sector_Ct   0x0033   252   252   010    Pre-fail  Always       -       0
195 Hardware_ECC_Recovered  0x003a   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   252   252   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   252   252   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0036   200   200   000    Old_age   Always       -       0

sdg     Serial Number:      S2HGJDWZ806049      
  5 Reallocated_Sector_Ct   0x0033   252   252   010    Pre-fail  Always       -       0
195 Hardware_ECC_Recovered  0x003a   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   252   252   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   252   252   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0036   200   200   000    Old_age   Always       -       0

sdh     Serial Number:      S2HGJ1BZ836643      
  5 Reallocated_Sector_Ct   0x0033   252   252   010    Pre-fail  Always       -       0
195 Hardware_ECC_Recovered  0x003a   100   100   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   252   252   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   252   252   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0036   200   200   000    Old_age   Always       -       0

sdi     Serial Number:      ML0220F30BW4TD
  5 Reallocated_Sector_Ct   0x0033   100   100   005    Pre-fail  Always       -       0
197 Current_Pending_Sector  0x0022   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0008   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x000a   200   200   000    Old_age   Always       -       4

cinjun · Jul 10, 2014

Code:

Thu Jul 10 10:54:27 EDT 2014
sda     Serial Number:      S12PNEAD312575M
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
195 Hardware_ECC_Recovered  0x001a   200   200   000    Old_age   Always       -       0
199 UDMA_CRC_Error_Count    0x003e   100   100   000    Old_age   Always       -       0

sdb     Serial Number:      4NF1F4VQ
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       21
195 Hardware_ECC_Recovered  0x001a   055   045   000    Old_age   Always       -       75215967
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

sdc     Serial Number:      Z1F1YNS3
  5 Reallocated_Sector_Ct   0x0033   096   096   010    Pre-fail  Always       -       3968
197 Current_Pending_Sector  0x0012   097   084   000    Old_age   Always       -       528
198 Offline_Uncorrectable   0x0010   097   084   000    Old_age   Offline      -       528
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

sdd     Serial Number:      WD-WMC4N2579186
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

sde     Serial Number:      WD-WMC4N2785832
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

sdf     Serial Number:      Z2409W96
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       0
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

sdg     Serial Number:      WD-WCAZA5794872
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0

sdh     Serial Number:      Z1F1WKR9
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

sdi     Serial Number:      Z1F1YLRC
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

sdj     Serial Number:      Z1F1YLYA
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

sdk     Serial Number:      9VS0T81H
  5 Reallocated_Sector_Ct   0x0033   100   100   036    Pre-fail  Always       -       8
195 Hardware_ECC_Recovered  0x001a   057   018   000    Old_age   Always       -       84541729
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0

sdk is an old USB attached drive. sdb is a drive that isn't even mounted, just a drive I pulled some old data off and never removed

cinjun · Jul 10, 2014

I could try disconnecting those devices (sdk and sdb), but I'm not sure if that will do anything.

drescherjm · Jul 10, 2014

sdc looks terrible. 3968 reallocated sectors and 528 that it had data on but can't read (meaning data loss caused by possible head failure). If you are using this in an array I would get it out now.

omniscence · Jul 10, 2014

Run a 'zpool status' please.

cinjun · Jul 10, 2014

Code:

  pool: tank0
 state: ONLINE
  scan: resilvered 64K in 0h0m with 0 errors on Wed Jul  9 09:48:14 2014
config:

        NAME                                 STATE     READ WRITE CKSUM
        tank0                                ONLINE       0     0     0
          raidz1-0                           ONLINE       0     0     0
            ata-ST3000DM001-1CH166_Z1F1WKR9  ONLINE       0     0     0
            ata-ST3000DM001-1CH166_Z1F1YLRC  ONLINE       0     0     0
            ata-ST3000DM001-1CH166_Z1F1YLYA  ONLINE       0     0     0
            ata-ST3000DM001-1CH166_Z1F1YNS3  ONLINE       0     0     0

errors: No known data errors

  pool: tank1
 state: ONLINE
  scan: scrub repaired 0 in 3h0m with 0 errors on Sun Jul  6 08:02:01 2014
config:

        NAME                                          STATE     READ WRITE CKSUM
        tank1                                         ONLINE       0     0     0
          raidz1-0                                    ONLINE       0     0     0
            ata-WDC_WD20EZRX-00D8PB0_WD-WMC4N2579186  ONLINE       0     0     0
            ata-WDC_WD20EZRX-00D8PB0_WD-WMC4N2785832  ONLINE       0     0     0
            ata-ST2000DM001-9YN164_Z2409W96           ONLINE       0     0     0
            ata-WDC_WD20EARS-00MVWB0_WD-WCAZA5794872  ONLINE       0     0     0

errors: No known data errors

There were read errors on 'ata-ST3000DM001-1CH166_Z1F1YNS3' which is sdc, but the tally was cleared after reimporting the pool.

drescherjm · Jul 10, 2014

So tank0 is empty?

cinjun · Jul 10, 2014

No, just the READ WRITE CKSUM results have been cleared

cinjun · Jul 10, 2014

drescherjm said:
sdc looks terrible. 3968 reallocated sectors and 528 that it had data on but can't read (meaning data loss caused by possible head failure). If you are using this in an array I would get it out now.

Seagate has been notified, replacement drive in the mail.

cinjun · Jul 17, 2014

Ok, so I got the new drive yesterday. Took out the old one put the new one in. Unforunately "someone", who shall rename nameless, decided to reorganize the power cables and not plug-in one of the other drives in the vulnerable array. So now when I try to mount the pool I get:

Code:

cannot import 'tank0': I/O error Destroy and re-create the pool from a backup source.

I rechecked the wiring shutdown the system and powered up all the drives now, but I still have the same error.

The only command that has actually returned a non-error so far is:

Code:

sudo zpool import -fFX tank0

That command is still running. Since it started I cannot connect to the system anymore, it responds to ping, but ssh just doesn't return. It doesnt timeout, but it doesn't return.

So I have two questions:
1.) How long should the import -fFX command take? Its running on 11 hours now.
2.) Is there anything else I can do? The data is all there, but am I hosed?

safelyRemove · Jul 18, 2014

always try -fF first before -fFX. But I would think in either case, 11 hours means it isn't working. Hit reset.

Was the system off when you changed out the drives? Can you start it back up with the bad drive?
Do you have an extra port to let you add the replacement drive without removing the bad one?

You might need to do zpool import -T xxx -o readonly=on tank0

run zdb -u -l /dev/sdc |grep txg

Run it for each drive in your pool. Look at all the txg and see if there is a mismatch between your drives, as to which was the highest "txg: " number that seems to show up on each drive, and also has a txg = line on all drives.(If most drives say 4100, but one drive says 4099, you would try 4099. going back more than 4 from the absolute highest number you see may not work) That would probably be the number to try putting in for import -T. If there is no mismatch, (all drives are the same) then it probably isn't the problem. This is a risky process, and can make things worse.

System Instability

cinjun

n00b

drescherjm

[H]F Junkie

cinjun

n00b

drescherjm

[H]F Junkie

fixedmy7970

Gawd

cinjun

n00b

drescherjm

[H]F Junkie

cinjun

n00b

cinjun

n00b

drescherjm

[H]F Junkie

omniscence

[H]ard|Gawd

cinjun

n00b

drescherjm

[H]F Junkie

cinjun

n00b

cinjun

n00b

cinjun

n00b

safelyRemove

n00b