• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

ZFS zpool replace question

Z

Zerosum

Limp Gawd
Joined
Feb 9, 2010
Messages
216
I'm running a pretty beefy zfs server for quite awhile now. Probably 3-4 years. And never really any problems.

I had drives go bad before, I mean it happens especially when I'm running 72 drives (at almost 200TB) for all this time. When I built my server and the chassis I always order an extra drive or 2 for this reason as spares. And when the time comes, a simple offline, hotswap, replace, online, scrub (iirc) and I'm back in business. Couldn't ask for anything simpler.

So this past week one of my drives died. I pulled it, swapped and replaced it without issue. But I noticed another issue with this pool.

Code: 
root@blackhole:/# zpool status Movies
  pool: Movies
 state: DEGRADED
status: One or more devices has been taken offline by the administrator.
        Sufficient replicas exist for the pool to continue functioning in a
        degraded state.
action: Online the device using 'zpool online' or replace the device with
        'zpool replace'.
  scan: scrub in progress since Fri May  2 11:42:36 2014
    787G scanned out of 52.0T at 602M/s, 24h46m to go
    0 repaired, 1.48% done
config:

        NAME        STATE     READ WRITE CKSUM
        Movies      DEGRADED     0     0     0
          raidz2-0  ONLINE       0     0     0
            c4t7d0  ONLINE       0     0     0
            c4t7d1  ONLINE       0     0     0
            c4t7d2  ONLINE       0     0     0
            c4t7d3  ONLINE       0     0     0
            c4t7d4  ONLINE       0     0     0
            c4t7d5  ONLINE       0     0     0
            c4t7d6  ONLINE       0     0     0
            c4t7d7  ONLINE       0     0     0
            c4t8d0  ONLINE       0     0     0
            c4t8d1  ONLINE       0     0     0
            c4t8d2  ONLINE       0     0     0
            c4t8d3  ONLINE       0     0     0
          raidz2-1  DEGRADED     0     0     0
            c4t8d4  ONLINE       0     0     0
            c4t8d5  ONLINE       0     0     0
            c4t8d6  ONLINE       0     0     0
            c4t8d7  ONLINE       0     0     0
            c4t9d0  ONLINE       0     0     0
            c4t9d1  ONLINE       0     0     0
            c4t9d2  ONLINE       0     0     0
            c4t9d3  ONLINE       0     0     0
            c4t9d4  ONLINE       0     0     0
            c4t6d5  OFFLINE      0     0     0
            c4t9d6  ONLINE       0     0     0
            c4t9d7  ONLINE       0     0     0

errors: No known data errors

The drive that WAS bad was c4t8d7. I've already replaced and got it back online. But if you notice c4t6d5 is 'faulted'. I took it offline when I noticed the fault. Which was strange as I never touched that drive nor was any problems reported on it on my Areca logs. All the previous replaces always alarmed my areca logs first which always prompted me to replace said drive asap.

The other issue is, this says c4t6d5. The drive should be c4t9d5

That drive does exist in my machine and I tried replace the one listed with that.

Code: 
root@blackhole:/# zpool replace Movies c4t6d5 c4t9d5
invalid vdev specification
use '-f' to override the following errors:
/dev/dsk/c4t9d5s0 is part of active ZFS pool Movies. Please see zpool(1M).

root@blackhole:/# zpool replace -f Movies c4t6d5 c4t9d5
invalid vdev specification
the following errors must be manually repaired:
/dev/dsk/c4t9d5s0 is part of active ZFS pool Movies. Please see zpool(1M).

So I'm wondering what I'm doing wrong or what could possibly be happening.

Just for reference: I'm running ESXI/OI with an areca vt-d and napp-it 0.9b3. That drive listed (c4t6d5) is actually in another pool in the same machine that's functioning perfectly. I just don't understand how this pool ended up with this drive/label in the incorrect pool. Duplicated and at the same time even.

I'll can provide any additional information you think would help or needed.
 
danswartz said:
Very weird. Have you tried exporting&importing the pool? Rebooting?
Click to expand...

No and no. I've never exported or imported anything before so I'm unfamiliar with the process. I thought that was only done when moving the drives to a new system.

I have not rebooted in awhile. At least over a month and that's only because I wanted to do some work on the main esxi boot drive. It's running ESXI and rarely have to reboot anything. There's a bunch of VMs on the system as well. It's very common for my machines to have 6-12 months uptime. Usually only during extended blackouts or major hardware upgrades. I've always just hot swapped out drives before without problems (Areca 1880).

I'm willing to reboot if you think it'll help. As for exporting and importing, I'm going to need alittle more guidance probably.

Downtime is downtime. It's my server and being so large and while not mission critical, it does serves more then just me. Basically keeps the family happy. =)
 
zpool replace needs as second disk an available disk that is/was not part of a pool-
optionally repartition the disk ex on windows to ntfs if it was part of a pool.

On modern SAS2 systems in IT mode, these disk id's are unique (WWN) like the Mac adress of a nic.
If you move such a disk around (ex c6t50014EE204EC5FCCd0) the id from t500 up is always
the same and you can identify a disk by this id.

Systems like yours with short id's like c4t1d0 are controller port or driver generated id's.
If you move a disk, you get a new id. Do not try to "re-use" old id's from what they were earlier.

What you can do is pool export/import as it re-reads the id's after a disk move with short id's
(napp-it menu pool export, pool import)


What I do on disk failures in both cases:
Insert a new disk (non-ZFS). On a hotplug capable system you need to wait some seconds until it is detected (=new-disk),
otherwise you need a reboot.The displayed id is the current id of the new unused disk.

do a zfs replace pool failed-disk new-disk or a napp-it menu disk-replace
no need to ofline/online/unconfigure/configure or remember id's (unless you want to try hotplug on a non hotplug capable system....)

or add a hotspare to your pool to auto replace a faulted disk.
Keep in mind. The hotspare disk keeps the hotspare property until you remove this or do a replace active hotspare -> new disk
to allow the hotspare disk to go back to a available hotspare state.
 
Last edited:
Thanks, I ended up rebooting and then exporting and importing the pools. Everything seems to be working fine now. Waiting for it to finish resilvering then I'll do a scrub just to be safe.

I did notice that I was able to export via Napp-it without issues but tring to import it gave me an orange box and some error that 'Movies already exist' message. I ended up just going to the CLI and %zpool import Movies, and it worked without any errors or warnings.

Also as I haven't done much to the pools since I originally installed the system, they currently say
Code: 
status: The pool is formatted using a legacy on-disk format.  The pool can
        still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'.  Once this is done, the
        pool will no longer be accessible on software that does not support feature
        flags.

Should I upgrade them? They always worked fine before and never saw a reason to upgrade just for some 'features'. But if it doesn't harm, I'll do the upgrades after all this is through.

Thanks for all your help!
 
Zerosum said:
Should I upgrade them? They always worked fine before and never saw a reason to upgrade just for some 'features'. But if it doesn't harm, I'll do the upgrades after all this is through.
Click to expand...
Generally, upgrading your pool makes it LESS compatible. I'd not recommend it unless there's a feature you need.
 
IIRC doing 'zpool upgrade -v' will tell you what is 'out of date'. Chances are it's features that pretty much everyone but oracle supports now - so unless you will want to use the pool with a legacy distro, you can upgrade to get rid of the messages. Some of the new features are performance improvements, others are things like lz4 compression, which is better than the original lzjb...
 
On the flip side, if you decide that you want the 4x rebuild speed improvement in the new Oracle-only ZFS, and you upgrade to some OmniOS/Illuminos/SmartOS/etc version, you're SOL.
 
Ah, so i see.

I'm using version 28. The only features I see new (with -v) are:
Code: 
FEAT DESCRIPTION
-------------------------------------------------------------
async_destroy                         (read-only compatible)
     Destroy filesystems asynchronously.
empty_bpobj                           (read-only compatible)
     Snapshots use less space.

Both of which are unimportant to me. So unless there's something internal, I guess I can just hold of upgrading. I never see the messages unless I console in. This machine is basically on and forget. I rarely have to log in for anything which is the way I like it. Less admin work.

I don't do anything fancy on the server. It's just a great big storage for my home use. Nothing enterprise-y.
 
I thought that was for deleting snapshots or clones. Not for individual files. I was interpreting that as filesystems and not files. But I'm far from someone who I would trust to be correct on these things. :D
 
Actually, I believe you're right. My memory played me false. Filesystems, volumes, etc....
 
You must log in or register to reply here.
Back
Top