これまでのあらすじ:
耐障害性の実験
前回は可用性の実験でした。今回はドライブの抜き差しを極端なやりかたで行うことで、やっていいことといけないことを見極め、障害時に採れる行動の感覚を身につけたりノウハウを確立したりが目標。
手順
- ストレージプールをRAIDZで作る
- 抜き差しする過程をそのまま貼り付け
- わかったことがあればメモ
とする。
1台をホットスワップ。zpool onlineコマンドを使った場合
まずzpoolをcreate:
heliopora# zpool create tank raidz /dev/ada[0-3] heliopora# zpool status pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 ONLINE 0 0 0 errors: No known data errors heliopora#
1台をofflineにします:
heliopora# zpool offline tank /dev/ada3 heliopora# zpool status pool: tank state: DEGRADED status: One or more devices has been taken offline by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scrub: none requested config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 OFFLINE 0 0 0 errors: No known data errors heliopora#
そして電源を入れたまま取り外し。コンソール（仮想コンソール1やdmesg）には:
(ada3:ahcich3:0:0:0): lost device (ada3:ahcich3:0:0:0): removing device entry
と出ました。zpoolの方は:
heliopora# zpool status pool: tank state: DEGRADED status: One or more devices has been taken offline by the administrator. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Online the device using 'zpool online' or replace the device with 'zpool replace'. scrub: none requested config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 OFFLINE 0 0 0 errors: No known data errors heliopora#
外す前と表示が変わらず。予備のディスク（ST32000542AS）を追加してみます。以下コンソール:
(ada2:ahcich2:0:0:0): lost device (ada2:ahcich2:0:0:0): removing device entry ada2 at ahcich2 bus 0 scbus2 target 0 lun 0 ada2:ATA-8 SATA 2.x device ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 ada3: ATA-8 SATA 2.x device ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C)
なんと、3台目のディスクまで一度リセットされてしまいました。
zpoolはどうなったでしょう:
heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: none requested config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 REMOVED 0 0 0 ada3 OFFLINE 0 0 0 errors: 2 data errors, use '-v' for a list heliopora#
ada2がREMOVEDに！これはまずそう。
とりあえずada3を参加させます。
heliopora# zpool online tank ada3
…戻ってこなくなりました。onlineじゃなくreplaceすべきだったか！
別端末から入ると、/var/log/messagesには:
Jul 25 11:46:33 heliopora root: ZFS: zpool I/O failure, zpool=tank error=28 Jul 25 11:46:33 heliopora last message repeated 2 times Jul 25 11:46:33 heliopora root: ZFS: vdev I/O failure, zpool=tank path= offset= size= error=
と出てます。さっきのstatusに「データエラーが2つあるよ。-vを付けるとリストアップ」というので-vを付けてみると:
heliopora# zpool status -v pool: tank state: DEGRADED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: none requested config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 FAULTED 0 0 0 corrupted data errors: Permanent errors have been detected in the following files::<0x0> :<0x14> heliopora#
これはどうしようもないかんじ。「action:」にあるようにclearしてみましょうかね。
heliopora# zpool clear tank heliopora# zpool status pool: tank state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: none requested config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 FAULTED 0 0 0 corrupted data errors: 2 data errors, use '-v' for a list heliopora#
あまり変わらないようだけど、最初の端末にプロンプトが戻ってきました。replaceしましょう:
heliopora# zpool replace tank ada3 ada3 invalid vdev specification use '-f' to override the following errors: /dev/ada3 is part of potentially active pool 'tank' heliopora#
以前実験したときのメタデータが残ってるみたい。強制的に使います:
heliopora# zpool replace -f tank ada3 ada3 heliopora# zpool status pool: tank state: ONLINE scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:09:39 2011 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 ONLINE 0 0 0 32.5K resilvered errors: No known data errors heliopora#
正常化！ ふー。
1台をホットスワップ。zpool replaceを使った場合。
それではもう一度外して、今度はonlineじゃなくreplaceを打ちます。まずは取り換えたところ:
heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:09:39 2011 config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 REMOVED 0 0 0 ada3 REMOVED 0 0 0 32.5K resilvered errors: 2 data errors, use '-v' for a list heliopora# zpool replace tank ada3 ada3 cannot replace ada3 with ada3: pool I/O is currently suspended heliopora#
やっぱりおかしなことに。poolのI/Oはデータ保護のためか停止されている状態。
heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:09:39 2011 config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 REMOVED 0 0 0 ada3 REMOVED 0 0 0 32.5K resilvered errors: 2 data errors, use '-v' for a list heliopora#
何も変わってません。まずはada2のonlineでしょうか:
heliopora# zpool online tank ada2
戻ってこなくなりました♪ 別端末からclearしてみます:
heliopora# zpool clear tank heliopora# zpool status pool: tank state: DEGRADED status: One or more devices has experienced an error resulting in data corruption. Applications may be affected. action: Restore the file in question if possible. Otherwise restore the entire pool from backup. see: http://www.sun.com/msg/ZFS-8000-8A scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:09:39 2011 config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 FAULTED 0 0 0 corrupted data errors: 2 data errors, use '-v' for a list heliopora#
ふーむ、なんとかなりそう。
heliopora# zpool replace tank ada3 ada3 heliopora# zpool status pool: tank state: ONLINE scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:19:58 2011 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 ONLINE 0 0 0 37K resilvered errors: No known data errors heliopora#
問題なくいけました。
- わかったこと
- 困ったときは、まずclear。
REMOVEDなどになってしまったときはclearすることでリセットされるみたいです。そこから通常の操作をすれば、なんとかなるという感じ。
外してシャッフルしつつ1台は取り換える
結構安全そうなので極端に行ってみます。
上の全部オンラインになったプールをそのまま使います。電源オンのまま1台を取り外し:
heliopora# zpool status pool: tank state: DEGRADED scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:19:58 2011 config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 REMOVED 0 0 0 37K resilvered errors: No known data errors heliopora#
普通ですね。もう1台取り外し:
heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:19:58 2011 config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 REMOVED 0 0 0 ada3 REMOVED 0 0 0 37K resilvered errors: 2 data errors, use '-v' for a list heliopora#
2台が外れた状態になりました。ところで、ボリュームの中身はどうなってるんだろう？？
heliopora# ls /tank
…また戻ってきません。^Cしてもだめ。どうやら使えない状態のボリュームにアクセスするとペンドされるようです。よく見るとSTATE UNAVAILになってるし。上の方の例でも、戻ってこなくなるときはUNAVAILになってました。
- わかったこと
- UNAVAILのボリュームにアクセスすると操作できなくなる。OS自体は正常なので他から入ってどうにかするとよい。
それでは予備のディスクを入れてみましょう。statusは自動で変わるか？:
heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:19:58 2011 config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 REMOVED 0 0 0 ada3 REMOVED 0 0 0 37K resilvered errors: 3 data errors, use '-v' for a list heliopora#
変わりなし！clearしてみます:
heliopora# zpool clear tank cannot clear errors for tank: I/O error heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:19:58 2011 config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 FAULTED 0 0 0 corrupted data ada3 REMOVED 0 0 0 37K resilvered errors: 3 data errors, use '-v' for a list heliopora#
だいぶひどいことに♪
- わかったこと
- ドライブの交換は順序を保った方がよい。
- わかったこと
- 元々アレイのメンバーでありエラーの無いディスクでも、外すと戻ってこられなくなることがある。
3台目だったドライブを4台目に入れてclearしてみましょうかね:
heliopora# zpool clear tank cannot clear errors for tank: I/O error heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:19:58 2011 config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 FAULTED 0 0 0 corrupted data ada3 FAULTED 0 0 0 corrupted data errors: 3 data errors, use '-v' for a list heliopora#
完全にやってしまった感じです。
- わかったこと
- I/Oエラーの出てるストレージプールにclearは利かない。
強制的にクリアできないかな:
heliopora# zpool clear -f tank cannot open '-f': name must begin with a letter heliopora# zpool clear tank heliopora# zpool clear tank ada3 cannot clear errors for ada3: I/O error heliopora# zpool clear tank ada2 cannot clear errors for ada2: I/O error heliopora# zpool clear tank ada3 cannot clear errors for ada3: I/O error heliopora# zpool clear tank -f ada3 too many arguments usage: clear[device] heliopora# zpool clear tank ada3 cannot clear errors for ada3: I/O error heliopora#
完全に駄目！死亡フラグ！！
- わかったこと
- clearに-fはない。
実際の利用状況からは遠ざかりますが（元のドライブが壊れてないという前提になる）、元のディスクを順番通りに入れ直してみます:
heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:19:58 2011 config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 FAULTED 0 0 0 corrupted data ada3 FAULTED 0 0 0 corrupted data errors: 3 data errors, use '-v' for a list heliopora#
変わらず。何やっても無駄っぽい。でも念のためclear:
heliopora# zpool clear tank heliopora#
あれ？？
heliopora# zpool status pool: tank state: ONLINE scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:41:10 2011 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada3 ONLINE 0 0 0 1.50K resilvered errors: No known data errors heliopora#
正常に戻った！！ たぶん実データがほとんど入ってないせいだと思うのだけど、ものすごいですね。lsしたコンソールもこの時点でプロンプトが戻ってきました。
- わかったこと
- 壊れたアレイも可能な限り元通りにしてクリアすれば取り戻せる場合がある。
- わかったこと
- UNAVAILでペンドされてた処理はONLINEになると実行される。
シャッフル
今度は単純に、全部外して順番を入れ替えます。
まずは極端な例として、ada0,1,2,3の順番だったものを3,2,1,0に入れ替え:
heliopora# dmesg 中略 (ada0:ahcich0:0:0:0): lost device (ada0:ahcich0:0:0:0): Synchronize cache failed (ada0:ahcich0:0:0:0): removing device entry (ada1:ahcich1:0:0:0): lost device (ada1:ahcich1:0:0:0): Synchronize cache failed (ada1:ahcich1:0:0:0): removing device entry (ada2:ahcich2:0:0:0): lost device (ada2:ahcich2:0:0:0): removing device entry (ada3:ahcich3:0:0:0): lost device (ada3:ahcich3:0:0:0): removing device entry ada0 at ahcich0 bus 0 scbus0 target 0 lun 0 ada0:ATA-8 SATA 3.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) (ada0:ahcich0:0:0:0): lost device (ada0:ahcich0:0:0:0): removing device entry ada0 at ahcich1 bus 0 scbus1 target 0 lun 0 ada0: ATA-8 SATA 2.x device ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada0: Command Queueing enabled ada0: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada1 at ahcich2 bus 0 scbus2 target 0 lun 0 ada1: ATA-8 SATA 3.x device ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada1: Command Queueing enabled ada1: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada2 at ahcich0 bus 0 scbus0 target 0 lun 0 ada2: ATA-8 SATA 3.x device ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada2: Command Queueing enabled ada2: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) ada3 at ahcich3 bus 0 scbus3 target 0 lun 0 ada3: ATA-8 SATA 3.x device ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes) ada3: Command Queueing enabled ada3: 1907729MB (3907029168 512 byte sectors: 16H 63S/T 16383C) heliopora#
派手にやったりました。一度認識されたada0がada1を入れたときにリセットし直されてる様子があるので、やっぱ0と1、2と3は組で考えた方がよさそうです。さてstatusは:
heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:41:10 2011 config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 REMOVED 0 0 0 ada3 REMOVED 0 0 0 1.50K resilvered errors: 2 data errors, use '-v' for a list heliopora#
なぜか2台が最初からオンライン！ clearできるかな？:
heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:41:10 2011 config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada0 UNAVAIL 6 0 0 experienced I/O failures ada1 UNAVAIL 6 0 0 experienced I/O failures ada2 FAULTED 0 0 0 corrupted data ada3 FAULTED 0 0 0 corrupted data errors: 2 data errors, use '-v' for a list heliopora#
無茶苦茶なことに…。importを試みましょう:
heliopora# zpool import tank cannot import 'tank': no such pool available heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:41:10 2011 config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada0 UNAVAIL 3 0 0 experienced I/O failures ada1 UNAVAIL 3 0 0 experienced I/O failures ada2 FAULTED 0 0 0 corrupted data ada3 FAULTED 0 0 0 corrupted data errors: 2 data errors, use '-v' for a list heliopora#
これも駄目！元の順序に戻してみましょう:
heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:41:10 2011 config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada0 UNAVAIL 9 0 0 experienced I/O failures ada1 UNAVAIL 9 0 0 experienced I/O failures ada2 ONLINE 0 0 0 ada3 ONLINE 0 0 0 1.50K resilvered errors: 2 data errors, use '-v' for a list heliopora#
あれ？なんか変だよなあ。もしかしてada0とada1は順番を間違ってる？というわけで、入れ替えてclear:
heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:41:10 2011 config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada0 UNAVAIL 9 0 0 experienced I/O failures ada1 UNAVAIL 9 0 0 experienced I/O failures ada2 ONLINE 0 0 0 ada3 ONLINE 0 0 0 1.50K resilvered errors: 2 data errors, use '-v' for a list heliopora#
やっぱり駄目だった…と思ってもう一度抜いたら、メッセージの出る順序が違う。2番目のベイのドライブが先に認識された模様。もう一度正しい順序で認識させてclearする:
heliopora# zpool clear tank cannot clear errors for tank: I/O error heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 12:41:10 2011 config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada0 UNAVAIL 9 0 0 experienced I/O failures ada1 UNAVAIL 9 0 0 experienced I/O failures ada2 ONLINE 0 0 0 ada3 ONLINE 0 0 0 1.50K resilvered errors: 2 data errors, use '-v' for a list heliopora#
駄目だった！うーん、ada2/3は大丈夫なのに、ちょっとここらへんはよくわからない。clearを試みると状態が変わってしまうのか？
駄目な状態でリブートしてみる
リブートを掛けたら次のような状態になりました:
★写真（下の方は電源ボタンを押してうまくいってない状態）
20分放置したけど変わらず。SATAまわりが刺さってるのかな？と思い、ada0/1を外してもこれ以上動かず。すごく駄目な気分で電源ボタン長押しでオフし、もういちど起動してみると:
heliopora# zpool status pool: tank state: ONLINE status: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected. action: Determine if the device needs to be replaced, and clear the errors using 'zpool clear' or replace the device with 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-9P scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 22:32:22 2011 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 1 ada3 ONLINE 0 0 0 errors: No known data errors heliopora#
治っておった…。
- わかったこと
- シャッフルすると大混乱でI/Oエラーに落ちることがある。
- わかったこと
- I/Oエラーは再起動でリセットされる。
シャッフル状態で再起動
I/Oエラーが再起動で直るなら、シャッフルしても平気かもしれないので、その実験。
まず4,3,2,1を外して付け直してみると…:
heliopora# zpool status pool: tank state: DEGRADED status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 22:32:22 2011 config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 3 0 raidz1 DEGRADED 0 6 0 ada0 ONLINE 0 6 0 ada1 ONLINE 0 6 0 ada2 ONLINE 0 6 1 ada3 REMOVED 0 0 0 errors: 2 data errors, use '-v' for a list heliopora#
あれ？なんか普通に認識されてる感じ。clearしてみる:
heliopora# zpool clear tank cannot clear errors for tank: I/O error heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 22:32:22 2011 config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada0 UNAVAIL 6 0 0 experienced I/O failures ada1 UNAVAIL 6 0 0 experienced I/O failures ada2 UNAVAIL 6 0 0 experienced I/O failures ada3 FAULTED 0 0 0 corrupted data errors: 2 data errors, use '-v' for a list heliopora#
駄目だった♪ 期待通りになったので再起動します。またAll buffers synced.で止まりました。電源ボタン長押しでオフ、再度オンしたあと:
heliopora# zpool status pool: tank state: ONLINE scrub: none requested config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ada3 ONLINE 0 0 0 ada2 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada0 ONLINE 0 0 0 errors: No known data errors heliopora#
再起動したら結構直りますね〜。おもしろいのが、ada3,2,1,0の順序になっていること。元の名前を保存しているということらしい。
シャッフルして1台交換
シャッフルが平気なら1台交換しちゃっても大丈夫かも知れない、という実験。
4台を取り外し、バラバラに入れ替えてから1台を予備ディスクに交換してみる。交換するディスクは、1台目とかハードル高そうだから4台目にした。
heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: none requested config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada3 REMOVED 0 0 0 ada2 REMOVED 0 0 0 ada1 REMOVED 0 0 0 ada0 REMOVED 0 0 0 errors: 2 data errors, use '-v' for a list heliopora#
かなり無茶苦茶。clearしてみる:
heliopora# zpool clear tank cannot clear errors for tank: I/O error heliopora# zpool status pool: tank state: UNAVAIL status: One or more devices are faulted in response to IO failures. action: Make sure the affected devices are connected, then run 'zpool clear'. see: http://www.sun.com/msg/ZFS-8000-HC scrub: none requested config: NAME STATE READ WRITE CKSUM tank UNAVAIL 0 0 0 insufficient replicas raidz1 UNAVAIL 0 0 0 insufficient replicas ada3 FAULTED 0 0 0 corrupted data ada2 FAULTED 0 0 0 corrupted data ada1 ONLINE 0 0 0 ada0 FAULTED 0 0 0 corrupted data errors: 2 data errors, use '-v' for a list heliopora#
無理無理！！データのコラプションはかなり怖い感じです。I/Oエラーとも違うので、復活は微妙？
再起動してみる:
heliopora# zpool status pool: tank state: DEGRADED status: One or more devices could not be opened. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Attach the missing device and online it using 'zpool online'. see: http://www.sun.com/msg/ZFS-8000-2Q scrub: none requested config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ada3 UNAVAIL 0 0 0 cannot open ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 errors: No known data errors heliopora#
まっとうな動作になりました。clearとかreplaceとかして完全に修復してみましょう:
heliopora# zpool clear tank heliopora# zpool status pool: tank state: DEGRADED status: One or more devices could not be used because the label is missing or invalid. Sufficient replicas exist for the pool to continue functioning in a degraded state. action: Replace the device using 'zpool replace'. see: http://www.sun.com/msg/ZFS-8000-4J scrub: none requested config: NAME STATE READ WRITE CKSUM tank DEGRADED 0 0 0 raidz1 DEGRADED 0 0 0 ada3 FAULTED 0 0 0 corrupted data ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 errors: No known data errors heliopora# zpool replace tank ada3 ada3 heliopora# zpool status pool: tank state: ONLINE scrub: resilver completed after 0h0m with 0 errors on Mon Jul 25 13:57:15 2011 config: NAME STATE READ WRITE CKSUM tank ONLINE 0 0 0 raidz1 ONLINE 0 0 0 ada3 ONLINE 0 0 0 40K resilvered ada0 ONLINE 0 0 0 ada1 ONLINE 0 0 0 ada2 ONLINE 0 0 0 errors: No known data errors heliopora#
ばっちり。
- わかったこと
- RAID的に助かってるはずのアレイは助かる。順序はあまり関係ない。
- わかったこと
- 助かってるアレイがオンラインにならないときは再起動で直る。
- わかったこと
- ホットスワップ＆クリアは使えることもあるが、あまり信用できない。困ったら再起動してみるべき。
だいぶ感覚が掴めてきた。Linux mdよりずっと使いやすいことは確か。
