Transferred from my old blog.
I ran into an issue this weekend where my raid 6 array absolutely refused to rebuild (I was adding in a new spare since another drive had legitimately died), it kept marking multiple drives as failed and setting the array as degraded. Looking at /var/logs/messages I saw it was peppered with sections of errors like the below:
Nov 12 20:48:41 articles kernel: ata9.03: status: { DRDY DF }
Nov 12 20:48:41 articles kernel: ata9.03: cmd 60/40:d0:10:9c:22/00:00:02:00:00/40 tag 26 ncq 32768 in
Nov 12 20:48:41 articles kernel: res 60/40:d0:10:9c:22/00:00:02:00:00/40 Emask 0×9 (media error)
Nov 12 20:48:41 articles kernel: ata9.03: status: { DRDY DF }
Nov 12 20:48:41 articles kernel: ata9.03: error: { UNC }
Nov 12 20:48:41 articles kernel: ata9.03: cmd 60/c0:e8:50:93:22/00:00:02:00:00/40 tag 29 ncq 98304 in
Nov 12 20:48:41 articles kernel: res 55/35:00:00:00:00/00:00:00:d0:55/00 Emask 0×81 (invalid argument)
Nov 12 20:48:41 articles kernel: ata9.03: status: { DRDY ERR }
Nov 12 20:48:41 articles kernel: ata9.03: error: { IDNF ABRT }
Nov 12 20:48:41 articles kernel: ata9.04: exception Emask 0×100 SAct 0×0 SErr 0×0 action 0×6 frozen
Nov 12 20:48:41 articles kernel: ata9.05: exception Emask 0×100 SAct 0×0 SErr 0×0 action 0×6 frozen
Nov 12 20:48:41 articles kernel: ata9.15: hard resetting link
Nov 12 20:48:41 articles kernel: ata9: controller in dubious state, performing PORT_RST
This seems to be a common problem with port multipliers on Linux, or at the very least a common problem with the Silicon Image controllers on Linux — and there seem to be precious few outside of Silicon Image. (Side note — I’m running the latest firmware available for the controller card) Even still under heavy read/write conditions I see these error messages in my system logs, but they no longer result in drives being marked as failed and getting kicked out of the array.
Here’s what I did to get mdadm to stop marking drives as failed after the Silicon Image controller performed port resets. There is a feature for most modern SATA drives called NCQ (Native Command Queuing). This feature takes 32 read-write commands in queue and then optimizes their order based on physical location of the read head in order to minimize the movement needed to complete all requests and thus (ideally) speed up performance and extend drive life time. This only leads to increased performance under certain (usually server oriented) load conditions. I set the NCQ depth to 1 for all the drives in my array, which effectively disables NCQ — i.e. there is only one item in queue at any given time, so there is no optimization taking place. Again this did not get rid of the port resets, but it did stop the drives from getting marked as failed by mdadm.
Here’s my theory why. NCQ queues 32 read/write commands. That means whenever the sil24 driver decides the controller is in a bad state (which it will do when it has three or more devices with outstanding commands, I believe — I read mention of this in a post by one of the devs), it will perform a port reset which will results in the drives powering off and on again, when this happens they drop their queue (this is speculation — but I would be very surprised if the queue survived a power cycle), which means up to 32 pending read writes go bye bye. In that case data isn’t written to or read from the drive as requested and could end up causing mdadm to assume the drive is faulty. This is only a theory, but regardless of its correctness this action did fix my array problems.
To set the NCQ depth to 1:
echo 1 > /sys/block/sdX/device/queue_depth
To turn NCQ back on:
echo 31 > /sys/block/sdX/device/queue_depth
Where X above is the device letter. (Note you need root privileges to run the above commands, so become root or use sudo.)
My server is using:
* Fedora Core 11 — latest available kernel (Aging yes, but not yet worth the effort to upgrade)
* Silicon Image 3132 based PCIe eSATA controller
* Sans Digital 8 TR8M bay eSATA tower (using a port multiplier to convert 8 SATA ports to 2 eSATA ports
No comments:
Post a Comment