• Some users have recently had their accounts hijacked. It seems that the now defunct EVGA forums might have compromised your password there and seems many are using the same PW here. We would suggest you UPDATE YOUR PASSWORD and TURN ON 2FA for your account here to further secure it. None of the compromised accounts had 2FA turned on.
    Once you have enabled 2FA, your account will be updated soon to show a badge, letting other members know that you use 2FA to protect your account. This should be beneficial for everyone that uses FSFT.

Freenas/ZFS issues

J

Jaheed

n00b
Joined
May 31, 2010
Messages
47
So I have had this problem intermittently for some time, but connecting to my freenas server over my home network seems to be a very high latency proposition. I don't think it is a disk health issue, but I could be wrong.

Essentially, what I see is that, intermittently, I will try to do something on the server (move a file, open a folder, rename something, really anything), and then I get a blissful 10-20 seconds of wait before it actually happens. Alternatively, it will just time out. Try the process again, and it may take a while or time out.

This will happen for periods of time that could probably be measured in hours, and then everything will work fine for a few days or weeks, and then the same thing. I figured it might be some diagnostics or something running on the server, but the web interface doesn't indicate anything.

I am running freenas 8.2.0 RC and accessing via Win 7 Boxes.

Any help or ideas are greatly appreciated.
 
If you go to a shell and do 'dmesg' right after recovering from a mini-hang, what do you see?
 
Well, I can access the web interface and run the dmesg during one of these mini-hangs or ultralong-latency periods. Both of these interfaces run without issue or delay.

I will post the dmg results when I can get my SSH client to hold more than 25 or so lines.
 
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
vgapci0: <VGA-compatible display> port 0xcc00-0xcc07 mem 0xfb800000-0xfbbfffff,0xd0000000-0xdfffffff irq 16 at device 2.0 on pci0
agp0: <Intel Ironlake (D) SVGA controller> on vgapci0
agp0: detected 32764k stolen memory
agp0: aperture size is 256M
pci0: <simple comms> at device 22.0 (no driver attached)
uhci0: <UHCI (generic) USB controller> port 0xc880-0xc89f irq 21 at device 26.0 on pci0
uhci0: [ITHREAD]
usbus0: <UHCI (generic) USB controller> on uhci0
uhci1: <UHCI (generic) USB controller> port 0xc800-0xc81f irq 18 at device 26.1 on pci0
uhci1: [ITHREAD]
usbus1: <UHCI (generic) USB controller> on uhci1
uhci2: <UHCI (generic) USB controller> port 0xc480-0xc49f irq 19 at device 26.2 on pci0
uhci2: [ITHREAD]
usbus2: <UHCI (generic) USB controller> on uhci2
ehci0: <Intel PCH USB 2.0 controller USB-B> mem 0xfb7f8000-0xfb7f83ff irq 16 at device 26.7 on pci0
ehci0: [ITHREAD]
usbus3: EHCI version 1.0
usbus3: <Intel PCH USB 2.0 controller USB-B> on ehci0
pci0: <multimedia, HDA> at device 27.0 (no driver attached)
pcib1: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pci1: <network> at device 0.0 (no driver attached)
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.1 on pci0
pci2: <ACPI PCI bus> on pcib2
ahci0: <AHCI SATA controller> port 0xdc00-0xdc07,0xd880-0xd883,0xd800-0xd807,0xd480-0xd483,0xd400-0xd40f mem 0xfbdfe000-0xfbdfffff irq 17 at device 0.0 on pci2
ahci0: [ITHREAD]
ahci0: AHCI v1.00 with 1 3Gbps ports, Port Multiplier supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich0: [ITHREAD]
pcib3: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci3: <ACPI PCI bus> on pcib3
re0: <RealTek 8168/8111 B/C/CP/D/DP/E PCIe Gigabit Ethernet> port 0xe800-0xe8ff mem 0xfafff000-0xfaffffff,0xfaff8000-0xfaffbfff irq 18 at device 0.0 on pci3
re0: Using 1 MSI-X message
re0: Chip rev. 0x28000000
re0: MAC rev. 0x00000000
miibus0: <MII bus> on re0
rgephy0: <RTL8169S/8110S/8211B media interface> PHY 1 on miibus0
rgephy0: 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
re0: Ethernet address: 00:01:2e:2f:dc:dd
re0: [ITHREAD]
pcib4: <ACPI PCI-PCI bridge> irq 19 at device 28.3 on pci0
pci4: <ACPI PCI bus> on pcib4
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xfbffe000-0xfbffffff irq 19 at device 0.0 on pci4
xhci0: [ITHREAD]
usbus4 on xhci0
uhci3: <UHCI (generic) USB controller> port 0xc400-0xc41f irq 19 at device 29.0 on pci0
uhci3: [ITHREAD]
usbus5: <UHCI (generic) USB controller> on uhci3
uhci4: <UHCI (generic) USB controller> port 0xc080-0xc09f irq 16 at device 29.1 on pci0
uhci4: [ITHREAD]
usbus6: <UHCI (generic) USB controller> on uhci4
uhci5: <UHCI (generic) USB controller> port 0xc000-0xc01f irq 18 at device 29.2 on pci0
uhci5: [ITHREAD]
usbus7: <UHCI (generic) USB controller> on uhci5
ehci1: <Intel PCH USB 2.0 controller USB-A> mem 0xfb7f2000-0xfb7f23ff irq 23 at device 29.7 on pci0
ehci1: [ITHREAD]
usbus8: EHCI version 1.0
usbus8: <Intel PCH USB 2.0 controller USB-A> on ehci1
pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci5: <ACPI PCI bus> on pcib5
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel 5 Series/3400 Series PCH SATA300 controller> port 0xbc00-0xbc07,0xb880-0xb883,0xb800-0xb807,0xb480-0xb483,0xb400-0xb40f,0xb080-0xb08f irq 19 at device 31.2 on pci0
atapci0: [ITHREAD]
ata2: <ATA channel 0> on atapci0
ata2: [ITHREAD]
ata3: <ATA channel 1> on atapci0
ata3: [ITHREAD]
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
atapci1: <Intel 5 Series/3400 Series PCH SATA300 controller> port 0xac00-0xac07,0xa880-0xa883,0xa800-0xa807,0xa480-0xa483,0xa400-0xa40f,0xa080-0xa08f irq 19 at device 31.5 on pci0
atapci1: [ITHREAD]
ata4: <ATA channel 0> on atapci1
ata4: [ITHREAD]
ata5: <ATA channel 1> on atapci1
ata5: [ITHREAD]
acpi_button0: <Power Button> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: [FILTER]
acpi_hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 900
orm0: <ISA Option ROMs> at iomem 0xcd000-0xcefff,0xcf000-0xcffff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
atkbd0: [ITHREAD]
fdc0: <Enhanced floppy controller> at port 0x3f0-0x3f5,0x3f7 irq 6 drq 2 on isa0
fdc0: [FILTER]
ppc0: cannot reserve I/O port range
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
coretemp0: Tj(target) value 105 does not seem right.
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 17
device_attach: est0 attach returned 6
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
coretemp1: Tj(target) value 105 does not seem right.
est1: <Enhanced SpeedStep Frequency Control> on cpu1
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 17
device_attach: est1 attach returned 6
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
coretemp2: Tj(target) value 105 does not seem right.
est2: <Enhanced SpeedStep Frequency Control> on cpu2
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 17
device_attach: est2 attach returned 6
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
coretemp3: Tj(target) value 105 does not seem right.
est3: <Enhanced SpeedStep Frequency Control> on cpu3
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 17
device_attach: est3 attach returned 6
p4tcc3: <CPU Frequency Thermal Control> on cpu3
Timecounters tick every 1.000 msec
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 4.8Gbps Super Speed USB v3.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 12Mbps Full Speed USB v1.0
usbus8: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen2.1: <Intel> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen3.1: <Intel> at usbus3
uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen4.1: <0x1033> at usbus4
uhub4: <0x1033 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus4
ugen5.1: <Intel> at usbus5
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen6.1: <Intel> at usbus6
uhub6: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
ugen7.1: <Intel> at usbus7
uhub7: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus7
ugen8.1: <Intel> at usbus8
uhub8: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus8
uhub4: 4 ports with 4 removable, self powered
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub6: 2 ports with 2 removable, self powered
uhub7: 2 ports with 2 removable, self powered
uhub3: 6 ports with 6 removable, self powered
uhub8: 8 ports with 8 removable, self powered
ugen3.2: <Generic> at usbus3
umass0: <Generic USB2.0-CRW, class 0/0, rev 2.00/19.81, addr 2> on usbus3
(probe0:umass-sim0:0:0:0): TEST UNIT READY. CDB: 0 0 0 0 0 0
(probe0:umass-sim0:0:0:0): CAM status: SCSI Status Error
(probe0:umass-sim0:0:0:0): SCSI status: Check Condition
(probe0:umass-sim0:0:0:0): SCSI sense: UNIT ATTENTION asc:28,0 (Not ready to ready change, medium may have changed)
ada0 at ata2 bus 0 scbus1 target 0 lun 0
ada0: <Hitachi HDS5C3030ALA630 MEAOA580> ATA-8 SATA 3.x deviceda0 at umass-sim0 bus 0 scbus5 target 0 lun 0
da0: <Generic- SD/MMC 1.00> Removable Direct Access SCSI-0 device
da0: 40.000MB/s transfers
da0: 3837MB (7858176 512 byte sectors: 255H 63S/T 489C)

ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada0: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada1 at ata2 bus 0 scbus1 target 1 lun 0
ada1: <Hitachi HDS5C3030ALA630 MEAOA580> ATA-8 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada1: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada2 at ata3 bus 0 scbus2 target 0 lun 0
ada2: <Hitachi HDS5C3030ALA630 MEAOA580> ATA-8 SATA 3.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada2: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada3 at ata3 bus 0 scbus2 target 1 lun 0
ada3: <Hitachi HDS5C3030ALA630 MEAOA580> ATA-8 SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada3: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada4 at ata4 bus 0 scbus3 target 0 lun 0
ada4: <Hitachi HDS5C3030ALA630 MEAOA580> ATA-8 SATA 3.x device
ada4: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada4: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada5 at ata5 bus 0 scbus4 target 0 lun 0
ada5: <Hitachi HDS5C3030ALA630 MEAOA580> ATA-8 SATA 3.x device
ada5: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada5: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
GEOM: da0s1: geometry does not match label (16h,63s != 255h,63s).
Trying to mount root from ufs:/dev/ufs/FreeNASs1a
ZFS filesystem version 4
ZFS storage pool version 15
re0: link state changed to DOWN
re0: link state changed to UP
 
Hmmm, that seems to have only stuff up to boot. Try the last 20 or so lines from /var/log/messages
 
Aug 30 10:19:43 Shag freenas: Starting sshd.
Aug 30 10:19:44 Shag freenas[1731]: Executing: /bin/pgrep -F /var/run/sshd.pid sshd
Aug 30 10:19:54 Shag freenas[1731]: Executing: /usr/sbin/service ix-sshd quietstart
Aug 30 10:19:54 Shag freenas[1731]: Executing: /usr/sbin/service sshd restart
Aug 30 10:19:54 Shag freenas: Stopping sshd.
Aug 30 10:19:54 Shag freenas: Starting sshd.
Aug 30 10:19:55 Shag freenas[1731]: Executing: /bin/pgrep -F /var/run/sshd.pid sshd
Aug 30 10:25:16 Shag freenas[1731]: Executing: /usr/sbin/service ix-sshd quietstart
Aug 30 10:25:16 Shag freenas[1731]: Executing: /usr/sbin/service sshd restart
Aug 30 10:25:16 Shag freenas: Stopping sshd.
Aug 30 10:25:16 Shag freenas: Starting sshd.
Aug 30 10:25:16 Shag freenas[1731]: Executing: /bin/pgrep -F /var/run/sshd.pid sshd
Aug 30 10:26:19 Shag freenas[1731]: Executing: /usr/sbin/service ix-sshd quietstart
Aug 30 10:26:19 Shag freenas[1731]: Executing: /usr/sbin/service sshd restart
Aug 30 10:26:19 Shag freenas: Stopping sshd.
Aug 30 10:26:19 Shag freenas: Starting sshd.
Aug 30 10:26:20 Shag freenas[1731]: Executing: /bin/pgrep -F /var/run/sshd.pid sshd

Does that work? the lines before that looked like some sort of PKI authentication stuff.
 
Right now I am experiencing about 15 second latency opening a folder, or doing simple stuff like that. Same for opening a video file, once open it seems to play fine, but seeking causes a similar delay.
 
So it's doing that even as you were cut&pasting the log output? If so, this is weird. I was hoping it was something being complained about in the system log, but apparently not. I wish I had a helpful idea, sorry :(
 
And now we are back to completely normal operation, so the period of observed slowness lasted about 4 hours, but could have been happening in the middle of the night.
 
yes, accessing it through the windows file system, i was having 15+ seconds of latency, while the web interface and SSH were both instantaneous.

I have noticed however, that if the timeouts or latency was very heavy, it would take a few seconds to load the storage page in the webgui. Even then, it would tell me that everything was healthy.

I remember reading something about freenas default memory allocation sometimes causing trouble? cant remember where i read it though, it has been about a year.
 
Check [fsck] the health of the USB flash drive where FreeNAS is installed.

BTW
Is it at USB v.3 port [usbus4: 4.8Gbps Super Speed USB v3.0] ?
 
Hmm good point. I'd forgotten this is running on a USB stick - possibly getting delays that are not fatal (and therefore not causing log messages...)
 
The port should be USB 3.0 (Blue is 3.0 right?), though I am pretty sure the USB fab I am running it all off of is 2.0.

Also I haven't done command line *nix stuff in about 10 years, and am already at the limits of my knowledge. How would I run an FSK on the boot drive?
 
Reboot in single user mode - via startup options or "shutdown" in console [must be super user- su ,ENTER ,root password].
Then simply enter fsck -it will check all partitions at the flash drive,but not ZFS file system.
Also I'd put the said flash drive at USB v2.0 port.

You can start with fsck w/o rebooting in single user,but the utility will be not able to recover the errors.Try that first.
 
You mentioned PKI stuff in the logs. Is there a windows domain running here? Double-check that the time is set correctly on all your workstations and the server. And this also includes the timezone. it's ok for machine to be on different timezones (but that would be odd within the same location) but they do have to be all using the same basic time. Best is to set them all to use an NTP network time server.

this is because the underlying authentication mechanisms totally depend on accurate and sync'd time between server and workstations. If the time is off by more than a minute things start timing out, retrying and the like. So just check that all the machines are using the same time consistently.
 
wkearney99 said:
You mentioned PKI stuff in the logs. Is there a windows domain running here? Double-check that the time is set correctly on all your workstations and the server. And this also includes the timezone. it's ok for machine to be on different timezones (but that would be odd within the same location) but they do have to be all using the same basic time. Best is to set them all to use an NTP network time server.

this is because the underlying authentication mechanisms totally depend on accurate and sync'd time between server and workstations. If the time is off by more than a minute things start timing out, retrying and the like. So just check that all the machines are using the same time consistently.
Click to expand...

How can I set freenas to look for a network time server?
 
You must log in or register to reply here.
Back
Top