Fun with Fans

I’ve been running a Dell PowerConnect 5324 as my secondary switch at home that connects all the rooms together. The switch is rather loud in the location it’s sitting so I’ve decided to replace the fans in it to something quieter.

The switch uses 2 40mm high speed fans that exhaust air out of the enclosure. I had several 40mm fans that I could try to get the best noise/cooling ratio. I’ve used a digital multimeter with a thermal probe to measure the efficiency of the fans, my simple SPL meter to measure noise level and a simple Android app to measure the sound spectrum. The switch was ran for 30 minutes in each configuration to let the temperature of the switch settle.

Testing Methodology:
Ambient Noise: 23.3 dbA
Ambient Temperature: 23C

Configuration Temperature Sound Level
Stock Fan 26.2C 59.2 dBA

In the stock configuration the switch blasts out almost 64 dbA.

What would happen if I invert the fans so that they blow air into the switch?

Configuration Temperature Sound Level
Inverted Stock Fan 26.5C 63.7 dbA

Wow. Definitely not an improvement. Not only this configuration is slightly louder. The fans emit a noise 1.5Khz range that’s very, very annoying.

The new fans are significantly thinner and operate at lower RPMs. They do not move quite as much air as the original fans.
A quick note about the fan pinout on the Dell PowerConnect 5324. The fan pins are actually non-standard and have to be swapped around. The Dell PowerConnect fans use Positive-Sense-Negative pinout as opposed to the standard Negative-Positive-Sense pinout. Connecting the fans incorrectly doesn’t seem to have any detrimental effect on the fans themselves (as I found out) but the fans will simply not operate.

Once the fans were installed. I ran the test.

Configuration Temperature Sound Level
New Fan 27.4C 50.9 dbA

A 9.1 dbA drop is quite significant, that’s almost half as noisy as the original fans and the switch is now quieter than the ambient noise in the room the networking equipment is in. The temperature with the new fans running only went up a few notches.

Unfortunately the switch did not like the reduced RPM on the fans. The Fan LED now blinks alternating red/green. However there doesn’t seem to be any performance degradation of the switch.

Even though this was the configuration I was most likely stick with, just for kicks, I’ve went ahead and tried the inverted fan configuration.

Configuration Temperature Sound Level
Inverted New Fan 27.9C 46.5 dbA

Even though in the inverted configuration the fans were quieter, the temperature went up a few more degrees and once again, the fans exhibited a noticeable whine 700hz range.

In the end I ended up going with the smaller fans in the same configuration as the stock fans, blowing air out of the switch. While slightly louder than the inverted configuration, it was lacking the whine which was quite noticable.

Watchguard x500 Hacking – Part 3 – ZeroShell

Another day, another opportunity to see what other firewall distros can be deployed on this old Watchguard Firebox x500. In this case I’ll try ZeroShell. I’ve used ZeroShell many times in the past, typically as a small VM. ZeroShell is one of the fastest and easiest Firewall Distros I’ve tried. Back in the day when I was trying to see if I can bond multiple cable modems together for site-to-site connections I’ve used ZeroShell due to it’s very easy bonding of OpenVPN connections.

Another reason why I’m eager to try ZeroShell on this Firebox is the fact that unlike pfSense and m0n0wall, it’s not BSD based OS. It’s actually linux based which means that there’s a very good possibility that the Watchdog Timeout issue might not happen as this seems to be a driver issue in BSD related to the Realtek NICs on the x500.

On to the install process.

Just like earlier first step is to simply load the firmware image onto the compact flash card. To do that once again need to clean the compact flash card from any existing partitions

Microsoft Windows [Version 6.2.9200]
(c) 2012 Microsoft Corporation. All rights reserved.
 
C:\Windows\System32>diskpart
 
Microsoft DiskPart version 6.2.9200
 
Copyright (C) 1999-2012 Microsoft Corporation.
On computer: ARES
 
DISKPART> list disk
 
  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online          238 GB      0 B
  Disk 1    Online           74 GB      0 B   *
  Disk 2    Online           74 GB      0 B   *
  Disk 3    Online           74 GB      0 B   *
  Disk 4    Online         2048 MB      0 B
  Disk 5    Offline        1024 GB      0 B        *
  Disk 6    Online          500 GB  1024 KB   *
  Disk 7    Online         1024 GB      0 B        *
 
DISKPART> select disk 4
 
Disk 4 is now the selected disk.
 
DISKPART> clean
 
DiskPart succeeded in cleaning the disk.
 
DISKPART> exit
 
Leaving DiskPart...
 
C:\Windows\System32>

Then use physdiskwrite to write the image onto the card. In this case the image being loaded is the ALIX image that can be downloaded here. The latest image (RC1) requires a minimum 2GB Compact Flash card to write the image to. Luckily I still have a few of those laying around.

Microsoft Windows [Version 6.2.9200]
(c) 2012 Microsoft Corporation. All rights reserved.
 
c:\2>physdiskwrite.exe ZeroShell-2.0.RC1-Alix-2GB.img
 
physdiskwrite v0.5.2 by Manuel Kasper <mk@neon1.net>
Which disk do you want to write? (0..7) 4
About to overwrite the contents of disk 4 with new data. Proceed? (y/n) y
 
c:\2>

Once the compact flash card is installed into the firewall, connect the serial cable and use a terminal program at 38400-8-n-1 to watch the bootup process. During my bootup there seemed to be some errors at runtime that actually took a few seconds longer to get going.

ÿ[    1.004441] platform pc8736x_gpio.0: no device found
[   31.842568] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[   31.863676] ata1.00: failed command: READ DMA
[   31.876728] ata1.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[   31.876732]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[   31.920494] ata1.00: status: { DRDY }
[   62.960333] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[   62.961451] ata1.00: failed command: READ DMA
[   62.966530] ata1.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[   62.966534]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[   62.970295] ata1.00: status: { DRDY }
[   94.002083] ata1.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen
[   94.003202] ata1.00: failed command: READ DMA
[   94.008282] ata1.00: cmd c8/00:08:00:00:00/00:00:00:00:00/e0 tag 0 dma 4096 in
[   94.008285]          res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)
[   94.012049] ata1.00: status: { DRDY }
Loading Zeroshell ZS-2.0.RC1 ...
DEVICE=/dev/sda2
Mounting ISO image  ...
mount: warning: /cdrom seems to be mounted read-only.
Loading root filesystem into RAM device... Success
mount: warning: /.root/cdrom seems to be mounted read-only.
Successfully mounted device ISO
INIT: version 2.85 booting
INIT: Entering runlevel: 3nterface...
[  OK  ] udevd daemon...
[  OK  ]ing for attached devices...
Checking for other PCI hardware ...
Loading  ...................................   [pata_acpi]
Loading  ...................................   [intel-rng]
Scanning for SCSI,SATA,IDE,USB storage devices...
--------------------------------------------------------------------
PROFILE   : Default Profile
Disk      : ATA       SanDiskSDCFH-20
Partition : sda3
Alias     : _DB.001
--------------------------------------------------------------------
[  OK  ]Time Zone  [Europe/Rome]
[  OK  ] Clock (LOCALTIME) --> System Time
[  OK  ]hostname to zeroshell.example.com
[  OK  ] configuration files...
[  OK  ]ng swap file...
Starting X.509 Certification Authority...
Generating zeroshell.example.com host certificate ...
Generating a 2048 bit RSA private key
...........................................+++
............+++
writing new private key to '/tmp/x509default.key'
-----
Using configuration from /etc/ssl/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 3 (0x3)
        Validity
            Not Before: Nov  9 22:43:38 2012 GMT
            Not After : Nov  9 22:43:38 2014 GMT
        Subject:
            organizationalUnitName    = Hosts
            commonName                = zeroshell.example.com
        X509v3 extensions:
            X509v3 Extended Key Usage:
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Subject Alternative Name:
                DNS:zeroshell.example.com, IP Address:192.168.250.254, IP Address:192.168.0.75
Certificate is to be certified until Nov  9 22:43:38 2014 GMT (730 days)
 
Write out database with 1 new entries
Data Base Updated
Generating admin user certificate ...
Generating a 2048 bit RSA private key
.........................................................+++
....+++
writing new private key to '/tmp/x509default.key'
-----
Using configuration from /etc/ssl/openssl.cnf
Check that the request matches the signature
Signature ok
Certificate Details:
        Serial Number: 4 (0x4)
        Validity
            Not Before: Nov  9 22:43:40 2012 GMT
            Not After : Nov  9 22:43:40 2014 GMT
        Subject:
            organizationalUnitName    = users
            commonName                = admin
        X509v3 extensions:
            X509v3 Key Usage:
                Digital Signature, Key Encipherment, Data Encipherment
            X509v3 Extended Key Usage:
                TLS Web Client Authentication, E-mail Protection
Certificate is to be certified until Nov  9 22:43:40 2014 GMT (730 days)
 
Write out database with 1 new entries
[  OK  ]e Updated
[  OK  ] LDAP daemon...
[  OK  ] DNS service...
[  OK  ] system log daemon...
[  OK  ] kernel log daemon...
[  OK  ]connection tracking modules (h323,ftp,sip,irc,pptp,tftp)
[  OK  ]NAT tracking modules (ftp,pptp)
[  OK  ]g Layer 7 protocol definitions (l7-protocols-2009-05-28)
Starting Firewall...
Starting Captive Portal ...
--> Gateway disabled
--> Web Login Authentication Server disabled
Starting Network...
Starting WiFi subsystem ...
--> No supported Wi-Fi hardware has been found.
Detecting ethernet interfaces...
ETH00 (hardware changed) :  Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20)
ETH01 (hardware changed) :  Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20)
ETH02 (new) :  Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20)
ETH03 (new) :  Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20)
ETH04 (new) :  Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20)
ETH05 (new) :  Realtek Semiconductor Co., Ltd. RTL-8139/8139C/8139C+ (rev 20)
Configuring interfaces...
ETH00: 192.168.0.75/255.255.255.0
VPN99: 192.168.250.254/255.255.255.0
Starting Routing...
Starting Quality of Service on:
   NONE. No interfaces configured for QoS
[  OK  ] NTP daemon...
[  OK  ] Dynamic DNS client daemon...
[  OK  ] Log Watcher...
Starting Kerberos 5 KDC
[  OK  ]istribution Center process
[  OK  ]n administration process
[  OK  ] httpd daemon...
Checking HTTP Transparent Proxy and AntiVirus configuration...
[  OK  ] cron daemon ...
Starting MRTG ...
[  OK  ]ing MRTG ...
[  OK  ] Daemon Watcher ...
[  OK  ] AutoUpdate daemon...
Zeroshell
[  OK  ] caching background process ...
-------------------------------------------------------------------------------
 Z e r o S h e l l - Net Services  2.0.RC1          November 09, 2012 - 23:43
-------------------------------------------------------------------------------
  Hostname : zeroshell.example.com
  CPU (1)  : Intel(R) Celeron(TM) CPU                1200MHz  1202MHz
  Kernel   : 3.4.6-ZS
  Memory   : 512388 kB                          http://192.168.0.75
  Uptime   : 0 days, 0:2                        User     : admin
  Load     : 2.12 0.89 0.33                     Password : zeroshell
  Profile  : Default Profile
-------------------------------------------------------------------------------
 COMMAND MENU
  <A> Activate Profile              <P> Change admin password
  <D> Deactivate Profile            <T> Show Routing Table
  <S> Shell Prompt                  <F> Show Firewall Rules
  <R> Reboot                        <N> Show Network Interface
  <H> Shutdown                      <Z> Fail-Safe Mode
  <B> Create a Bridge               <I> IP Manager
  <W> WiFi Manager
 
 
                                                Select:

Now that the firewall is up. I configured the WAN and LAN interfaces via the shell. By default the DHCP server on LAN side is not enabled so in order to access the firewall via the browser I had to set a static IP address on my machine. Once the IP has been configured, just launch a the browser and point to the address as displayed in the console (default http://192.168.0.75)

I also disovered that by default the WAN interface allows access to the ZeroShell interface also, and since the firewall WAN is actually on my LAN, I was able to access and configure the UI from my workstation. This also means that it’s very important to change the default password if the firewall is internet facing as anyone coming across it can reconfigure it.

Running some unencrypted performance tests. I was able to achieve 11.9MB/s (95.2 Mbit/s) throughput across the firewall. This is actually not bad considering the same test on this box running pfSense the throughput was 11.1MB/s.

I came across an interesting article on the ZeroShell forums about the HTTP Anti-Virus Proxy and Compact Flash cards. Specifically about the HAVP’s work directory being used during its operation. To create a RAM drive to store these temporary files instead shut down HAVP and execute these commands from console or through SSH:

root@zeroshell root> cd /Database
root@zeroshell Database> dd if=/dev/zero of=HAVP.ext2 count=100000
100000+0 records in
100000+0 records out
 
root@zeroshell Database> mkfs.ext2 HAVP.ext2
mke2fs 1.42 (29-Nov-2011)
HAVP.ext2 is not a block special device.
Proceed anyway? (y,n) y
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
12544 inodes, 50000 blocks
2500 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=51380224
7 block groups
8192 blocks per group, 8192 fragments per group
1792 inodes per group
Superblock backups stored on blocks:
        8193, 24577, 40961
 
Allocating group tables: done
Writing inode tables: done
Writing superblocks and filesystem accounting information: done
 
root@zeroshell Database> mount -o loop HAVP.ext2 /mnt
root@zeroshell Database> chown havp.havp /mnt
root@zeroshell Database> umount /mnt
root@zeroshell Database> cat /Database/HAVP.ext2 > /dev/ram3
root@zeroshell Database> mount -omand,noatime /dev/ram3 /Database/var/register/system/havp/tmp
root@zeroshell Database>

The last two lines have to be added to the pre-boot script so that they execute on device restart.

Now just restart the HAVP service and it’s done.

After more than 24 hours of various traffic passing through the firewall, I have not had any issues yet with the Watchguard Timeout. So far so good. The firewall performs pretty well. There’s probably no chance getting the LCD working easily at this point. Though there’s a small possibility that ZeroShell will at one point support LCDProc in which case the LCD can live again.

I have also since added few more remounts to ensure longer CF card life. Apparently just because the ZeroShell distro image is aimed at embedded devices, it still performs regular writes to the local storage. Since CF cards have limited write cycles, remounting the writeable locations in RAM drive should significantly extend the life of the Compact Flash card.

mount -t tmpfs -o size=64m,mode=1777,nosuid,nodev,exec tmpfs /tmp
mount -t tmpfs -o size=16m,mode=755,nosuid,nodev tmpfs /var/run
mount -t tmpfs -o size=16m,mode=755,nosuid,nodev tmpfs /var/lock
mount -t tmpfs -o size=64m,mode=755,nosuid,nodev tmpfs /Database/LOG
mount -t tmpfs -o size=16m,mode=755,nosuid,nodev tmpfs /Database/var/register/system/mrtg/counters
mount -t tmpfs -o size=32m,mode=755,nosuid,nodev tmpfs /Database/var/register/system/mrtg/html

Watchguard x500 Hacking – Part 2 – m0n0wall

When I originally picked up the Watchguard Firebox x500, I only had the intention of trying pfSense on it. But now that I have pfSense running succesfully on the next generation Watchguard Fireboxes (x550e, x750e and 1250e). I figured might as well try m0n0wall on it (and some other OSs later). One particular reason why I’m interested in trying other firewall software on the Watchguard is that there’s a known problem with the Realtek NIC’s on the x500 and the current pfSense version (2.0). The firewall will randomly issue “watchdog timeout” error and then simply stop responding to traffic. Rebooting the firewall seems to be the only way to get it moving again.

From a little bit of research, getting m0n0wall running on x500 is just as trivial as getting pfSense running. As an added bonus, the m0n0wall image is much smaller than the pfSense image and the original Compact Flash card can be used. The embedded image for version 1.33 comes in at only 7.6 MB.

Loading the m0n0wall image onto the compact flash is identical to pfSense.

Step 1. Clean the CF Card

 
C:\>diskpart
 
Microsoft DiskPart version 6.2.9200
 
Copyright (C) 1999-2012 Microsoft Corporation.
On computer: ARES
 
DISKPART> list disk
 
  Disk ###  Status         Size     Free     Dyn  Gpt
  --------  -------------  -------  -------  ---  ---
  Disk 0    Online          238 GB      0 B
  Disk 1    Online           74 GB      0 B   *
  Disk 2    Online           74 GB      0 B   *
  Disk 3    Online           74 GB      0 B   *
  Disk 4    Online          122 MB      0 B
  Disk 5    No Media           0 B      0 B
  Disk 6    No Media           0 B      0 B
  Disk 7    No Media           0 B      0 B
  Disk 8    Offline        1024 GB      0 B        *
  Disk 9    Online          500 GB  1024 KB   *
  Disk 10   Online         1024 GB      0 B        *
 
DISKPART> select disk 4
 
Disk 4 is now the selected disk.
 
DISKPART> clean
 
DiskPart succeeded in cleaning the disk.
 
DISKPART> exit
 
Leaving DiskPart...
 
C:\>

Step 2. Load m0n0wall image onto the card
Once again I used physdiskwrite + PhysGUI to load the image onto the card.

Select the proper disk

The disk image is being written to the CF card

And we’re done.

Now just a matter of taking the Firebox cover off and plugging in the new CF card. Once that’s done. Connect a serial cable to the console at 115200-8-n-1 and watch the boot process.

Step 3. Configure m0n0wall.

Copyright (c) 1992-2008 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 6.4-RELEASE-p5 #0: Sun Jan  9 22:24:57 CET 2011
    root@mb64.neon1.net:/usr/src/sys/i386/compile/M0N0WALL_EMBEDDED
Timecounter "i8254" frequency 1193182 Hz quality 0
CPU: Intel(R) Celeron(TM) CPU                1200MHz (1202.73-MHz 686-class CPU)
  Origin = "GenuineIntel"  Id = 0x6b4  Stepping = 4
  Features=0x383f9ff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,MMX,FXSR,SSE>
real memory  = 536870912 (512 MB)
avail memory = 499331072 (476 MB)
wlan: mac acl policy registered
ath_hal: 0.9.20.3 (AR5210, AR5211, AR5212, RF5111, RF5112, RF2413, RF5413)
cpu0 on motherboard
pcib0: <Intel 82815 (i815 GMCH) Host To Hub bridge> pcibus 0 on motherboard
pir0: <PCI Interrupt Routing Table: 11 Entries> on motherboard
$PIR: Using invalid BIOS IRQ 9 from 2.13.INTA for link 0x63
pci0: <PCI bus> on pcib0
pcib1: <PCIBIOS PCI-PCI bridge> at device 30.0 on pci0
pci2: <PCI bus> on pcib1
re0: <RealTek 8139C+ 10/100BaseTX> port 0xd500-0xd5ff mem 0xefefa000-0xefefa1ff irq 10 at device 9.0 on pci2
miibus0: <MII bus> on re0
rlphy0: <RealTek internal media interface> on miibus0
rlphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
re0: Ethernet address: 00:90:7f:31:cc:60
re0: [FAST]
re1: <RealTek 8139C+ 10/100BaseTX> port 0xd600-0xd6ff mem 0xefefb000-0xefefb1ff irq 5 at device 10.0 on pci2
miibus1: <MII bus> on re1
rlphy1: <RealTek internal media interface> on miibus1
rlphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
re1: Ethernet address: 00:90:7f:31:cc:61
re1: [FAST]
re2: <RealTek 8139C+ 10/100BaseTX> port 0xd900-0xd9ff mem 0xefefc000-0xefefc1ff irq 11 at device 11.0 on pci2
miibus2: <MII bus> on re2
rlphy2: <RealTek internal media interface> on miibus2
rlphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
re2: Ethernet address: 00:90:7f:31:cc:62
re2: [FAST]
re3: <RealTek 8139C+ 10/100BaseTX> port 0xda00-0xdaff mem 0xefefd000-0xefefd1ff irq 12 at device 12.0 on pci2
miibus3: <MII bus> on re3
rlphy3: <RealTek internal media interface> on miibus3
rlphy3:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
re3: Ethernet address: 00:90:7f:31:cc:63
re3: [FAST]
re4: <RealTek 8139C+ 10/100BaseTX> port 0xdd00-0xddff mem 0xefefe000-0xefefe1ff irq 9 at device 13.0 on pci2
miibus4: <MII bus> on re4
rlphy4: <RealTek internal media interface> on miibus4
rlphy4:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
re4: Ethernet address: 00:90:7f:31:cc:64
re4: [FAST]
re5: <RealTek 8139C+ 10/100BaseTX> port 0xde00-0xdeff mem 0xefeff000-0xefeff1ff irq 6 at device 14.0 on pci2
miibus5: <MII bus> on re5
rlphy5: <RealTek internal media interface> on miibus5
rlphy5:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, auto
re5: Ethernet address: 00:90:7f:31:cc:65
re5: [FAST]
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
atapci0: <Intel ICH2 UDMA100 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xff00-0xff0f at device 31.1 on pci0
ata0: <ATA channel 0> on atapci0
ata1: <ATA channel 1> on atapci0
pmtimer0 on isa0
orm0: <ISA Option ROM> at iomem 0xe0000-0xe0fff on isa0
sio0 at port 0x3f8-0x3ff irq 4 flags 0x10 on isa0
sio0: type 16550A, console
sio1: configured irq 3 not in bitmap of probed irqs 0
sio1: port may not be enabled
unknown: <PNP0c01> can't assign resources (memory)
unknown: <PNP0501> can't assign resources (port)
RTC BIOS diagnostic error 20<config_unit>
Timecounter "TSC" frequency 1202733613 Hz quality 800
Timecounters tick every 1.000 msec
Fast IPsec: Initialized Security Association Processing.
IP Filter: v4.1.33 initialized.  Default = block all, Logging = enabled
md0: Preloaded image </mfsroot> 16777216 bytes at 0xc086b0e8
ad0: 122MB <SanDisk SDCFJ-128 HDX 4.09> at ata0-master PIO4
Trying to mount root from ufs:/dev/md0
kern.coredump: 1 -> 0
Found configuration on ad0.
re0: link state changed to DOWN
re1: link state changed to DOWN
re2: link state changed to DOWN
re3: link state changed to DOWN
re4: link state changed to DOWN
re5: link state changed to DOWN
Initializing timezone... done
Configuring firewall... done
Configuring LAN interface... done
Configuring WAN interface... done
Starting syslog service... done
Starting webGUI... done
Starting DNS forwarder... done
Starting DHCP service... done
Starting NTP client... done

There you have it, m0n0wall running on the x500. Now it’s just a matter of configuring the interfaces for WAN and LAN. In this case the “External” interface was used as WAN (re0) and the first interface as LAN (re1)

 
*** This is m0n0wall, version 1.33
    built on Wed Mar 16 12:01:42 CET 2011 for embedded
    Copyright (C) 2002-2011 by Manuel Kasper. All rights reserved.
    Visit http://m0n0.ch/wall for updates.
 
 
    LAN IP address: 192.168.1.1
 
    Port configuration:
 
    LAN   -> sis0
    WAN   -> sis1
 
 
m0n0wall console setup
**********************
1) Interfaces: assign network ports
2) Set up LAN IP address
3) Reset webGUI password
4) Reset to factory defaults
5) Reboot system
6) Ping host
 
Enter a number: 1
 
Valid interfaces are:
 
re0     00:90:7f:31:cc:60   (up)   RealTek 8139C+ 10/100BaseTX
re1     00:90:7f:31:cc:61   (up)   RealTek 8139C+ 10/100BaseTX
re2     00:90:7f:31:cc:62          RealTek 8139C+ 10/100BaseTX
re3     00:90:7f:31:cc:63          RealTek 8139C+ 10/100BaseTX
re4     00:90:7f:31:cc:64          RealTek 8139C+ 10/100BaseTX
re5     00:90:7f:31:cc:65          RealTek 8139C+ 10/100BaseTX
 
Do you want to set up VLANs first?
If you're not going to use VLANs, or only for optional interfaces, you
should say no here and use the webGUI to configure VLANs later, if required.
 
Do you want to set up VLANs now? (y/n) n
 
If you don't know the names of your interfaces, you may choose to use
auto-detection. In that case, disconnect all interfaces before you begin,
and reconnect each one when prompted to do so.
 
Enter the LAN interface name or 'a' for auto-detection: re1
 
Enter the WAN interface name or 'a' for auto-detection: re0
 
Enter the Optional 1 interface name or 'a' for auto-detection
(or nothing if finished):
 
The interfaces will be assigned as follows:
 
LAN  -> re1
WAN  -> re0
 
The firewall will reboot after saving the changes.
 
Do you want to proceed? (y/n) y
 
The firewall is rebooting now.
Waiting (max 60 seconds) for system process `vnlru' to stop...done
Waiting (max 60 seconds) for system process `bufdaemon' to stop...done
Waiting (max 60 seconds) for system process `syncer' to stop...
Syncing disks, vnodes remaining...0 done
All buffers synced.
Uptime: 36s
Rebooting...

A few moments later the web UI was fully accessible and ready to be configured.

Unfortunately just a few moments later, while digging around the Web GUI the console showed the dreaded watchdog timeout error. And all network communication with the firewall and beyond has stopped.

*** This is m0n0wall, version 1.33
    built on Wed Mar 16 12:01:42 CET 2011 for embedded
    Copyright (C) 2002-2011 by Manuel Kasper. All rights reserved.
    Visit http://m0n0.ch/wall for updates.
 
 
    LAN IP address: 192.168.1.1
 
    Port configuration:
 
    LAN   -> re1
    WAN   -> re0
 
 
m0n0wall console setup
**********************
1) Interfaces: assign network ports
2) Set up LAN IP address
3) Reset webGUI password
4) Reset to factory defaults
5) Reboot system
6) Ping host
 
Enter a number: re1: watchdog timeout
re1: watchdog timeout
re1: watchdog timeout

Well, so much for that idea..on to the next firewall….ZeroShell?

WatchGuard – pfSense – Tweaks

Continued work on improving pfSense running on my Watchguard x550e/x750e/x1250e firewalls. I got the x750e firewall nicely mounted at the utility board where my internet connections arrive at home.

Though I ran into an issue mounting the firewall due to its depth. The standard bracket was not long enough to fit the firewall with the power cable protruding out the back. I ended up picking up a 90 degree cable that just made it fit.

Had the firewall running for a while now and during this time I’ve worked on it a bit more. There’s a known issue with the MSK interfaces timing out under pfSense 2.0. I’ve experienced MSK failure twice since installing 2.0. I’ve since upgraded to 2.1 Beta and so far it’s been stable. Was pretty happy about the fact that all I had to do to upgrade to 2.1 was to backup the config from 2.0 and simply restore it on 2.1 once I wrote out the new 2.1 image to the compact flash card.

In the meantime I also implemented a few more tweaks to all the firewalls.

Throttle down CPU
Enabled PowerD in System->Advanced->Miscellaneous. This however caused a flood of errors in the log and console when the system was attempting to throttle down the CPU.

kernel: timecounter TSC must not be in use when changing frequencies; change denied
kernel: timecounter TSC must not be in use when changing frequencies; change denied
kernel: timecounter TSC must not be in use when changing frequencies; change denied
kernel: timecounter TSC must not be in use when changing frequencies; change denied

This was easily fixed via a new tunable under System->Advanced->System Tunables.
Added a new tunable.
Tunnable Name: kern.timecounter.hardware
Value: i8254
Then rebooted the firewall.

Throttle Fans / Change Armed LED
Another great tweak was the Fan Throttle mod. The firewall is fairly loud with the fans running at 100%. This can be resolved thanks to the people on the pfSense forums. The program to control the Watchguard fans (and LED) is called WGXepc

Simply upload the file to the firewall. I used the File Manager package to upload the file to /tmp. One word of warning, by default the file system on the nanobsd build is set to read only. It has to be made writable by executing:

[2.1-BETA0][admin@aura.olympia.local]/tmp(4): /etc/rc.conf_mount_rw
[2.1-BETA0][admin@aura.olympia.local]/tmp(5):

One the file has been uploaded to /tmp

[2.1-BETA0][admin@aura.olympia.local]/(7): cd /tmp
[2.1-BETA0][admin@aura.olympia.local]/tmp(8): gunzip WGXepc.gz
[2.1-BETA0][admin@aura.olympia.local]/tmp(9): copy WGXepc /home
[2.1-BETA0][admin@aura.olympia.local]/tmp(10): cd /home
[2.1-BETA0][admin@aura.olympia.local]/home(11): chmod +x WGXepc

To add the automatic fan throttle to bootup process execute the following script. The value can be anywhere between 00 and FF (hex 0-255).

[2.1-BETA0][admin@aura.olympia.local]/home(12): echo "/home/WGXepc -f 30" >> /etc/rc.local

Lastly it would be nice to change the Armed LED to green when bootup is complete.

[2.1-BETA0][admin@aura.olympia.local]/home(13): echo "/home/WGXepc -l green" >> /etc/rc.local

Functional LCD
Also got the LCD working on the unit. This was actually quite simple simply install LCDProc and LCDproc-devel packages and configure as follows.

There is an issue currently with this as on reboot the processes do not correctly start in the proper order and cause the package to crash. The solution right now is to simply manually start the service once the firewall has completed booting.

vSphere + NexentaStor + iSCSI + MPIO + Jumbo Frames = ?

A while ago I build a new NexentaStor server to serve as the home lab SAN. Also picked up a low latency Force10 switch to handle the SAN traffic (among other things).
Now the time came to test vSphere iSCSI MPIO and attempt to achieve faster than 1Gb/s connection to the datastore which has been a huge bottleneck when using NFS.

The setup on each machine is as follows.

NexentaStor

  • 4 Intel Pro/1000 VT Interfaces
  • Each NIC on separate VLAN
  • Naggle disabled
  • Compression on Target enabled

vSphere

  • 4 On-Board Broadcom Gigabit interfaces
  • Each NIC on separate VLAN
  • Round Robin MPIO
  • Balancing: IOPS, 1 IO per NIC
  • Delayed Ack enabled
  • VM test disk Eager Thick Provisioned

Network on vSphere was configured via a single vSwitch though pNICs were assigned individually to each vNIC.

Round robin balancing was configured via vSphere and changed the IOPS per NIC via the console

~ # esxcli storage nmp psp roundrobin deviceconfig set --device naa.600144f083e14c0000005097ebdc0002 --iops 1 --type iops
~ # esxcli storage nmp psp roundrobin deviceconfig get -d naa.600144f083e14c0000005097ebdc0002
   Byte Limit: 10485760
   Device: naa.600144f083e14c0000005097ebdc0002
   IOOperation Limit: 5
   Limit Type: Iops
   Use Active Unoptimized Paths: false

Testing was done inside a CentOS VM because for some reason testing directly in vSphere Console only results in maximum transfer of 80MB/s even though the traffic was always split evenly across all 4 interfaces.

Testing was done via DD commands

[root@testvm test]# dd if=/dev/zero of=ddfile1 bs=16k count=1M
[root@testvm test]# dd if=ddfile1 of=/dev/null bs=16k

The initial test was done with what I thought was the ideal scenario.

NexentaStor MTU vSphere MTU VM Write VM Read
9000 9000 394 MB/s 7.4 MB/s

What the? 7.4 MB/s reads? Repeated the test several times to confirm. Even tried it on another vSphere server and new Test VM. Doing some Googling it might be MTU mismatch so let’s try with standard 1500 MTU.

NexentaStor MTU vSphere MTU VM Write VM Read
1500 1500 367 MB/s 141 MB/s

A bit of loss in write speed due to smaller MTU but for some reason reads are only maxed at 141MB/s. Much faster than MTU 9000 but nowhere near the write speeds. Definitely MTU issue at work when using Jumbos even though the fact that it’s limited to 141MB/s in reads still doesn’t make sense. The traffic was still evenly split across all interfaces. Trying to match up the MTU’s better. Could it be that either NexentaStor or vSphere doesn’t account for the TCP header?

NexentaStor MTU vSphere MTU VM Write VM Read
8982 8982 165 MB/s ? MB/s

Had to abort the read test as it seemed to have stalled completely. During writes the speeds flactuated wildly. Yet Another test.

NexentaStor MTU vSphere MTU VM Write VM Read
9000 8982 356 MB/s 4.7 MB/s
8982 9000 322 MB/s ? MB/s

Once again had to abort reads due to stalled test. Not sure what’s going on here. But for giggles, decided to try another uncommon MTU size of 7000.

NexentaStor MTU vSphere MTU VM Write VM Read
7000 7000 417 MB/s 143 MB/s

Hmm. Very unusual. Not exactly sure what the bottleneck here is. Still, definitely faster than single 1Gb NIC. Disk on the SAN is definitely not the issue as the IO never actually hits the physical Disk.

Another quick test was done by copying a test file to another via DD. The results were also quite surprising.

[root@testvm test]# dd if=ddfile1 of=ddfile2 bs=16k

This is another one I didn’t expect. The result was only 92MB/s which is less than the speed of a single NIC. At this point I spawned another test VM to test concurrent IO performance.
The same test repeated concurrently on two VM’s resulted in about 68MB/s each. Definitely not looking too good.
Performing a pure DD read on each VM did however achieve 95MB/s per VM so the interfaces are better utilized. Repeating the tests with MTU 1500 resulted in 77MB/s (copy) and 133MB/s (pure read).

Conclusion: Jumbo Frames at this point do not offer any visible advantage. For stability sake sticking with MTU 1500 sounds like the way to go. Further research required.

Watchguard Firebox x550e/x750e/x1250e – pfSense

Overview

Last week I picked up this Watchguard Firebox x500 for cheap to experiment with. It turned out to be a great success so it was time to try it for “real” on better, faster, production capable hardware. I’ve been following this thread with great interest for a while, a few guys in the thread spent a lot of time getting these things working with pfSense. If it wasn’t for these guys, this conversion would be extremely time consuming if not impossible.

I’ve bought 3 Fireboxes on eBay, x550e, x750e and an x1250e. Even though they are all different models and WatchGuard sells them as products with increasing price/performance for each higher model, the actual hardware inside these firewalls is almost identical.

The “e” series Fireboxes are significantly deeper than the x500/x700 series, which turns out is actually too long for the 4U bracket I bought for the uplink shelf. The x750e is 14″ deep and it still requires another 2″ to accommodate the power plug. The Firebox x500 comes in at 9″ plus the plug.

I’ve started with the mid-level Core x750 as the Guinea pig. A bit of irony with the sticker asking to install Firebox software. It’s never gonna happen.

Force10 S50 Racking

It’s time to put the Force10 S50 into use. This means unfortunately a lot of work as it involves taking down all the virtual and physical servers, pulling all the wiring, the original switches, rewiring everything from scratch and documenting all patch panel and port changes. I use a Visio document to keep track of each patch panel and switch port mappping. Also use it to map out all my networks at home and at the various data centers.

Pulled all the patch cords out.

Pulled the two Dell 5324’s and racked the Force10 switch up.

Completely rewired the back of the servers. Organized the cables better so that pretty much all connections to each server are sequential. In original setup I had each switch perform a different role. One was strictly for SAN/NAS traffic, the other was for LAN/DMZ/Web traffic. With a single switch now, all that will be split strictly via VLAN so it makes the wiring much, much simpler and easier to follow.

Everything connected and running again. The only thing left to configure is LAGs to connect to two other switches. I want to make sure that the switch runs fine and performance is optimal before playing with LAG/LACP and Spanning Trees.

The whole process of swapping switches and rewiring took me almost 6 hours. Though a lot of that time was spent planning the layout and documenting the ports.

Force10 S50

Over the years I’ve been adding more and more hardware to my home network. This led to a big increase in network complexity, especially once I started adding multiple managed switches. Got to a point where I had 4 managed switches + 4 unmanaged switches running the house. Combined with many VLAN’s across switches, LAG groups and Spanning Trees, things got pretty crazy. So, I’ve decided to simplify the setup a bit. Get rid of some of the switches and change the network layout to a proper star topology with one core switch in the center.
For the core switch, I needed something fast. A switch that will be able to handle all the traffic I can throw at it, but also cheap, cause I’m on a budget.

So I ended up buying this Force10 S50 switch on eBay. Hoping to replace the two Dell 5324’s I have in the main rack and have it act as my core switch. These Force10 switches are supposedly incredibly fast. They’re known for their super low latency which is perfect for an iSCSI setup I’m planning.

To be perfectly honest I’ve never heard of the Force10 brand, but a friend of mine who’s something of a networking guru highly recommended it. The price was definitely right and since this switch used to cost over $6K (+$5K L3 Routing Option), it was definitely a high end switch in its time.

The Force10 S50 is a 48 Port, Gigabit, Managed Layer 2 Switch / Layer 3 router. This particular model is SA-01-GE-48T, running SFTOS, which is kind of a bummer as it can not be upgraded to the more recent FTOS but it shouldn’t be too big of a deal if the switch works correctly.

This switch uses an RJ45 connector for console access which I believe is the same as most Cisco switches. It’s still a regular serial RS-232 port but with a different plug. Not exactly sure what the reasoning behind this is.

It so happened that I had an RJ45 to DB-9 cable kicking around that was still in its original packaging, since I’ve never had to use it before. All my current switches are standard DB-9 plugs but I tend not to toss cables unless I have oodles of them.

However, turns out that whatever that cable came with used different signal paths as it didn’t work with the S50. The switch didn’t show any output during it’s bootup sequence.
Luckily, I also had a converter plug that plugs into a DB-9 socket but also has an RJ-45 plug in it. Using a standard network patch cable worked in this case and I was able to communicate with the switch via PuTTY.

Powered up the switch. I was surprised to find out how quietly it ran, considering several small fans are used in it. Not that it matters too much as the servers occupying the same rack will drown out any other noise.
The system booted up and ran a self diagnosis with no issues. The BPS LED is amber due to the fact that there’s no Backup Power Supply hooked up. The power draw on the switch is about 90W which seems a bit high, I’m curious to see if that usage goes up when the switch starts moving traffic.

More Toys!

Made a quick trip to CBI today to pickup some eBay purchases waiting for me there. Looks like Christmas came early this year. 🙂

3 WatchGuard Firebox Firewalls (x550e, x750e and x1250e) and a Dell Force10 S50 L2/L3 switch…

…and a new 4U bracket for my uplink wall, replacing an existing 2U bracket.

Also a whole bunch of Dual Port Intel PRO/1000 NICs and few CPU’s for the firewalls.

It’s going to be a fun next few days getting everything working.

WatchGuard Firebox X500 – pfSense Hack

Overview

I’ve been using pfSense for a few years but almost always on a dedicated PC or a virtual machine. For a while now I’ve been toying with the idea of getting pfSense running on an actual firewall box. The advantage of running it on an actual firewall is twofold, size and power draw. Plus, it’s common hardware, easier to develop.

I picked up this WatchGuard Firebox X500 Core from Kijiji. Price was great and best of all the guy was about 5 minutes away from me.

As soon as I got home I wasted no time taking it apart. Removing the final screw behind the Void Warranty sticker was quite satisfying.

The interior guts of the firewall. Ugh. Disgusting filthy inside, must have been running in some crappy closet.

Some good blasts of air and it looks much better. Now to analyze the components. The WatchGuard Firefox is essentially just an x86 PC. The motherboard implements Intel 815 Chipset.
It comes with an Intel Pentium III based Celeron M 310 1.2Ghz as its processor. There’s a possibility of upgrading this CPU to a faster processor like the SL8BA,SL8BG Pentium M 1.7Ghz or SL6N5 LV version of Pentium M 1.7Ghz. The firewall comes with 6 10/100Mbit Ethernet Ports. These ports are driven by on-board Realtek chips. Even though one of the ports is designated as WAN, in pfSense any port or combination of ports can be used for WAN functionality.

The Firebox also comes with 256MB of PC-133 Non-Ecc Memory. The chipset supports up to 512MB so I asked around and a buddy of mine happened to have a few 512MB sticks.

Had to break another seal, another proof of voided warranty.