VOIP at Home

asterisk-bubble
As part of a side project I jumped into the “exciting” world of VOIP and Telephony. Of course, as soon as someone mentioned VOIP or PBX I think Asterisk. Now, I’ve heard of Asterisk for years, and I’ve always considered setting up a PBX at home but I could never figure out how I would utilize the features on a day to day basis. Now that that side project has come up, it was the perfect excuse.

First thing first, I jumped on Kijiji and picked up a set of Cisco 7960G phones.
IMG_0766

The phones were in decent condition and already have been preloaded with the SIP version of the firmware. They were however each running different version of the firmware so first step was to factory reset these phones and update to latest available SIP firmware.
To factory reset the phone.

* Plug in the phone
* Hold # key until message …
* Press the following key sequence 123456789*0#
* Press 2 to delete network config
* wait for the phone to reboot

Updating firmware is pretty trivial. Once a TFTP server has been configured, simply drop the updated firmware onto the TFTP server and point the phones at the IP address. These phones automatically check for updated firmware on the server during the bootup process.

If DHCP option for TFTP isn’t configured, an Alternate TFTP server can be configured on the phone via Network Configuration. Before attempting to change the setting the phone must be unlocked via option 9. The default password for the phones: cisco

When the phone boots up it checks the TFTP server for configuration files. The two most important files are SIPDefault.cnf and SIP<mac>.cnf. These are two of the files that the phone will look for during start up to self-configure.

The SIPDefault is a great place to put common settings for all phones.
Example Format for SIPDefault.cnf

image_version: P0S3-8-12-00
proxy1_address: "voip.olympia.local"            ; Can be dotted IP or FQDN
proxy2_address: ""              ; Can be dotted IP or FQDN
proxy3_address: ""              ; Can be dotted IP or FQDN
proxy4_address: ""              ; Can be dotted IP or FQDN
proxy5_address: ""              ; Can be dotted IP or FQDN
proxy6_address: ""              ; Can be dotted IP or FQDN
proxy_register: 1
messages_uri:   "*97"
phone_password: "cisco" ; Limited to 31 characters (Default - cisco)
sntp_mode: unicast
sntp_server: "pool.ntp.org"
time_zone: "EST" ; assuming you're in GMT
time_format_24hr: 0 ; to show the time in 24hour format
date_format: "D/M/Y"  ; format you would like the date in
dial_template: dialplan
autocomplete: 0
call_hold_ringback: 1
#nat_received_processing: 0
logo_url: "http://voip.olympia.local/cisco/logo.bmp"
services_url: "http://voip.olympia.local/cisco/services.php"
directory_url: "http://voip.olympia.local/cisco/directory.xml"

SIP<mac>.cnf (SIP<MAC>.cnf replaced with actual MAC address of the phone)

#office phone
 
image_version: P0S3-8-12-00
 
line1_name: 100
line1_authname: "100"
line1_shortname: "Ext 100" ; displayed on the phones softkey
line1_password: "secret" ; replace with a strong password
line1_displayname: "THC Inc 100"; the caller id
 
proxy1_port: 5060
proxy1_address: voip.olympia.local
 
# Line 2 Setup
line2_name: 1000
line2_authname: "1000"
line2_shortname: "Intercom"
line2_password: "secret"
line2_displayname: "Intercom";
 
phone_label: "THC Inc.  " ; add a space at the end, looks neater
 
#remote access to the phone
telnet_level: 2 
phone_password: "cisco" ; Limited to 31 characters (Default - cisco)
 
# uncomment below to connect over the internet
#nat_enable: 1
 
#custom phone logo
#logo_url: "http://kermit/asterisk-tux.bmp"
 
user_info: none

Couple of great resources for Cisco config file:
http://wiki.siftah.com/Cisco_7960G_IP_Phone_on_Asterisk
http://www.jtech.net/ip_phone/cisco/7940_sip_config.aspx

There are two providers I’ve signed up with. VOIP.ms and CallCentric.com. Both provide support for SIP/IAX2 phones and have very low rates, which is great for someone who doesn’t use land lines all that often. Having two providers also adds failover for outgoing calls.

I’ve always known about Asterisk but I also found few derived projects like AsteriskNow (turn key distro) and Elastix which had a much more polished web GUI. There’s tons of articles on the web about configuring Asterisk/Elastix and it, in itself is pretty trivial. But there are quite a few gotchas, especially if the server is hosted and open to the internet. For this exercise I ended up with Elastix as it seemed more user friendly than AsteriskNow with the default FreePBX UI.

So, couple of items when configuring a web open Asterisk server.

* Make sure extension passwords are VERY strong.
Since the password is never entered manually (only in Asterisk config and TFTP file config). It can be made impossibly strong and long.

* Protect the server via fail2ban.
Fail2ban is a fantastic solution to brute force attacks. It simply scans the log file file for failed authorization attempts and then simply blocks the incoming connection at the firewall effectively shutting the remote address out. This makes brute-force attacks impractical since the attacker can only try 4-5 passwords / hour for every IP they have.

Elastix already comes with fail2ban pre-installed, just needs to be configured.

/etc/asterisk/logger.conf

[general]
dateformat=%F %T
 
messages => security

This will create a new log file /var/log/asterisk/message

Add the following to /etc/fail2ban/jail.conf

[asterisk-iptables]
# if more than 4 attempts are made within 6 hours, ban for 24 hours
enabled  = true
filter   = asterisk
action   = iptables-allports[name=ASTERISK, protocol=all]
              sendmail[name=ASTERISK, dest=email@domain.com, sender=fail2ban@yourdomain.com]
logpath  = /var/log/asterisk/messages
maxretry = 4
findtime = 21600
bantime = 86400

Create a new file /etc/fail2ban/filter.d./asterisk.conf

 Fail2Ban configuration file
# Author: Xavier Devlamynck
 
[INCLUDES]
 
# Read common prefixes. If any customizations available -- read them from
# common.local
before = common.conf
 
[Definition]
 
# Option:  failregex
# Notes.:  regex to match the password failures messages in the logfile.
# Values:  TEXT
#
#log_prefix= \[\]\s*(?:NOTICE|SECURITY)%(__pid_re)s:?(?:\[\S+\d*\])? \S+:\d*
 
failregex = SECURITY.* SecurityEvent="FailedACL".*RemoteAddress=".+?/.+?/<HOST>/.+?".*
            SECURITY.* SecurityEvent="InvalidAccountID".*RemoteAddress=".+?/.+?/<HOST>/.+?".*
            SECURITY.* SecurityEvent="ChallengeResponseFailed".*RemoteAddress=".+?/.+?/<HOST>/.+?".*
            SECURITY.* SecurityEvent="InvalidPassword".*RemoteAddress=".+?/.+?/<HOST>/.+?".*
 
# Option:  ignoreregex
# Notes.:  regex to ignore. If this regex matches, the line is ignored.
# Values:  TEXT
#
ignoreregex =

* Firewall issue (Extension UNREACHABLE)
There’s a potential issue when using extensions that connect to the host PBX via the internet. Specifically, shortly after the extension connects it’ll disconnect and Asterisk will report the extension as UNREACHABLE, even though the ping is less than 2000ms (default for qualify=yes).
Changing the value of qualify to higher numbers (i.e. qualify=6000) has no effect.

The problem is that by default Asterisk sends a keep alive command to the phones once every 60 seconds, a lot of firewalls will close the UDP socket within 60 seconds as part of cleanup. So, to combat this the fix is to change the default “qualifyfreq” on Asterisk to a value less than 60 seconds. I changed the value to 45 seconds and it seemed to fix the problem for me.
The file to modify is located at /etc/asterisk/sip_custom.conf.

Simply add the following lines to the file:

qualify=5000
qualifyfreq=45
keepalive=yes

This will ping all the connected devices every 45 seconds and it’ll wait 5 seconds for response before timing out.

Additionally, in pfSense go to: Firewall -> System -> Advanced -> Firewall/NAT -> Firewall Optimization Options and change the option to “Conservative”. This will increase the timeout for the UDP connections before pfSense considers the connection closed and removes the socket.

* Locking down Web Interface.
If you absolutely must have port 80/443 open to the web, it’s a good idea to move the default web site to a virtual hosted site as it’ll make it a bit harder for bots to discover the site since it can only be accessed via proper url.

* Cisco Phone Logo
The logo image on the 7960G phones can be customized. It is a simple 4 bit image with a size of 90×56 pixels. The phone will automatically rescale larger images but it won’t look very good. I found the best way to save a compatible image is to use MS Paint and save the image as 256 color bitmap. The 7960G will also attempt to dither the image to display different shades.
IMG_0775

* Intercom System / Paging
This turned out to be more a challenge than I originally thought. Cisco SIP firmware doesn’t officially support paging so a workaround is to configure a line with auto-answer capability and then create a paging group in Elastix. Even though the 7960G has auto_answer configuration option, this feature can not be enabled via the config file and has to be manually configured on each phone. Not a big deal when dealing with a dozen or so phones, definitely an issue when dealing with large deployments.

New Mail Server – The Search

Looking to set up a new mail server. Currently running Exchange Server but it’s a pain to manage multiple domains. Spam filtering not great. I have few dozen domains and I’d like to consolidate them. I need an easy to use and manage server where I don’t have to spend too much time on it.

Required Features:

* Free (or really cheap)
* Webmail
* Push via Activesync or IMAP Idle
* Anti-Spam Filtering
* Multi-Domain / Aliases
* All-In-One Solution
* Web Based Admin / No manual file editing

Ability to easily create email aliases, great for registering at web sites. Also to determine which web site is selling email addresses.

Product Free WebMail Anti-Spam Push Notes
Axigen Almost Free (100 User Limit) Yes No* No* * Available in full version
Kolab 3.0 Free ? ? ? Failed to install
hMailServer Free No Yes Yes (IMAP Idle) No web admin
SoGo Free ? ? ? Steep learning curve
Blue Mind Free Yes No ? Missing features
Horde Free ? ? ? Steep learning curve
Zimbra Community Free Yes Yes Yes Worth a second look
Synovel Collabsuite Free Yes Yes ? Not sure if Push works

It’s quite a list but I wanted to make sure I touch on all the popular products.

First thing’s first. Configure a Virtual Machine for testing. Most of the products are Linux based, and I’m fine with that but some products run only under Windows so a Windows VM will also be required.

CentosInstall

Once both VM’s were configured and all the updates have been installed I took a snapshot of each VM to make moving onto the next product easy.

WinInstall

I also created a local hosts entry to see how the product responds accessed “remotely”

# Copyright (c) 1993-2009 Microsoft Corp.
#
# This is a sample HOSTS file used by Microsoft TCP/IP for Windows.
#
# This file contains the mappings of IP addresses to host names. Each
# entry should be kept on an individual line. The IP address should
# be placed in the first column followed by the corresponding host name.
# The IP address and the host name should be separated by at least one
# space.
#
# Additionally, comments (such as these) may be inserted on individual
# lines or following the machine name denoted by a '#' symbol.
#
# For example:
#
#      102.54.94.97     rhino.acme.com          # source server
#       38.25.63.10     x.acme.com              # x client host

# localhost name resolution is handled within DNS itself.
#	127.0.0.1       localhost
#	::1             localhost

192.168.77.35 linux.mailtest.com
192.168.77.36 windows.mailtest.com

On To Testing

Upgrading Force10 S50

I picked up another Force10 S50 switch from eBay. This one will be going to the datacenter to replace a couple of Dell switches.

IMG_0601

First thing’s first. Need to configure the switch for remote management. Let’s see what we got.

Force10 Boot Code...
Version 01.00.25 05/12/2005
 
Select an option. If no selection in 2 seconds then
operational code will start.
 
1 - Start operational code.
2 - Start Boot Menu.
Select (1, 2):2
 
 
Boot Menu Version 01.00.25 05/12/2005
 
 
 
Options available
1 - Start operational code
2 - Change baud rate
3 - Retrieve event log using XMODEM (64KB).
4 - Load new operational code using XMODEM
5 - Display operational code vital product data
6 - Update Boot Code
7 - Delete operational code
8 - Reset the system
9 - Restore Configuration to factory defaults (delete config files)
[Boot Menu]

Hmm. That’s a pretty old version of the boot code. In any case, I needed to reset the switch to factory in order to clear the admin password. Once the switch finished booting I went on to see what’s it running.

(Force10 S50) #show hardware
 
Switch: 1
 
System Description............................. Force10 S50
Vendor ID...................................... 07
Plant ID....................................... 01
Country Code................................... 04
Date Code...................................... 072005
Serial Number.................................. DE4526001
Part Number.................................... 759-00001-00
Revision....................................... 0A
Catalog Number................................. SA-01-GE-48T
Burned In MAC Address.......................... 00:01:E8:D5:A2:A0
Software Version............................... <strong>2.1.4</strong>
 
Additional Packages............................ Force10 QOS
                                                Force10 Stacking

Eek! That’s a pretty old version of the SFTOS firmware. In fact it’s one of the first, if not the first release for the S50. Really need to upgrade it to something a bit more recent. This actually turned out to be much bigger deal than I anticipated. Force10 site seems to be of no help trying to find an updated firmware for this box. After a LOT of Googling, I finally stumbled upon a 2.5.1 .bin file on an IBM ftp site. That’s the only image I have found after hours of searching. While not the latest version it sure is a big step up from the current 2.1.4 SFTOS.

SFTOS-SA-2.5.1.3

Of course there was no way for me to verify that this image is in fact legitimate, and there was a high risk of bricking the switch if the image wasn’t up to par. But I decided to risk it and go ahead with the upgrade.

First thing’s first. Need to configure networking on the switch in order to TFTP the bin file to it from my workstation. The config process is quite different in 2.1.4 than in 2.5.3 but after a bit of research, I found the proper commands.

 
User:admin
Password:
(Force10 S50) >enable
Password:
 
(Force10 S50) #network parms 192.168.77.248 255.255.255.0 192.168.77.1
 
(Force10 S50) #network mgmt_vlan 1
 
(Force10 S50) #show network
 
IP Address..................................... 192.168.77.248
Subnet Mask.................................... 255.255.255.0
Default Gateway................................ 192.168.77.1
Burned In MAC Address.......................... 00:01:E8:D5:A2:A0
Locally Administered MAC Address............... 00:00:00:00:00:00
MAC Address Type............................... Burned In
Network Configuration Protocol Current......... None
Management VLAN ID............................. 1
Web Mode....................................... Disable
Java Mode...................................... Disable
 
(Force10 S50) #config
 
(Force10 S50) (Config)#interface 1/0/1
 
(Force10 S50) (Interface 1/0/1)#vlan participation include 1
 
(Force10 S50) (Interface 1/0/1)#no shutdown
 
(Force10 S50) (Interface 1/0/1)#exit
 
(Force10 S50) (Config)#exit
 
(Force10 S50) #ping 192.168.77.6
 
Send count=3, Receive count=3 from 192.168.77.6

Once I confirmed the switch is on the network. It was time to send up the .bin file. I configured a tftp server on my workstation and copied/renamed the bin to sftos.bin to save myself some typing.

 
(Force10 S50) #copy tftp://192.168.77.6/sftos.bin system:image
 
Mode........................................... TFTP
Set TFTP Server IP............................. 192.168.77.6
TFTP Path......................................
TFTP Filename.................................. sftos.bin
Data Type...................................... Code
 
Are you sure you want to start? (y/n) y
 
TFTP code transfer starting
 
TFTP receive complete... storing in Flash File System...
 
 
File transfer operation completed successfully.

The copying process took only a few seconds. Took a few minutes to store the File in Flash. Now for the main part. Reboot the switch and hope that the new image will take. At the end of this either the switch will work or it’ll be an expensive paper weight.

(Force10 S50) #reload
 
Management switch has unsaved changes.
Would you like to save them now? (y/n) n
 
Configuration Not Saved!
Are you sure you want to reload the stack? (y/n) y
 
 
Reloading all switches.
 
Force10 Boot Code...
 
tffsDevCreate failed.
 
Storing configuration files
Storing Code base
usrTffsConfig returned 0xffffffff, formatting...
Calling FORMAT ROUTINE

The switch was formatting for quite a long time. After a while I was pretty sure the switch was hosed, but decided to stick it out and see if it comes back. Sure enough, few more minutes later the rest of the upgrade process completed.

Format routine returned with status 0x0
Recover configuration files
CPU Card ID:   0x508245
dimInitialize returned 3
adding the default image - code.bin to the list
dimImageAdd returned -3
Boot Menu Version: 30 Aug 2006
Version 02.01.43 08/30/2006
 
Select an option. If no selection in 2 seconds then
operational code will start.
 
1 - Start operational code.
2 - Start Boot Menu.
Select (1, 2):
 
 
Operational Code Date: Thu Jan 11 02:38:37 2007
Uncompressing.....
 
                       50%                     100%
||||||||||||||||||||||||||||||||||||||||||||||||||
Attaching interface lo0...done
 
Adding 40920 symbols for standalone.
PCI device attached as unit 0.
PCI device attached as unit 1.
PCI device attached as unit 2.
PCI device attached as unit 3.
PCI device attached as unit 4.
Configuring CPUTRANS TX
Configuring CPUTRANS RX
MonitorTask - Active
ConsoleDebugger - Disabled
 
(Unit 1)>STACK: master on 0:1:e8:d5:a2:a0 (1 cpu, 5 units)
STACK: attach 5 units on 1 cpu
This switch is manager of the stack.
 
 
User:
 
  ******* Binary configuration file detected, migration in progress....  *******
  ******* To prevent loss of data, DO NOT POWER OFF MACHINE!             *******
  ******* Migration to text configuration file completed.                *******
Saved Configuration being applied...Please Wait....
 
  ******* Applying text configuration.                                   *******
 
  ******* The following lines in "startup-config" failed execution:
  ******* Line 12:: logging facility -À
  ******* Line 14:: logging history 5595
  ******* Line 15:: logging history size 838875251
 
User:
  ******* Finished text configuration                                    *******

So, that looked like it worked. Let’s reboot the box and see if all is well.

 
User:admin
Password:
 
Force10-S50>enable
Password:
 
Force10-S50#reload
 
Are you sure you want to reload the stack?(y/n) y
 
 
Reloading all switches.
Calling hardware API to reset the box....
If system doesn't reset within 1 minute, hardware might have become faulty....
 
.............................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
 
Force10 Boot Code...
 
 
 
CPU Card ID:   0x508245
Boot Menu Version: 30 Aug 2006
2Version 02.01.43 08/30/2006
 
Select an option. If no selection in 2 seconds then
operational code will start.
 
1 - Start operational code.
2 - Start Boot Menu.
Select (1, 2):2
 
 
 
Boot Menu Version: 30 Aug 2006
 
Options available
1  - Start operational code
2  - Change baud rate
3  - Retrieve event log using XMODEM
4  - Load new operational code using XMODEM
5  - Display operational code vital product data
6  - Run flash diagnostics
7  - Update boot code
8  - Delete operational code
9  - Reset the system
10 - Restore configuration to factory defaults (delete config files)
11 - Activate Backup Image
[Boot Menu] 10
[Boot Menu] 9
 
Are you SURE you want to reset the system? (y/n):y
 
Calling hardware API to reset the box....

Eureka! We’re good to go. Switch seems healthy and fully functional. Out of curiosity I wanted to see what features came with the image downloaded from the IBM site.

User:admin
Password:
Force10-S50>enable
Password:
 
Force10-S50#show hardware
 
Switch: 1
 
System Description............................. Force10-S50 48GE 2TENGIG L3 Stackable switch
Vendor ID...................................... 07
Plant ID....................................... 01
Country Code................................... 04
Date Code...................................... 072005
Serial Number.................................. DE4526001
Part Number.................................... 759-00001-00
Revision....................................... 0A
Catalog Number................................. SA-01-GE-48T
Burned In MAC Address.......................... 00:01:E8:D5:A2:A0
Software Version............................... 2.5.1.3
 
Additional Packages............................ Force10 QOS
                                                Force10 Multicast
                                                Force10 Stacking
                                                Force10 Routing
 
Pluggable Modules and Transceivers:
 None
 
--More-- or (q)uit
Force10-S50#

Ah. That’s much better. Not only it’s running great the new image adds few more features including L3 routing. Not too shabby.

Now the switch is ready to be configured as per my earlier post here.

How to bring down a rack in just few quick easy steps.

Note to self. pfSense full + SSD = Not Good.

Had a bit of a kerfuffle over the long weekend. Was working remotely on a firewall trying to get NAT through VPN working properly, the firewall didn’t appear to work as it should so I decided to reboot it to make sure all routes are clear. Turns out it was a bad idea. The firewall failed to come back up which resulted in a 100km/drive to the data center to investigate the issue.

When I got there, I quickly plugged in a monitor to the 1U server acting as the firewall and was greeted by a nice “Boot Disk Failure” error on the screen. Good thing I brought a spare firewall, one of my modified Watchguard x750e’s. A quick swap and few minutes later the new firewall was running a restore script on the Watchguard (thanks to a config backup).

Back at home a little digging showed that non-embedded version of pfSense do not play nicely with SSD’s. Especially if RRD is installed. Turns out that RRD writes out new traffic images to physical disk every minute. Multiply that by 8 months and that’s a lot of writes to an SSD that doesn’t support write leveling.

Lesson learned. I bought a couple more Watchguard x750e’s from eBay. One to serve as a replacement at another data center, which is also running installed pfSense on a PC/SSD combo and another Firebox as a warm spare.

Windows 7 PPTP VPN Local Domain Authentication

Here’s a hidden gem. When connecting to another site domain via Windows PPTP (Yeah I know it’s no longer secure). Windows will lose connectivity to the local domain and might even cause an account lockout as Windows will attempt to use VPN credentials on local domain and will fail authentication often without a prompt.

There’s a hidden setting in the .pbk file that prevents credentials from being overwritten.

The file is located in C:\Users\<USERNAME>\AppData\Roaming\Microsoft\Network\Connections\Pbk

* Open the file in Notepad.
* Change UseRasCredentials=1 to UseRasCredentials=0
* Save file and reconnect VPN

And that’s it. Now even with the gateway being redirected all local authentications will continue to work.

pfSense on Watchguard Firebox x5000

It’s been a while since I played around with firewalls. Picked up a really cheap Watchguard Firebox x5000 Peak on eBay. Could not resist taking the Peak platform for a spin with pfSense. I’ve heard good things about the power of the “Peak” platform. Even though the x5000 is older generation than the typical x550/x750s I’ve been playing with.

Unlike the x500 which comes with only 10/100 Realtek (problematic) interfaces. The x5000 is equipped with 3x 10/100/1000 nics and 8x 10/100 nics, all by Intel.

This box also comes with a second COM port at the rear of the unit. Though it serves no real purpose when running pfSense as all output is defaulted to COM1.

This firewall is actually just slightly shallower than the x750e series boxes. The x750 firewall is just a bit too deep for wall mount racks and required a right-angle power cable in order to fit. The x5000 is about an inch shorter.

Taking the cover off exposes all the various components of this box. The Intel NIC chips can be clearly visible.

The “Peak” comes with 512MB of RAM standard. Unfortunately I could not locate any more DDR RAM to upgrade it, so for now will stick with the stock amount. I used to have tons of original DDR RAM modules, but got rid of them thinking I’m never gonna need them again.

Just like most other boxes. This one comes with a 128MB Compact Flash card. And just like most of the other boxes, the IDE port is available for additional internal storage.

I wanted to take a quick peek at the CPU. Specifically to see what the stock CPU is. What I discovered was quite shocking. First of all, I had a problem getting the heatsink off from the CPU. Ended up tearing the heatsink and CPU directly from the ZIF socket. Closer examination revealed the cause of the problem. The amount of heat paste on this thing is insane. The gap between the heatsink and CPU was at least a couple of millimeters. It was everywhere. Someone really went to town on this sucker.

It took me almost 30 minutes to clean the CPU, heatsink and motherboard from all the thermal compound. I used plenty of rubbing alcohol and tons of cloths to get everything cleaned. Everything was completely covered with the goop.

The CPU could finally be identified. Intel Pentium 4 2.8Ghz SL6PF. Not the most energy efficient CPU, quite the opposite. This firewall definitely sucks down a lot of juice. It’s actually a pretty fast CPU for the task. With firewalls raw speed matters most when dealing with latency. This one has plenty of speed.

Loading pfSense on this firewall is incredibly easy. There’s no BIOS flashing required. It boots pfSense from a 2GB CF card without any problems at all. Unlike the x500 firewall, which has Realtek NICs and randomly stops responding, the Intel NICs are bulletproof.

Once I got the firewall configured. It was time to do some throughput testing. I was expecting the speeds to be sub-par simply because both the 1000Mbit and 100Mbit interfaces are all on the PCI bus. PCIe did not exist in P4 era.

The interfaces on pfSense are identified as follows.
em0-em2 = 10/100/1000 NICs
fxp0-fxp6 = 10/100 NICs

Interestingly enough, the 8th 10/100 NIC does not appear in the list of devices. Not sure if this is due to pfSense limitation or a problem with the port itself.

On with the testing. The test was performed by copying a 8GB file between two machines. Both machines stored the file on an SSD to eliminate the disk as the bottleneck. The test was performed 3 times for each direction and values averaged over the test.

Source Interface Destination Interface Transfer Speed CPU Usage
fxp0 em0 10.3 MB/s 20%
em0 fxp0 11.5 MB/s 21%
em0 em1 42.8 MB/s 41%
em1 em0 97.8 MB/s 100% *
fxp0 fxp1 11.5 MB/s 27%
fxp1 fxp0 11.5 MB/s 27%

* pfSense UI interface stopped responding during the transfer.

Once again it’s been shown that the bottleneck is the PCI interface. With maximum theoretical speed of 133 MB/s shared across the PCI bus, the firewall will never be able to attain faster combined throughput than roughly 100 MB/s. This is not terrible though for a small network as that kind of saturation is rather rare. What I can’t explain is why pfSense shows different max speeds depending on direction of data. This technically shouldn’t make a difference. I saw the same thing happen when benchmarking the x550e firewalls.

In the end, this is actually a pretty sweet box for what it costs nowadays. Even with “only” 512MB of RAM it’d be sufficient to run a small rack. However I do not recommend it for office use. The fans run 100% speed all the time and are fairly loud, haven’t found a way to throttle the fans other than replacing them with something quieter. The box, running pfSense draws about 52W at idle and almost 100W under load.

pfSense on Watchguard Firebox – More Tweaks

There’s a known problem with pfSense 2.0.1 and Watchguard Firebox x750e and x1250e. Namely the additional 4 NIC interfaces have a tendency to drop out with a Watchdog error and the only remedy is to reboot the box. I came across a fix that seems to resolve the issue permanently. Been running for 2 weeks on the PCI-e interfaces without dropout. Previously I’d be luck to get 3 days on those.

The fix is to add these lines to /boot/loader.conf.local

hw.bce.tso_enable=0
hw.pci.enable_msix=0
hw.pci.enable_msi=0
net.inet.tcp.tso=0
hw.re.msi_disable=0
hw.re.msix_disable=0

Additionally I picked up a few Western Digital 4GB MicroDrives from eBay. The idea behind this was to replace the flash based CF card in the firewall. In theory the MicroDrive does not have the write limitation of a Flash card so it could potentially store a lot more information on the card like logs, graphs, a/v definitions etc.

I had some problems getting the card read on the computer. First I tried reading directly with my multi card reader but it just went bonkers, wouldn’t read the card at all. Another card reader I had simply shut itself down when I plugged the drive in. I then tried a CF-to-SATA converter and plugged it into a Drive Toaster but after a few seconds it’d drop out. Lastly I tried to connect the drive directly to the computer. I eventually got it working by switching the BIOS from AHCI to SATA. One I got the drive detected properly, the method for loading the pfSense image onto it is the same as with a regular Compact Flash card. The only issue I ran into was when running “clean” on diskpart, the drive seemed to take forever to clean the partition.

Getting the card working on the x750e was effortless. The machine booted up with no issues. The boot process did take slightly longer which is understandable as this is a mechanical device with the same random IOP limit as a regular disk drive.

Netgear GS716T Fan Fix

I’ve had this Netgear GS716T for about 5 years, even then I already bought it used from eBay. Couple of days ago the fans finally gave up the ghost. Replacing them is pretty trivial though as I’ve already replaced fans in several switches recently.

It’s easy enough to remove the cover as none of the screws are hidden and there are no “warranty void” stickers to remove.

The two existing 40mm fans. Turns out one was completely dead, the other was just about to ready to quit via a rather noisy bearing.

Interestingly enough, the fans are held in place via these “nuts” that get punched into the fan housing.

I’ve had few more 40mm fans laying around from my last Dell PowerConnect experiment. Same size but different specs. The new fans again are slightly slower / quieter than the original fans.

Once again had to pay attention to the pinout. Fortunately the GS716T does not use an RPM pin so it won’t even know if the fan is running slower or not.

The fan mounts tapped back into the new fans. Took quite a bit of force to get these in.

New fans mounted in place and switch ready to be closed up. The whole process took maybe 30 minutes. Hopefully will get more life out of the switch yet. Not the greatest switch in the world but perfect for my lab environment.

Piece of Cake.