3 Very Good Things…

I am back! My blog is migrated to new servers, happy days.

To kick off 3 things which have very much impressed me in the last couple of months:

1) Windows 7 RC. Microsoft have an absolute win here, its faster and feels tidier than Vista, the search is lightening quick, the new combined launch/taskbar kicks ass, the XP fallback mode works, it multithreads properly, I haven’t found any kit I can’t make work with it, and its all incredibly stable. I don’t actually know why anyone currently slogging it out on Vista wouldn’t run as fast as they can to do an in place upgrade today.

2) GMail Apps. I dipped my toe in the water using GMail for benking.me.uk a few months ago, I have now migrated warwicknet.com lock stock and barrel. Having your email all in gmail is pure bliss. The labelling effectively gives ‘3D’ folders, and having the power of Google search on your email is awesome. No longer do I have to slog it out in Outlook/Exchange land sorting my inbox by ‘From’ to find that one email from the 17000 in my inbox (I don’t like filing stuff). Its fair to point out though that that new features in Windows 7, means that it can search your outlook email about as quick as well. The downside of GMail of course is Google owning you.

3) Solwise 3G router. This little toy is a work of  art. Basically its a plug in the wall router, that will do wireless, ethernet, and 3G routing (by plugging your existing USB 3G dongle into the provided USB port). It is a right swiss army knife and will do web cam server, file server, print server. The amazing thing is that my T-Mobile 3G connection is far faster via the solwise and then wireless to my laptop than when plugged directly in my laptop. To be honest I am not doing it justice here, read the full review on The Register here.

I would heartily recommend all of the above.

VM Server 2 and Vista Ultimate 64

I have just been trying to get VM Server 2 installed on my new Vista Ultimate 64 installation, and found I just couldn’t logon using either my own Vista username and password or the administrator one, lots of googling revealed all sorts of people having the same problem.

The error message you get when trying to logon is:

The VMware Infrastructure Web Service at “http://localhost:8222/sdk” is not responding (Connection Refused)

This happened regardless of how I was trying to access it i.e. https or http and by hostname or IP.

Turns out that in my Vista Ultimate 64 installation the host file entry for localhost has been reduced to  the IPV6 lookup only:

::1             localhost

Adding back the usual:

127.0.0.1     localhost

Fixed the problem! It is always the little things!!!

Vyatta – Desert Deployment!

I have deployed Vyatta to a lot of different locations, however the deployment I did last week was a little different…

Yas Island is a naturual island on the coast of the United Arab Emirates of about 2,500 hectares or which 1,700 hectares is being developed. It is to be a $40 billion playground of marinas, shops, theme park, water park, hotels and villas not to mention a Formula 1 track.

At the minute though it is little more than a lot of sand, some mounds of earth, a few roads and a lot of cranes, and I get the pleasure on behalf of my client Benoy (architects), of extending their existing Vyatta network to cover both their Abu Dhabi city office and their Yas Island site office.

There were a number of challenges with the deployment:

  1. The connectivity; we had ordered a 2mbit/s leased line from Etisalat, the UAE telco, this was being delivered via a microwave link back to Abu Dhabi, at the point of landing in the country, we had no idea of the reliability, IP Scheme and weren’t even confident about the presentation!
  2. Disruption; the users were using a shared network provided by the client, which was painfully slow, but worked to give them email and basic web access, we had to minimise downtime.
  3. Reliability; we had to do everything we could to ensure reliability and remote maintainability of the network once we had left.

The Kit

Vyatta was the natural choice not only because we were using it across the rest of the Benoy network, but also because of the cost effectiveness of the hardware required to deploy a resilient configuration.

At each site we deployed 1U Dell 860s, with:

  • Dual core Xeon processors
  • 2GBs of Ram
  • Hardware mirrored Sata drives
  • Additional Intel Dual NIC card (giving 4 ethernet interfaces in total)
  • Vyatta 2.3.1

The Configuration

  • 4 Subnets: Workstations, Servers, Internet 1 (leased line), Internet 2 (ADSL)
  • All subnets clustered across the two routers
  • DHCP for workstation subnets (split across the two routers)
  • Masquerade NAT for internal subnets
  • Incoming NAT for email and video conferencing
  • IPSec VPN tunnels back to the UK network and the other Abu Dhabi site
  • Internal and external firewalling

The Microwave Link

The microwave link was a V35 serial presentation that we passed through a Cisco 1841 before passing onto the Vyattas, the resulting connection performed remarkably well giving us about 14ms round trip on pings back to the main Abu Dhabi office.

The Result

The end result is fantastic, speed and response of performance at both sites far exceeded expectations. At the main site we were replacing a Firebox VPN tunnel back to London, which had proved to be a little unreliable and extremely slow, we were putting this down to the quality of the Etisalat connection, however once we replaced it with the Vyatta VPN the network response and reliability was far in excess of expectations and performs as well as the MPLS circuits we have connecting other sites.

Martin Neal, IT Director of Benoy, said ‘I am really pleased with the speed and also the “feel” of the network.

Photos

The Yas Island site office…

Yas Island Construction office

The Benoy team at Yas Island…

Yas Island Benoy Office

Our Microwave Link…

Our microwave link.

Vyatta – Glendale is coming… and I am excited!

The next major release of Vyatta (VC4 – codename Glendale) is due for final release on 22nd April, and I am soo excited!

This release promises to deliver plethora of new features and functionality, including:

1) FusionCLI – A new CLI based around bash that gives simultaneous access to Vyatta configuration and the underlying Linux shell. Woohoo – no more exiting from xorpsh to do basic bash stuff.

2 ) Remote Access VPN – Remote vpn clients are now supported via both PPTP and L2TP/IPSec.

3)  Tunnel Interfaces –  GRE support means that we can now tunnel both non-ip tunnels (Appletalk, etc. – should you really want to), as well as more importantly ip in ip tunnels, which are somewhat more useful, not least for doing resilient routing via backup VPN links.

4) QOS – Although QOS via TC has long been available to the ‘hardcore’ under the hood via tc and the shell, this is the first time that Vyatta have exposed this functionality to masses via the CLI. The initial release notes suggest that it will just be SFQ (Stochastic Fair Queuing), and shaping around packet marking. As is always the case with Vyatta they start simple and grow from there, so you can expect to see HTB, etc. added later on. Personally I am hoping they have ticked the 2.6 kernal option to enable IFB (Intermediate Functional Block), to enable traffic shaping over multiple interfaces… we will see!

5) Redesign of Routing Protocols – Vyatta has in the past been criticised for its routing protocol performance particularly in terms of BGP convergence, I am guessing this is one of the many reasons they have completely revisited the router manager. And they are keen to shout about it, this Tolly report demonstrates $8000 of Vyatta running on an IBM server, giving $30,000+ of Cisco 7204 a complete and utter kicking.

6) VRRP interfaces by VIF – This one is worth special mention because it was my first Vyatta enhancement request that has made it through to release (given I have only ever submitted 2 thats not bad going!). Basically VRRP could only previously be deployed on real ethernet interfaces, great unless like me you subnet networks up and run a ‘router on a stick’ configuration, in which case you need to be able to deploy VRRP across VIFs, you now can! woohoo!

This is but the tip of the iceberg, there are many other great improvements.

I will be deploying a version of Glendale soon, in a test environment, and I will report back!

Vyatta – Clustering

The latest subscription release of Vyatta, 2.3, has seen the addition of clustering capability, which has added greatly to the high availability features of the product.

Previously high availability was really limited to VRRP, which was great but had a couple of issues:

  • You couldn’t use VRRP across VIF interfaces, which made high availability for ‘router on a stick solutions’ tricky.
  • We experienced a few issues with interface bouncing, especially on gigabit interfaces.

VRRP is however a very nice solution, each virtual address is associated with a virtual MAC address that the currently actively router associates with the appropriate interface, the switchover is nearly instanteous.

The new clustering functionality in Vyatta is based upon the Linux HA project, which takes a slightly more simple but arguably more effective approach to the HA whereby when a failure is detected the virtual IP is reassigned to appropriate interface on the secondary router and a gratuitous arp sent out across the associated network segment to mitigate any arp cache issues.

The HA functionality also allows for failover of the ipsec vpn service, at the moment this works pretty simplistically by simply stopping or starting the service as needed, thus on the currently inactive server the VPN service and therefore tunnels simply aren’t up.

Lets take a look at a relatively simple multisite HA Vyatta solution and the associated configuration.

Vyatta Cluster Example

We have two sites, each with a pair of Vyattas configured as router, vpn, firewall, and nat. Behind them is a multi-segment internal network.

Interfaces

ldn-router1 interfaces:

interfaces { loopback lo { 

 address 10.1.1.251 { 

 	prefix-length: 24 

 } 

} 

ethernet eth0 { 

 description: "Internet" 

 address 98.76.54.31 

 	prefix-length: 28 

 } 

} 

ethernet eth1 { 

 description: "Servers" 

 address 10.1.10.251 { 

 	prefix-length: 24 

 } 

} 

ethernet eth2 { 

 description: "Workstations" 

 address 10.1.101.251 { 

 	prefix-length: 24 

 } 

} 

ldn-router2 interfaces:

interfaces { loopback lo { 

 address 10.1.1.252 { 

 	prefix-length: 24 

 } 

} 

ethernet eth0 { 

 description: "Internet" 

 address 98.76.54.32 { 

 	prefix-length: 28 

 } 

} 

ethernet eth1 { 

 description: "Servers" 

 address 10.1.10.252 { 

 	prefix-length: 24 

 } 

} 

ethernet eth2 { 

 description: "Workstations" 

 address 10.1.101.252 { 

 	prefix-length: 24 

 } 

} 

The important thing to notice here is that, the virtual ‘active’ addresses aren’t configured on the network interfaces themselves, instead they come later in the cluster configuration.

The New York site configuration is the same, except of course the IP addresses are changed accordingly.

Cluster
cluster { interface eth0 

 pre-shared-secret: "!secret!" 

 keepalive-interval: 2 

 dead-interval: 10 

 group "ldn-cluster1" { 

 	primary: "ldn-router1" 

 	secondary "ldn-router2" 

 	auto-failback: true 

 	monitor 12.34.56.73 

 	service "98.76.54.33" 

 	service ipsec 

 	service "10.1.10.1" 

 	service "10.1.101.1" 

 } 

} 

The cluster configuration on each router is identical (unless you want to do certain clever things such as run a different routing configuration in failover!). The interface definition is just for the interface that you want to monitor via. You can have multiple monitors however a failover will occur if any monitor returns a failure, in some ways this is a help and some ways its a hindrance, personally I prefer to just monitor an outside address and if its not available then go to failover where hopefully it will be (especially if we use different external blocks by router).

When a router becomes the active member of the cluster, it scans the route table for matches to the service IP and assigns the service IP to the appropriate interface, it then sends a gratuitous arp out of that interface to avoid any arp cache issues.

Routes

One downside of the Vyatta downing the ipsec tunnel when that router is not active, is that you can then only address that router on its dedicated addresses, for example if I wanted to do some remote maintenance ldn-router2 from the New York site while it wasn’t active, the only way I would be able to do so is either to log onto a machine on the London subnet and go via that, or use the public external IP (which I probably don’t want publically accessible anyway).

The solution is very simple, due to the way that VPN route matching works. When making a packet routing decision Vyatta checks the VPN tunnels for a local/remote match first, then checks against the routing table, therefore if we add a static route to each router for the whole internal network to go via its partner, we get a really neat solution:

protocols { static { 

 	route 10.0.0.0/8 { 

 	next-hop: 10.1.10.252 

 	} 

 } 

} 

Thus if a router has the VPN tunnel up (i.e. its active), it never checks the routing table and traffic goes direct, if the router has no VPN tunnel (i.e. its passive), it simply forwards the traffic to the active router.

VPN

The VPN configuration in a cluster is basically the same as a standard configuration, except the local and remote public IPs are the cluster addresses.

vpn { ipsec { 

 	ipsec-interfaces { 

 	interface eth0 

 } 

 ike-group "ike-ny" { 

 	proposal 1 { 

 		encryption: "aes256" 

 	} 

 	lifetime: 3600 

 } 

 esp-group "esp-ny" { 

 	proposal 1 { 

 		encryption: "aes256" 

 	} 

 	proposal 2 { 

 		encryption: "3des" 

 		hash: "md5" 

 	} 

 	lifetime: 1800 

 } 

 site-to-site { 

 	peer 12.34.56.73 { 

 		authentication { 

 		pre-shared-secret: "secret" 

 	} 

 	ike-group: "ike-ny" 

 	local-ip: 98.76.54.33 

 	tunnel 13 { 

 		local-subnet: 10.1.0.0/16 

 		remote-subnet: 10.3.0.0/16 

 		esp-group: "esp-ny" 

 	} 

 } 

} 

NAT

An easy pitfall on the NAT configuration is to forget that Vyatta processes source NAT before checking vpn or routing table matches. The fix is simply to exclude your internal network as a destination in the NAT configuration.

nat { rule 101 { 

 	type: "source" 

 	outbound-interface: "eth0" 

 	source { 

 		network: "10.1.101.0/24" 

 		} 

 	destination { 

 		network: "!10.0.0.0/8" 

 	} 

 	outside-address { 

 		address: 98.76.54.31 

 	} 

 } 

} 

VIFs

As i mentioned earlier, Vyattas implementation of VRRP doesn’t allow you to use VRRP on virtual VLAN interfaces, which is frankly a little annoying (although it will be fixed in the next release hopefully).

However under clustering it works perfectly, as the service IP can match and be assigned to any interface, real or virtual.

Conclusion

The clustering in Vyatta has added just enough simple HA clustering functionality that ‘just works’ to enable us to deploy far more complex and reliable solutions than was previously possible.

This is also just the tip of the iceberg, in future releases we can expect to see multiple cluster (allowing Active/Active configurations) and extra services added to the failover capability.

Vyatta – Forwarding traffic to Squid

If you are using Vyatta and want to transparently forward traffic at the router level to a separate Squid proxy you will find that the standard firewall configuration in Vyatta just isn’t up to the job (yet!).

The workaround is to use the /etc/rc.local file to make IPTables do the job for you, heres how we did it:

#!/bin/sh -e
#
# rc.local
#
# Modified to forward to squid cache
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will “exit 0″ on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#

IPTABLES=”/sbin/iptables”
IP=”/sbin/ip”
SQUID=”10.1.1.1″      # Internal address of our squid box

# Webcache jump to cache
echo Setting up jump to webcache

# clear any existing entries
$IPTABLES -t mangle -F
$IPTABLES -t mangle -X

# Don’t mark webcache traffic
$IPTABLES -t mangle -A PREROUTING -j ACCEPT -p tcp –dport 80 -s $SQUID
# Internal subnets to exclude
$IPTABLES -t mangle -A PREROUTING -j ACCEPT -p tcp –dport 80 -d 10.0.0.0/8     #Don’t cache internal

# External sites to exclude
$IPTABLES -t mangle -A PREROUTING -j ACCEPT -p tcp –dport 80 -d 1.2.3.4 #IP address of site you want to exclude from going to the cache

# Now mark our traffic, we have a number of subnets on virtual interfaces we want to grab, if you aren’t using vifs simply use eth1 or whatever you are using
$IPTABLES -t mangle -A PREROUTING -j MARK –set-mark 3 -i eth3.102 -p tcp –dport 80
$IPTABLES -t mangle -A PREROUTING -j MARK –set-mark 3 -i eth3.103 -p tcp –dport 80

# Send the marked traffic to table 2 (you can actaully use whatever table you want, i used 2 because we are using eth2 for the subnet squid is on.
$IP rule add fwmark 3 table 2

# set the default route for table 2, change eth2 for the interface you are on
$IP route add default via $SQUID dev eth2 table 2

# Make sure we exit
exit 0

Vyatta – Cisco on a shoestring?

We use Linux extensively at bit10, for DNS, front line mail handling, proxying, web hosting, development platforms, you name it we probably do it in Linux or have at least had a good go…

Probably the most important job it does for us though is for our firewalling, core network routing and traffic management for both bit10s internal systems and for the ISP hosting side of the business.

Our implementation is a highly bespoke customisation of Debian using things like iptables, IMQ, vconfig, etc. all good stuff and happily handles the routing and traffic management for our entire colocation and ISP services, however we would be the first to admit it isn’t the easiest to maintain and look after. The options in the commercial club (Cisco/Juniper) are simply way to expensive and don’t offer the flexibility we require.

A couple of months a go a client came to me needing some fairly significant network reorganisation. They had multiple offices around the world, at each site they had a very unsegmented ‘flat’ network, between the offices they had a mixture of MPLS and VPN tunnel solutions, and a number of single points of failure.

What we needed to do was to segment up each site into sensible subnets and bring additional resilience to the firewalls and routers.

Our initial reaction was to simply do another custom Linux configuration similiar to our own setup, however we were concerned about the time to get this right and the implications of future maintenance, so we look off the shelf and very quickly discovered Vyatta.

Vyatta for the uninitiated of you punts itself as ‘ The dawn of open-source networking’, personally I think this is a bit of pessimistic afterall we have been doing routing and firewalling with Linux as long as I can remember, its more of a ‘The late afternoon after a good siesta of open-source networking’.

However here at bit10 ‘Loving Doing Digital’ headquarters we can’t really criticise people on their taglines… 🙂

So why is Vyatta different?

The answer is simply its relatively easy to get to up and running, has a pretty web interface for those who have command line fear, and above all fantastic support.

Vyatta comes in two flavours, the fully open-source free ‘community edition’ and the ‘supported edition’. The community edition will suit you down to the ground if you have relatively simple requirements, basic routing/firewalling/etc., however if you have pushing the envelope you are going to the need the supported edition, which comes in two flavours ‘$647/£325’ for web only support, and ‘$897/£450’ for full telephone support, both supported flavours include free updates with the lastest fixes.

The telephone support is superb.

Vyatta does have limitations, especially if you are used to getting under the hood and having the full flexibility of Linux based routing, however the payback is a solution thats far simpler to manage.

Things we didn’t like:

  1. We couldn’t configure the built in firewall to transparently push traffic to a Squid proxy server. We got around this by going under the hood and having a custom rc.local file that tag and forwarded the traffic (I will post our script on followup blog).
  2. VRRP over VIFs. Vyatta supports VRRP (Virtual Router Redundancy Protocol) and VLANs out of the box, however you can only run VRRP on real ethernet inferfaces, which is troublesome if you are doing a ‘router on a stick’ solution. We have spoken to Vyatta about this and they have pencilled the functionality in for an upcoming build – good stuff!
  3. The CLI, in a nutshell the CLI isn’t like IOS, its good and fulfill its job, its just that mental switch you have to make from IOS mode to something different.

Things we really liked:

  1. VRRP (Virtual Router Redundancy Protocol) out of the box… VRRP was so simple to set up (on real ethernet interfaces), the village idiots really stupid cousin could have done it.
  2. The separation of configuration from installation. Take a clean server, insert Vyatta CD, one line to install it to the local hard drive, copy your configuration onto it. Job done.
  3. Support. The Vyatta team are passionate about their product, both on the telephone and on the web, and they know its limitations and will tell you so, so you don’t waste any time trying to make it do something it won’t.

Top Tips:

  1. Unless you are doing something dead basic, go for the supported package, this ensures you get the latest version, you get the support and frankly you support Vyatta.
  2. Don’t mess around trying to deploy it on that 5 yr old bodge of a server you have sat in the corner gathering dust, your firewall/routers will be business critical, so splash out on some decent hardware, we deployed on Dell 860s (<£1,000 each), which is all supported hardware and had no hardware related issues.
  3. Think about what you are trying to achieve, plan it first.
  4. Make sure you actually know a little about networking, while making life as simple as possible for the user, its not dumbed down to the level of a £50 ADSL router from PCWorld. If you don’t know the basics about routing, firewalling, etc. get someone to help you.

Performance:

I haven’t done any formal performance testing on Vyatta, however we have deployed it in bandwidth heavy environments, with upwards of 200 of LAN users across 10 network segments, and a single processor Dell 860 is running well under 10% load…
So is Vyatta Cisco on Shoestring… in my opinion if Vyatta can do the job you want, its definitely preferable to the Cisco option, probably the strongest reason beyond just price, is that Vyatta abstracts the software from the hardware, i.e. within reason you can redeploy a Vyatta configuration on any server with enough interfaces.

All in all well worth looking at.

Ubuntu, PPTP, Windows 2003 VPN Server

My transition from a Windows to a Linux desktop has raised a few teething problems, almost the most annoying was the fact that I for some reason I could only VPN into certain MS PPTP VPN servers.

In the end I figured out that that the difference was I could VPN into Windows 2000 servers but not Windows 2003.

In the logs I was getting:

LCP terminated by peer (random load of symbols)

I tracked it down in the end to one setting under the PPTP config.

Under KNetworkManager this appears under ‘Compression & Encryption’, ‘Encryption’, ‘Require 128 bit MPPE encryption’. Check this and it all works.

Exchange 2003 and Exmerge Woes

Today I needed to extract some mailboxes to PSTs from our Exchange 2003 server.

I kept getting a permissions error in my Exmerge.log file:

'Verify that the Microsoft Exchange Information Store service is running and that you have the correct permissions to log on. (0x8004011d)'

This is one of those annoying ‘exchange being oversecure for no good reason moments’, basically the administrator user (under which you are probably logged on for such an exercise) is denied permission to access individual mailboxes.

I found a couple of MS documents on the subject:

http://support.microsoft.com/kb/322312

and,

http://technet.microsoft.com/en-us/library/aa996410.aspx

Neither of them unfortunately resolved the problem in my case. It appears that even adding the administrator user to a temporary windows ‘exchange recovery’ group still didn’t allow access to the mailbox for extraction.

The solution is to:

  1. Create a new Active Directory Group called ‘Exchange ExMerge’ (or whatever you like).
  2. Give the new group full permissions on the store as per the 2nd MS article above.
  3. Create a new user and add it to the Exchange ExMerge group and Domain Admins.
  4. Logon to the Exchange server as the new user, run Exmerge and it should finally work.

You could of course just create the user and give it the appropriate permissions but I was just being ‘proper’! 🙂

Unlucky Raid 1

The scenario…

Dell 2800, 2x36Gig (RAID 1 – Systems Boot)), 3x146Gig (RAID 5 – Data), primary file server, primary AD controller.

12PM Yesterday Dell Server Manager began reporting a fault on disc 1 (36Gig), the first step in this scenario is always to reseat the disc to rule out any possible connection issues.

This indeed brought up both drives again, only to be followed 5 minutes later by a failure of both drives in the set.

Rebooting the server to the RAID BIOS revealed that there were faults found  on disk 0 not disk 1 (as reported by the Dell Server Manager).

Lesson 1 – never underestimate the power of the hot spare.

At this current  moment in the time the company was without file, print and primary AD.

We decided to remove the disc 0 (that the bios said was faulty), and bring the disc 1 (that at bios said had no faults), back online.

Unfortunately a reboot in this scenario forced a windows chkdsk, which found faults on the drive. Although the system booted, AD did not come up and system was effectively broken.

We had no spare chasis available to drop the data disks  into, so we were in a fast reinstall situation.

The first step however was to seize the FSMO roles using NTDSUtil to one of our backup AD controllers, and cleanup all references to the old primary AD.

While these changes were propogating through AD we began the reinstall process. As we had no spare disks we were forced to reinstall to the one disk we thought was good.

All went well, and we were smart and didn’t repromote the server to AD yet.

Lesson 2 – only bring a server up as an AD controller when its a known good.

Unfortunately later that day the server failed again. Fortunately we were able to bring the disk online again without a problem.

The next day the first new disk arrived, we took the decision to not install it yet as we felt mirroring might force the remaining dodgy disk to fail…

… which it dutifully did.

We took the decision to then install the spare, allowing the RAID BIOS to resync the drives without rebooting the OS. We felt that we could have done it on the fly with OS running but that gave us a greater chance of the mirroring failing.

It took about 40 minutes for the mirror to take place.

We then removed the dodgy mirror, leaving us running on 1 disk until a further warranty replacement arrived.

Lesson 3: Always get disks of different ages, batches, brands if possible, two disks from the same batch can fail at the same time.

Sigh…