The workaround is to use the /etc/rc.local file to make IPTables do the job for you, heres how we did it:

#!/bin/sh -e
#
# rc.local
#
# Modified to forward to squid cache
#
# This script is executed at the end of each multiuser runlevel.
# Make sure that the script will “exit 0″ on success or any other
# value on error.
#
# In order to enable or disable this script just change the execution
# bits.
#

IPTABLES=”/sbin/iptables”
IP=”/sbin/ip”
SQUID=”10.1.1.1″      # Internal address of our squid box

# Webcache jump to cache
echo Setting up jump to webcache

# clear any existing entries
$IPTABLES -t mangle -F
$IPTABLES -t mangle -X

# Don’t mark webcache traffic
$IPTABLES -t mangle -A PREROUTING -j ACCEPT -p tcp –dport 80 -s $SQUID
# Internal subnets to exclude
$IPTABLES -t mangle -A PREROUTING -j ACCEPT -p tcp –dport 80 -d 10.0.0.0/8     #Don’t cache internal

# External sites to exclude
$IPTABLES -t mangle -A PREROUTING -j ACCEPT -p tcp –dport 80 -d 1.2.3.4 #IP address of site you want to exclude from going to the cache

# Now mark our traffic, we have a number of subnets on virtual interfaces we want to grab, if you aren’t using vifs simply use eth1 or whatever you are using
$IPTABLES -t mangle -A PREROUTING -j MARK –set-mark 3 -i eth3.102 -p tcp –dport 80
$IPTABLES -t mangle -A PREROUTING -j MARK –set-mark 3 -i eth3.103 -p tcp –dport 80

# Send the marked traffic to table 2 (you can actaully use whatever table you want, i used 2 because we are using eth2 for the subnet squid is on.
$IP rule add fwmark 3 table 2

# set the default route for table 2, change eth2 for the interface you are on
$IP route add default via $SQUID dev eth2 table 2

# Make sure we exit
exit 0

Probably the most important job it does for us though is for our firewalling, core network routing and traffic management for both bit10s internal systems and for the ISP hosting side of the business.

Our implementation is a highly bespoke customisation of Debian using things like iptables, IMQ, vconfig, etc. all good stuff and happily handles the routing and traffic management for our entire colocation and ISP services, however we would be the first to admit it isn’t the easiest to maintain and look after. The options in the commercial club (Cisco/Juniper) are simply way to expensive and don’t offer the flexibility we require.

A couple of months a go a client came to me needing some fairly significant network reorganisation. They had multiple offices around the world, at each site they had a very unsegmented ‘flat’ network, between the offices they had a mixture of MPLS and VPN tunnel solutions, and a number of single points of failure.

What we needed to do was to segment up each site into sensible subnets and bring additional resilience to the firewalls and routers.

Our initial reaction was to simply do another custom Linux configuration similiar to our own setup, however we were concerned about the time to get this right and the implications of future maintenance, so we look off the shelf and very quickly discovered Vyatta.

Vyatta for the uninitiated of you punts itself as ‘ The dawn of open-source networking’, personally I think this is a bit of pessimistic afterall we have been doing routing and firewalling with Linux as long as I can remember, its more of a ‘The late afternoon after a good siesta of open-source networking’.

However here at bit10 ‘Loving Doing Digital’ headquarters we can’t really criticise people on their taglines…

So why is Vyatta different?

The answer is simply its relatively easy to get to up and running, has a pretty web interface for those who have command line fear, and above all fantastic support.

Vyatta comes in two flavours, the fully open-source free ‘community edition’ and the ‘supported edition’. The community edition will suit you down to the ground if you have relatively simple requirements, basic routing/firewalling/etc., however if you have pushing the envelope you are going to the need the supported edition, which comes in two flavours ‘$647/£325′ for web only support, and ‘$897/£450′ for full telephone support, both supported flavours include free updates with the lastest fixes.

The telephone support is superb.

Vyatta does have limitations, especially if you are used to getting under the hood and having the full flexibility of Linux based routing, however the payback is a solution thats far simpler to manage.

Things we didn’t like:

1. We couldn’t configure the built in firewall to transparently push traffic to a Squid proxy server. We got around this by going under the hood and having a custom rc.local file that tag and forwarded the traffic (I will post our script on followup blog).
2. VRRP over VIFs. Vyatta supports VRRP (Virtual Router Redundancy Protocol) and VLANs out of the box, however you can only run VRRP on real ethernet inferfaces, which is troublesome if you are doing a ‘router on a stick’ solution. We have spoken to Vyatta about this and they have pencilled the functionality in for an upcoming build – good stuff!
3. The CLI, in a nutshell the CLI isn’t like IOS, its good and fulfill its job, its just that mental switch you have to make from IOS mode to something different.

Things we really liked:

1. VRRP (Virtual Router Redundancy Protocol) out of the box… VRRP was so simple to set up (on real ethernet interfaces), the village idiots really stupid cousin could have done it.
2. The separation of configuration from installation. Take a clean server, insert Vyatta CD, one line to install it to the local hard drive, copy your configuration onto it. Job done.
3. Support. The Vyatta team are passionate about their product, both on the telephone and on the web, and they know its limitations and will tell you so, so you don’t waste any time trying to make it do something it won’t.

Top Tips:

1. Unless you are doing something dead basic, go for the supported package, this ensures you get the latest version, you get the support and frankly you support Vyatta.
2. Don’t mess around trying to deploy it on that 5 yr old bodge of a server you have sat in the corner gathering dust, your firewall/routers will be business critical, so splash out on some decent hardware, we deployed on Dell 860s (<£1,000 each), which is all supported hardware and had no hardware related issues.
3. Think about what you are trying to achieve, plan it first.
4. Make sure you actually know a little about networking, while making life as simple as possible for the user, its not dumbed down to the level of a £50 ADSL router from PCWorld. If you don’t know the basics about routing, firewalling, etc. get someone to help you.

Performance:

I haven’t done any formal performance testing on Vyatta, however we have deployed it in bandwidth heavy environments, with upwards of 200 of LAN users across 10 network segments, and a single processor Dell 860 is running well under 10% load…
So is Vyatta Cisco on Shoestring… in my opinion if Vyatta can do the job you want, its definitely preferable to the Cisco option, probably the strongest reason beyond just price, is that Vyatta abstracts the software from the hardware, i.e. within reason you can redeploy a Vyatta configuration on any server with enough interfaces.

All in all well worth looking at.

Andrew has quite rightly identified one of bit10‘s recurring problems, people often fail to understand everything we do. Mostly they pigeonhole us into being experts in the thing they use us for…

Of course this is entirely our fault, we start by hiding behind ‘Full service digital agency’ who ‘Loves doing digital’, but few people realise what this means, then we somehow fail to make the effort to communicate to our customers what we actually do.

This problem is exacerpated by the fact that our customer research has told us that current bit10 customers NEVER look at our website (hence our site now is firmly targeted at new customers).

The solution is frankly simply down to plain old conversation, take the time to chat to your customers to identify their current and future business needs and then point our where you can help them! As per usual everyones enemy… ‘Time’ gets in the way.

For the record, bit10 do:

• Internet, intranet, digital marketing strategy and consultancy
• Creative web design
• Web application development
• Accessibility testing
• Usability testing
• Web analytics
• Online marketing (dm, adword and affiliate marketing management)
• Hosting
• Internet connectivity (University of Warwick Science Park only)

Basically everything Internet! You can see the full blurb here.

And thanks Andrew for giving me an excuse to list bit10s services on my blog!

Today I finally got someone on the phone at the MS Partner program who could help me.

Essentially there is a highly painful online process to go through, which i set off onto using Firefox (under Ubuntu)…

Why oh why do this MS? It just looks bad, and reinforces peoples view of you as monopolistic.

When I mentioned it to the guy at the partner program, its response was ‘Well we give you the software under the partner program, so you have no excuse not to use it’… Completely missing the point that I might prefer to use Linux for other reasons…

Anyway, towing the line I struck up IE6 (yes using CrossOver Linux), and somewhat amusingly most of the navigation wasn’t visible to me unless I hovered over it.

Sigh…

In the end I figured out that that the difference was I could VPN into Windows 2000 servers but not Windows 2003.

In the logs I was getting:

LCP terminated by peer (random load of symbols)

I tracked it down in the end to one setting under the PPTP config.

Under KNetworkManager this appears under ‘Compression & Encryption’, ‘Encryption’, ‘Require 128 bit MPPE encryption’. Check this and it all works.

I kept getting a permissions error in my Exmerge.log file:

'Verify that the Microsoft Exchange Information Store service is running and that you have the correct permissions to log on. (0x8004011d)'

This is one of those annoying ‘exchange being oversecure for no good reason moments’, basically the administrator user (under which you are probably logged on for such an exercise) is denied permission to access individual mailboxes.

I found a couple of MS documents on the subject:

http://support.microsoft.com/kb/322312

and,

http://technet.microsoft.com/en-us/library/aa996410.aspx

Neither of them unfortunately resolved the problem in my case. It appears that even adding the administrator user to a temporary windows ‘exchange recovery’ group still didn’t allow access to the mailbox for extraction.

The solution is to:

1. Create a new Active Directory Group called ‘Exchange ExMerge’ (or whatever you like).
2. Give the new group full permissions on the store as per the 2nd MS article above.
3. Create a new user and add it to the Exchange ExMerge group and Domain Admins.
4. Logon to the Exchange server as the new user, run Exmerge and it should finally work.

You could of course just create the user and give it the appropriate permissions but I was just being ‘proper’!

Now I have had an interesting email through from a really nice guy called Gavin Jenks, who works for Autoglass and has commented on my blog more than a couple of times.

Gavin has asked me if I can delete a comment he made, because he doesn’t want to get in trouble with the powers that be at Autoglass. 

I am really sorry Gavin mate but I have thought about it a lot and I just don’t think it would be right, heres my reasoning:

1. The only thing with a longer memory than the Internet is the taxman, so even if I delete it, the comment will still exist on archive sites, etc. and just put into question other content here.
2. If Autoglass don’t give staff guidelines on this sort of thing then really they can’t do anything against you.
3. If the Autoglass marketing people are any good, they will long ago have discovered these blog entries and read them and if they were concerned about your comments they would have already taken action.
4. I just don’t think its in the spirit of blogging.

Gavin, if you want to retract the comment, please feel free to post a retraction and I will ensure it gets published.

Wow, you know you have hit the bigtime when one of your clients gets in The Register with a full on bit of espionage.

To cut a long story short a client of ours, VBi Limited, got broken into a load of kit stolen under extremely suspicious circumstances.

The ‘baddies’ broke in, gained access to the server room within minutes (the server room, located in a highly unobvious location), and whipped in excess of 30 drives out of the servers.

Sunday 1st April, the phone rang, I will never forget the moment I had a particularly aggressive hangover, ‘Hi Ben, its Chris here, we’ve been robbed!’ (well he used something a little more aggressive than ‘robbed’, but I am trying to keep my blog relatively expletive free’).

We broke out the disaster plan, the first hour was assessment i.e. sent the client, Chris, in to find out how bad it was – meanwhile i engaged on a major recovery program of my own, consisting of Berocca, bacon of eggs and enough water to drown an otter.

A couple of hours later the verdict came back, we had lost pretty much every removable drive – so that’ll be a full on recovery then.

Monday 2nd April, Les and I were in the car at 6am on the road to rainy Blackburn, enroute trying to acquire 30 odd replacement hard drives, Dell frankly were as much use as a server without hard drives, in fact we couldn’t even find the right phone number to get someone who could vaguely begin to think about the merest possibility of helping us within the next week let alone the next few hours.

Fortunately our good friends at Serversource, in Northampton, came to the rescue and had all the drives complete with caddies, with us in Blackburn, by lunchtime. Good work fellas!

Thanks to Les and I working our magic, by the end of play Monday we had restored the AD, all servers base restored, by the end of play Tuesday we had completed a full recovery of all services…. CRM, Intranet, Exchange, the whole works. 

Thanks for coming!