I’ve been thinking about load balancing a bit lately, and ways to do it at a very large scale. With a typical load balancer set up scaling is usually achieved by scaling up, then out. That is, first we scale vertically by buying bigger, more powerful load balancers until we reach the limit of current hardware capabilities (or our checkbook). Then we can start to scale horizontally by deploying more load balancers, and assigning different virtual IPs (VIPs) to each pair of load balancers with DNS and global site load balancing tricks used to direct customers to one of the VIPs.

These load balancers are usually deployed in pairs for high availability, with one active and the other running in standby mode to give us redundancy, or — depending on your load balancer — both might be able to run in an active-active configuration. The reason for the pairwise configuration is due to the need for load balancers to maintain the state of the client connections, and the need to synchronize this state between the active and standby devices in case of a failure. This state synchronization becomes difficult beyond a single pair.

One obvious problem with this set up is that it’s not very cost efficient. The 1:1 redundancy means that you always have 50% of your capacity going unused. That is, if we have 10 pairs of load balancers, we must provision each pair with full 1:1 redundancy. When a single load balancer fails its entire load is only distributed to its standby partner. We still have nine other standby load balancers going unused. So if we have 50 Gbps of load we must provision 100 Gbps of capacity. It would be more efficient if we could do N+1 redundancy across the entire pool of load balancers. So if we have 50 Gbps of load, and expect to have no more than 10 Gbps of load offline at any time due to failures, then we only need to deploy 60 Gbps of capacity.

A second problem is that these off the shelf load balancers are usually very expensive. There is a trend towards using open source software and commodity hardware in many parts of the data center, and it would be good to do the same for our load balancers. So for our solution let’s assume we will be using commodity server hardware running Linux with some customizations to do the load balancing. Those customizations might include new user space software, kernel modules, and tweaks to the kernel.

So how could we solve the problem of making load balancing scale horizontally? To be clear, by scaling horizontally we mean N+1 redundancy, with an arbitrary number of load balancers in a pool, which are all actively directing connections for the same VIPs. Equal cost multipathing (ECMP) balances the traffic across our pool of load balancers and our load balancers then balance the connections across our pool of servers.

The ECMP configuration is flow based (as is the norm in most ECMP configurations). A flow is a stream of packets identified by a 4-tuple of source address, source port, destination address, destination port, or in simple words: one half of a bidirectional connection. Being flow based means that flows will take the same path through the network until there is a change in the network topology (due to link failure, node failure, or configuration change). Because of this we can expect the packets in a flow to traverse the same load balancer under normal conditions, and we can expect some number of flows to shift to different load balancers if there’s a failure. Whether those flows shift back to their original paths once the failure is redressed depends on our network equipment. With ECMP it’s not guaranteed that both directions of a connection will use the same path. This asymmetrical routing means we cannot rely on our load balancers seeing both directions of a connection.

So the major problem we’re trying to solve is how to direct an established connection to the correct server when a change in the network topology causes the flow to move from one load balancer to another.

One approach to this problem is to synchronize state between all load balancers, so the state of all flows are known by all load balancers, such that they can all direct the flow correctly. This is obviously feasible in a pair of load balancers, as we described earlier, but extending this to an arbitrary number of devices does not scale very well.

Another approach is not to fully synchronize state, but to store state in some distributed database such that it can be queried and synchronized on an as-needed basis. In a simplistic approach this might allow us to solve our problems at the expense of packet delay and the introduction of denial of service attack vector through state exhaustion. I believe it is possible to overcome these issues, and I have some ideas for this involving the use of DHTs and some funky overlay routing that are gestating at the moment, so I might write a follow up post on that in the near future.

Meanwhile, I want to describe an approach that I believe may be a solution to the problem. The idea is to use flow hashing to distribute connections amongst servers, much as ECMP on network distributes flows on links. The approach does not require state synchronization and appears to be robust against denial of service attacks. The drawback is that the connections may not be distributed fairly, and in some cases during a failure a small percentage of connections could be broken. To expand on that first drawback a bit: because the load balancers do not track the total number of connections on all servers, and make a decision to send new connections to servers with the least number of connections at that time, you may end up with longer lived connections building up on a server, and the load balancers continuing to send connections to it, potentially making the problem worse by slowing down processing of those connections on that server.

I should call out one design principal that has influenced the design here before I get too far along. In general I believe that the best place to manage and scale the connection state is on the servers. Clearly these servers have to track the state anyway. Also, it is inappropriate to use load balancers for any kind of front-end security. The front-end servers (e.g. proxies) should run hardened operating systems with modern network stacks and appropriate packet filters. This is combined with hardware ACLs in your routers. You should not be running or exposing any services on the front-end servers that you don’t explicitly know about and trust to be open.

Inbound traffic

The load balancers – as with all load balancers – would be configured with one or more VIPs. This could be configured with a virtual port and a real port in which case port translation will also be performed. Naturally all of our load balancers will share the same configuration of VIPs and real servers.

To handle network topology changes and load balancer failures we need to ensure that all load balancers will make the same decision in where to deliver a packet. The primary method for achieving this is to use flow hashing in a way that is analogous to ECMP. The load balancer produces a hash from the source address and port, and optionally the destination address and port, and then does a lookup in a hash table to find the server. If all load balancers have the same configuration, identical hash tables, and make use of the same hashing algorithm, then they will all derive the same answer.

Although a lot of work can be put into producing a hashing algorithm with a reliably good distribution, it is unlikely that it can ever be perfect or apply to every set up. To allow us to optimize the distribution we make our hash table sufficiently larger than the number of servers and have a many to one mapping between flow hashes and servers. Then by adjusting the number of slots each server occupies in the hash table we can improve the distribution of connections over our servers. If we wanted, we could also apply a weighting to servers so that more powerful servers will occupy more slots and receive more connections, and vice versa.

At this time I don’t have enough data to give good answers on how often the hash table would need to be optimized. This is clearly a field where much of the secret sauce will exist, and will depend on the circumstances of the deployment: nature of the service, distribution of the clients, and so on.

The hash table will need to be managed in a coordinated way, this assumes a central controller, which can monitor load and health of servers, optimize the hash table, and push it to the load balancers.

You may have noticed I’ve skipped over something important. If we update the hash table while there are established connections, and the server that those flows hash to is different in the new hash table, then those flows will be directed to the new server, which having no state for the connection will respond with a RST breaking the connection and forcing the client to reconnect. Now it may be that in many deployments the number of times this happens is very small as a percentage of total connections and is not even noticed, but as a principal I find it ugly so we should do something about it. To maintain connections during hash table updates we could consider two options.

The first option is for each load balancer to maintain two versions of the hash table. The local version is what all decisions for directing flows are made from. This table contains a timestamp on each slot that gets updated whenever the slot is hit. The master version is built and pushed by the central controller. The load balancers attempt to merge as much of the changes from the master version into their local versions as they can. In order to prevent breaking connections they only replace slots that are over a certain age. The load balancer keeps doing this until the master version is fully merged into its local version. It is possible that the local versions never get fully merged before a new master version is pushed. This option would also mean there would be some deviation between load balancers on the state of the hash tables being used to make forwarding decision.

The second option requires us to start maintaining some per connection state in a new table that is local to each load balancer. Inbound packets are first looked up in this table and directed according to the found state. If no state is found they are hashed and directed according to the hash table. We build this table as new connections arrive. A product of this is now we can update the hash table without fear of interrupting existing connections, including adding and gracefully removing servers from the pool. But we have unfortunately added some complexity and gone some way to violating our guiding principal of not having to track state.

Because all of our load balancers must accept and map established connections at any time they cannot be picky about seeing an arriving packet for a flow they have no state for. This means that DDoS attacks could target the state table of our load balancers with floods of spoofed packets, causing these to fill up with fake state. Because these are commodity servers it is cheap to simply install lots of memory as one part of the mitigation. It also makes sense to limit the size of the state table to prevent it consuming all available memory. Then the worst case scenario is that the attack fills the state table causing valid state to rotate out. If that happens the behavior is as if we simply didn’t have the state table, which means all valid flows will continue to be hashed and directed according to the hash table. If the velocity of changes to the hash table is kept low then the number of client connections that fail can be kept low, possibly near zero. Thus the attacker will not achieve a true denial of service in this scenario. From this it seems sensible for the central controller to be aware of the state table sizes of the load balancers in its pool, it can then potentially adjust the way it optimizes hash tables in the event that any of these are full in order to slow the velocity of hash table changes. It may also be that the deployment has other DDoS mitigation systems in place which can help prevent the attack anyway.

Although we have largely mitigated the issue of state table attacks, we might be able to do better and prevent fake state from being installed in the first place. To do this we would look to use some form of TCP cookies. Inbound SYN packets would not generate any state. Outbound SYN+ACK packets would be given a cookie with a HMAC where the secret is known only to the load balancers. The return ACK packet would then be authenticated at which time state could be created on the load balancer.

Another requirement is to remove state from the state table on a load balancer when a connection is closed. There will need to be a timeout after which the state is removed if no new packets for a flow a seen. In addition to this the state would be cleared when a RST or FIN packet is seen from the client. But if an RST packet is sent by the server, or a FIN packet is sent by a server to which the client does not send a reciprocal FIN, the load balancer over which the inbound flow traverses is unlikely to see this packet due to asynchronous routing and cannot remove the state. This needs consideration as to whether it’s possible for inbound and outbound load balancers to discover each other through some key in the HMAC or similar, and thus to relay RST and FIN state information. It may be that the timeout on the state table is sufficient to avoid this additional complexity and overhead.

Outbound traffic

Return traffic in this example is very simple. We do not adjust the source address and port so the return traffic simply passes through the load balancer which translates the real server address and port to the VIP. This configuration is a static part of the VIP configuration and requires no state to be kept.

Last year I was browsing the STV website. The website mentions a piece of software written by the Department of Internal Affairs (DIA) called the “STV calculator” that is used to count the votes in our STV elections. Being a nerd I naturally thought I’d like to get my hands on the source code and have a look. Not because I think there’s anything wrong with it, just because that’s what curious people do. I also wanted to test whether New Zealand was making its election software open for scrutiny by the public, a basic principal that shouldn’t need explaining. Unfortunately this hasn’t gone so well.

My first step was to try contacting the email address that is prominently displayed in large font on every page of the STV website (stv@dia.govt.nz). I did this twice in October last year but received no replies. Disturbed that this email address wasn’t being monitored I then tried contacting the DIA directly. They run the website and developed the STV calculator so surely they can help. I explained that I’d been to their STV website, emailed their STV email, and got no reply, requested they have someone check the website and remove the email address it if it’s no longer monitored, and provide the source code for the STV calculator. This wasn’t very fruitful either, they couldn’t help me, and they tell me they don’t even know what “STV” is. After much back and forth explaining they conclude by sending me back to their STV website.

Getting no luck with these avenues I took it to the Minister for Internal Affairs, Nathan Guy in March this year and asked him to treat it as an OIA request. Five days later my request was bounced to Rodney Hide, Minister for Local Government, and 13 days after that I was bounced back to the Department of Internal Affairs.

On the 29th of April the DIA notified me that they are extending the timeframe to reply to my OIA request by 10 days in accordance with Section 15A(1)(a) of the Act. The reason they give is that the source code is held by a third party and more time is needed to carry out consultation and make decisions about my request. The final decision came 21 days later on the 20th of May declining to release the source code under section 9(2)(ba)(ii) of the Act because the code is “subject to an obligation of confidence and release of the code would damage the public interest.”

What I find foul about this is that the DIA contracted an third party to develop software used in our elections, never made release of the source code a condition of that contract, and now I suspect the third party does not want to release it because they want to protect their IP. The public should have a right to review all source code and systems used in our elections.

I’ve since filed a complaint with the Ombudsman. But this is clearly a bad precedent; I would like to see some official policy or legislation around voting systems used in the country that ensures they are completely open to scrutiny by the public.

Props to WCC for the draft Wellington 2040 framework. This is just a framework and there’s a lot of work ahead, but I’m excited by what that work will produce if carried out by a good team of architects, designers, and planners. Of course producing great masterplans for our precincts is one thing, but it will amount to nothing if the Council doesn’t make the right decisions to bring those plans to fruition. These will be long term decisions, the benefits of which often won’t be realised until after the councillors making the decision have moved on.

Anyway, oddly, on reading the framework I couldn’t help but notice that half of Te Aro is suspiciously missing. Here’s the Victoria Precinct getting some serious attention in the landscape section, note the creation and reinforcement of connections, and just all round busyness going on in this picture.

Here’s the other side of Taranaki St, aka SoCo…

Err, why no love? It’s not just in this section, it’s all throughout the framework. A lot of attention is given to what could become of the Victoria precinct, but little to no attention given to SoCo. Of course this is only a framework, and the SoCo area should end up being properly addressed in one of the precinct masterplans that are meant to follow this. But here’s a few things I hope to see come through to improve pedestrian connectivity in this area (there’s absolutely none at the moment through the southern super block).

• Extend Ebor St to Taranaki St when the Capital City Motors site is redeveloped.
• Extend Alpha St, Fifeshire Ave, and Barker St when redvelopments permit.
• Protect pedestrian rights of way through the Methodist Church to Holland St, from Holland St north through the lane to Courtenay Plc, and through Moore Wilson’s.
• Develop a chain of midblock pedestrian connections from Ebor St southwards to Jessie St, Vivian St, and Frederick St.
• Provide a connection between Sages Ln to St Martin’s Sq.
• Of course ultimately I’d like to see Abel Tasman St extended to Cambridge Tce too. But that’s probably a hard sell.

WTF is SoCo you say? It’s the area South of Courtenay, for lack of a better name this works for me. Credit to Tom Beard for the term.

A repost on a proposal I made to Auckland’s Skyscrapercity forum on 21 August 2009. The proposal was that the railway for the North Shore should be routed via Devonport and the peninsula up to Takapuna, rather than the now default proposal for it to merge with the Northern Motorway around the area of Northcote Point or the Onewa Rd interchange.

We hear that Auckland will absorb something like half a million extra residents by 2050 (now people think it might be more). Nobody wants to put this additional population in the ever sprawling suburban hellzones that have grown up around Auckland’s fringes, instead the desire is to intensify existing urban areas. I believe Takapuna and the Devonport peninsula are perfect for this kind of intensification. The area is relatively flat, it’s a desirable place to live, it offers good recreation, and it’s close to employment areas.

Proper planning would see some medium density intensification around Devonport. Low to medium intensification northwards up the peninsula to Takapuna. With Takapuna potentially intensified to the same degree as Vancouver’s west end (below), allowing the area to absorb over 100 thousand new residents. And it goes without saying that public transport should underpin this, giving these residents rapid connections directly to Takapuna, Albany, and Auckland’s CBD.

On the other hand, the route currently planned for the railway runs along a motorway where nobody currently lives, where nobody wants to live, and which has no real opportunity for good intensification even if you forced people to live there. The catchment along this section will always be poor, it will consist primarily of transfers from busses from the western North Shore. Making a transfer at Onewa interchange when these western busses could simply run the last few kilometers into the city makes little sense.

Takapuna and the Devonport peninsula will intensify to some degree anyway, the question is how much, and will it be well planned. And when it does will we have built the right transport infrastructure to support it (it’s already choking on vehicle traffic).

With all the hype about this or that battery exploding in recent years the last thing you want to see when working away with a laptop in your actual lap is smoke fuming out the back. Thankfully no explosion in this case, just a poor design leading to a frayed cable connecting the screen. Appears this shorted and started to melt the plastic hence the nasty smoke.

Fortunately it wasn’t the magic smoke, so now I have a desktop for the first time in two years.

This could’ve been good timing, since I was hanging out for the new MacBook Air refresh, but I refuse to buy one since Apple didn’t include USB 3 (seriously, how lazy). I was willing to forgive the lack of page up/down and home/end keys. (Oh and for the love of god Apple throw HFS away already. ZFS please.) So now I just have to wait for Lenovo to release an update for the Thinkpad X1 with an S-IPS panel, 3x USB 3, HDMI, Thunderbolt, but none of the annoying crap I’m never going to use like a VGA port, built in 3G, or that nipple thing (does anyone use it?). Might be a long wait. So hard finding the perfect laptop.

Edit 5:50pm 07/09/2011: Seems I’m not the only one. Ultrabook: Intel’s $300 million plan to beat Apple at its own game — damn it, that Toshiba doesn’t look too bad.

TelstraClear aren’t always right, but they — and all of the other small telco’s that have been objecting lately — are right about this one. The Government’s plan to give the LFCs a 10 year regulatory holiday is a Bad Idea.

As of February 2011 there are 372,062 domains in .co.nz, the next closest is 24,373 in .org.nz, and following that 23,935 in .net.nz. So, what meaningful purpose do these three second level domains (2LDs) serve? I argue that they serve no useful purpose.

The original intent was for commercial domains to be registered under .co.nz, network infrastructure under .net.nz, and other organisations under .org.nz, but clearly this scheme isn’t followed and has proved to be impractical and unecessary. .co.nz has clearly become the default 2LD for almost all domains regardless of the type of registrant (for example my own domain punk.co.nz, clearly I’m not a commercial website). This makes sense, because most people only want to distinguish themselves, their website, or their organisation as being from New Zealand. Distinguishing the type of their organisation offers little value, but makes the domain more difficult to remember. And few people want to register a domain under .org.nz for example and risk someone else registering the same domain under the more popular .co.nz, probably leading to confusion for their customers. In fact .net.nz now gets mostly used by registrants who are not distinguishing network infrastructure, but who want to use the word network in their name (e.g. a website of an artists network).

Many other countries, much larger ones than New Zealand, do perfectly fine without 2LDs, for example Canada, France, and Germany. So scale is not the issue.

Note there are some obvious 2LDs that might make sense, but these are where moderation is used, such as .govt.nz and .school.nz, and these domains currently have very few domains under them.

So what do I think we should do? Well I think we should phase out .co.nz and open up registration under .nz. Second level domains would still be retained for distinguishing government organisations and similar and these would be moderated. .net.nz and .org.nz would be retained with a lower fee than .nz. Similarly .geek.nz, .gen.nz, and the like would be retained with similar lower fees, and a few extra 2LDs would be added for personal uses and reserved for any likely industries that might benefit from a moderated 2LD in future.

To get there I would suggest one of two courses of action.

Option 1: Open up .nz and permit registrations of new domains at the second level. Initially all registrants holding domains under .co.nz would be offered the option to shift their domain to .nz. If they elect to do this then their .co.nz domain will be retained for 12 months with the same configuration as their .nz equivelent. .co.nz would remain open to new registrations, but overtime most people will elect to use a .nz and the number of domains under .co.nz will reduce, probably in line with .net.nz and .org.nz.

Option 2: Make .nz a mirror of .co.nz, except where the domain under .co.nz conflicts with one of the existing, new, or reserved 2LDs. In that case the registrant will be given the option to pick a new domain. .co.nz could then be closed to new registrations, and possibly eventually withdrawn.

A couple of months ago (yes, I’m slow to post as always) I was made aware of a proposal that may mean the UFB networks favour certain types of applications and service providers over others, or favour legacy styles of service delivery such as walled gardens over the evolving trend towards over the top delivery.

What this issue boils down to is a guiding principal of neutrality. Now it’s hard to know what if any principals have been guiding the development of the UFB architecture since I cannot find any in the few documents that CFH and the TCF have so far made public. (This may be radical to suggest, but I think CFH needs to be more open and engaging. Hey CFH, how about doing a blog post on what you think about neutrality and take it from there?)

What should be the guiding principal here? Well we need to ask what we are building the UFB for and what we want it to achieve, and I can’t speak for other people, so this is only my perspective, but it’s one I think most people who consider the UFB as a project of strategic importance for New Zealand’s future will agree with.

The Government should not be taking on all the risk of building the UFB simply so that established providers can continue to provide the same services they do today, or entrench their positions. I believe the purpose of the UFB, and any similiar national broadband programme, is to provide a neutral platform that allows retail service providers (RSPs) to compete equally on the basis of how good their products are regardless of their size. For this reason the UFB must aim to minimize its influence on what shape those products take. Thus distinctions must not be made between types of applications or style of service delivery. That must be decided by the consumers and the competition between the RSPs.

So with that in mind I was concerned when I heard that the current proposal is have two classes of service, a high priority class and a low priority class, as illustrated in the table below taken from the TCF UFB Ethernet Access Service Description.

Let me be clear. This is stupid. (Or I’m stupid and not seeing how this is going to be used.)

What does this mean? Well it will depend on how it is going to be used. First let’s take a look at these two classes.

We have a high priority class which has tight SLAs. This class is not colour-aware, meaning that it doesn’t support the concept of bursting above the committed information rate (CIR). Excess traffic over the CIR is dropped. I do approve of the 5 ms frame delay (latency) and 1 ms frame delay variation (jitter) SLA. But I disagree with the 0.1% frame loss ratio. 0.1% is too much for what should be absolutely committed capacity through the network. Instead this frame loss ratio should be closer to 0.01%, but couched with availability, which is supported by the MEF framework that the TCF are sort-of using here. That is, we should say that 99% of the time frame loss will be below 0.01%, but 1% of the time there may be some failure or event on the network that causes it to go higher.

The other class has pathetic, almost non-existent SLAs. It is basically a service where if there is available network capacity at that time the packet will be forwarded, otherwise it may be queued for a full second, and may never get through. There is no defined FD or FDV, and the FLR at 2% is absolutely ridiculous as anyone who knows anything about networking will tell you. This document gives the impression that this class is colour-aware, but actually it’s still a single-rate class as the CIR is always zero.

I really don’t know how they arrived a this, I can imagine there was a lot of debate at the meetings, and I recognise that design by committee can be difficult at the best of times.

Ok-ok, so hurry and get to the point, you’re saying.

To me, the above implies that the two classes of service will be carried in separate queues through the UFB networks. The high priority class is the only class offering a CIR and will carry applications such as voice and video that require committed bandwidth. The low priority class offers no commitment and will carry Internet, possibly with an atrocious quality of service if the SLAs shown are any guide.

So imagine you’ve subscribed to a triple play package from your RSP. You have one phone, and one TV, so your RSP orders a service from the local fibre company with:

• 150 Kbps of high priority bandwidth for voice, enough for one voice call since you have only one phone.
• 10 Mbps of high priority bandwidth for video, enough for delivering one HD stream since you have only one TV.
• 50 Mbps of low priority bandwidth for Internet — an arbitrary number for this example, and presumably this will vary depending on each RSPs packages, e.g. there may be a 10 Mbps package, 50 Mbps package, and 100 Mbps package.

The UFB network receives traffic from many competing RSPs and delivers it to many subscribers over shared links, so queuing and scheduling must be used at these points within the UFB network to provide the desired differentiation between the two classes of service. The indication is that the high priority class will be given bandwidth before the low priority class in a strict priority or weighted fashion. (This is a simplification of the issue sufficient to illustrate the point below, but it should be noted that the problem is actually much larger, requiring the use of H-QoS to ensure fairness between the different RSPs and their respective subscribers.)

Now coming back to our scenario. Consider we are watching a VoD stream from our ISP during peak hour, and this is an HD stream requiring about 9 Mbps of bandwidth (depending on how much action is going on). As we outlined above this is delivered in the high priority class of which we have 10 Mbps to play with, so we can be certain the stream gets through in a timely manner without packet loss and the associated skipped frames or artifacts.

But we get bored of this, so turn to an over-the-top (OTT) video service (such as Hulu, NetFlix, or YouTube) for entertainment. These are delivered over our Internet service, which as we outlined above has 50 Mbps of low priority bandwidth with basically no SLAs.

Now we find that our OTT video service is getting a lot of buffering because it is peak time and there is congestion on the UFB network. It turns out everyone in our neighbourhood has come home after work and started watching TV. All of these TV streams are being carried in the high priority class so getting the share of the bandwidth on the UFB network before our Internet traffic. Our OTT service suffers as a result.

Hang on, you say, you’re no longer watching the VoD stream, but you are paying for 10 Mbps of high priority bandwidth to carry it. So when you’re not using that 10 Mbps for VoD where does it go? And who decided a VoD stream from your RSP is to be treated differently to a VoD stream from an OTT provider? You’re right to ask that. When you aren’t using your 10 Mbps of high priority bandwidth it gets shared out amongst everyone elses low priority traffic — it does not get assigned to you, you see only a fraction of it, even though you’re paying for it. And who decided that VoD from your RSP is more important than VoD from an OTT provider? Apparently CFH and the TCF did back in February.

So what’s the solution? Well this is a complex subject and there are a few solutions. In order to select the best solution the guiding principals must be established and the detailed requirements analysed. One could easily end up writing hundreds of pages on the subject, so given I get no reward for that, I’ll jump to those solutions that are readily apparent.

1. The above two classes of service could be modified to make the low priority class dual-rate, supporting a CIR and EIR, and tightenting its SLAs, with the high priority class priced to discourage its use except for very low bandwidth and special applications. The UFB network will perform the necessary H-QoS to ensure fairness between the RSPs, i.e. between the aggregate of subscribers for each RSP, but not at a per subscriber level. The RSPs will be responsible for performing the H-QoS necessary to ensure fairness between each of their subscribers and appropriately prioritise their applications and if they choose to also OTT applications. (Some inventive RSPs will choose to give the consumer granular control over this through a portal.)
2. As above, but the high priority class is simply dropped because it serves little useful purpose, and simplification is better.
3. The UFB network performs per-subscriber H-QoS on behalf of the RSPs based on the purchased bandwidth profiles. Each subscriber is guaranteed the amount of bandwidth they actually pay for, but the H-QoS is configured with multiple queues in a mix of strict priority and weighted round robin fashions. The RSPs can then manage the markings of different applications to indicate the relative priority of them and which queues they should go into.

If anyone in charge is listening — please take another hard look at this. I am happy to eat my hat if I’m wrong, but this is an issue fundamental to the purpose of the UFB, and we want to get matters such as this correct up front, rather than trying to address it later on when there is no regulatory powers to do so (given the Government’s insistence on a 10 year regulatory holiday).

I’ve been playing with PostGIS recently, so I thought I’d start part two with a quick map showing the dwelling density of each area unit from the 2006 Census and the roads from Whangarei District Council. The area units shown correspond reasonably closely to the deployment area indicated by the map in the Crown Fibre fact sheet.

Equipment

A good OLT will support a minimum of 80 GPON ports, which at a 1:32 split ratio permits over 2,500 premises to be served off each OLT. This OLT must also be capable of at least two 10 Gbps uplinks, but a good one should give us the option to go to four 10 Gbps uplinks, e.g. if an OLT ends up with a lot of high bandwidth services on it.

The other piece of equipment is the aggregation switch, I like the term broadband service aggregator (BSA). The role of the BSA in our network is to aggregate connections from the OLTs and handover the wholesale Ethernet services to the retail service providers. A decent BSA will support at least 48 10 Gbps ports today with a roadmap to twice that (or in some cases four or more times that). We don’t need 1 Gbps ports for our OLTs, those really must connect at 10 Gbps, but it is conceivable that some smaller retail service providers will want the option to connect at 1 Gbps. That said, we can get over 40 such ports in a single slot so there is no concern about 1 Gbps port capacity.

There is no need to consider DWDM in this network because it is so small.

Coming back to the statistics from part 1, we have 16,851 dwellings, “2,000 business premises, and over 300 medical and other healthcare services, with over 20 schools,” for an approximate total of 20,000 premises. We can see this is going to be quite a small network: we’ll only need 10 OLTs, and two BSAs. In a FTTH deployment this equipment is the cheap part. Under 400,000 for each of our OLTs for a total of $4m. Our two BSAs aren’t fully populated, and should come in south of 500,000 for a total of $1m.

Deployment

Thanks to the Whangarei District Council releasing their GIS data on Koordinates we can quickly estimate the total linear length of our deployment. A query of the total length of all roads in the deployment area (in this case as shown in the image above) returns 278 km. At a minimum fibre must be deployed once down the length of every road, and at a maximum twice this. The actual requirement will fall somewhere towards the top end as of course it is almost always necessary to serve both sides of the road. (Some readers may be thinking they’d only run one distribution cable down the street, but you still need to drill across the street to deliver the drops to each premise, and in New Zealand our frontages aren’t much different to the street width so it typically works out as twice the length.) If we took the time and had the relevant per unit costs we could quickly build an estimate of the total deployment cost from high level designs of sample areas. But for our purposes a quick first order approximation will suffice, to get that let’s assume an average cost of $100 per meter to encompass the pits, pipes, cable, and closures. 2 × 278 km × $100 = $55.6m.

Topology

To get a quick idea of how our 10 km rule of thumb for the GPON range applies to the deployment area I’ll turn to Free Map Tools to plot the 10 km driving radius in various directions from the centre of Whangarei. Another useful tool (for those without access to their own GIS) is the Google Maps route planner, it has the nifty ability to plot routes and let you drag and adjust them while giving you the distance.

As you can see our 10 km GPON range comfortably encapsulates the entire deployment area and then some. From this we might conclude that everything could all be delivered from one central office, so should we just bang everything into one building and be done with it? To answer that we’d need to fully develop the outside plant architecture and validate our range estimates against the actual topology, understand growth, compare costs, and analyse the availability requirements.

Existing infrastructure will have a major impact on the cost comparison. Telecom for example might have an existing central office that is highly resilient and has a lot of useful life left in it. In this case it might make sense to centralise most if not all of the equipment. Telecom would also have existing fibre, of which much must be overbuilt (reducing its value to almost nothing), but some won’t need to be overbuilt, particularly where it connects an isolated suburb or centre. Looking at our map above the Onerahi peninsula (bottom right) might be an example of this. The existing cable would need to support the 2230 dwellings: at a 1:32 split that’s 70 cores. If it couldn’t then we might find the cost in this case to run a new cable outweighed the cost of deploying a cabinet to serve those suburbs.

The most important consideration of course is meeting the availability requirements. There is always a risk that any facility can be taken offline, particularly in a natural disaster. The question becomes how many premises can we accept going without fixed line services, for how long, and under what circumstances? A cabinetized approach in this case merely divides this risk down.

One thing to note is that if we do centralise everything into one building then we cannot offer fully redundant wholesale Ethernet services. This is only likely to be requested for a small portion of the commercial premises. So our choices are to design the PON network with this in mind in those likely areas, allowing us to deploy two ONTs off of two different OLTs via diverse paths, or to omit this and instead when diversity is required the retail service provider will have to use two dark fibre services that follow diverse paths back to their chosen POPs. In the case of Whangarei this latter option probably makes the most sense.

On this note, I do believe we need to switch our attitude. There is an assumption by many that the fixed line network provides lifeline services and must be the most resilient. Some people seem overly concerned about things like battery backups for ONTs. I believe we must shift our emphasis to ensuring our wireless networks are resilient and always available during emergencies. We already do have outages on our existing fixed line network. (I once needed to call 111 in a life threatening emergency but landlines in our entire neighbourhood had been down all day due to an overzealous digger a few blocks up the street. Fortunately mobile services were operational.) Can we expect to use the mobile networks in an emergency? I think there’s an important discussion to be had here about how much we want our different networks in New Zealand to share facilities, and what regulatory requirements wireless networks should have for availability and location reporting during emergencies.

Whangarei is the first town off the blocks in the ultra-fast broadband space race, with Crown Fibre selecting Northpower for negotiations back in September, then the ceremonious connection of a school a few days ago — a completely meaningless event as there is still no solid definition around the UFB that differentiates this from any other fibre connection in the country. But Whangarei is a nice place, so I thought I’d have some fun and look at it as a short case study.

Deployment area

Crown Fibre has released a fact sheet which includes the following map of the deployment area. The deployment will cover the urban area of Whangarei that they claim consists of a “population of nearly 52,000, including more than 2,000 business premises, and over 300 medical and other healthcare services, with over 20 schools.” It’s also important to know how many homes there are, so for that we turn to the 2006 census and find that there are 28,149 occupied dwellings in Whangarei District. The district actually includes a large rural area not currently covered by Northpower’s indicated UFB deployment, but we can refine this using the meshblock data to find there are 16,851 occupied dwellings in the indicated deployment area.

But what about the rest of Northland? There are plenty of smaller centres with reasonable populations, for example Dargaville with around 4,500 residents, or Kaitaia and Kerikeri with around 5,000 each. I think we should have a road map for these centres. This is where Telecom should have an advantage over other proposals, if only they hadn’t spent so much time trying to derail the project and had instead embraced it from the start — but I want to talk about that in a future post, so for now let’s move onto the network design.

Network basics

The network will use GPON to deliver wholesale Ethernet services for use by the majority of customers, but the physical network will also support wholesale dark fibre for use by service providers, enterprise customers, and big institutions. The optical distribution network (ODN) has a dedicated fibre between every premise and the local convergence point, also known as a fibre distribution hub (FDH). These are small passive cabinets, and there is usually one for every 200 to 400 premises. We locate the splitters in the FDHs because this gives us the optimal flexibility, future proofing, and manageability. Feeders run from the FDH back to the central office where the OLT is housed. The term central office typically creates visions of a large exchange building, but this is not the case in our network. We only have a small amount of equipment and small number of fibre terminations to house, so a large exchange building is quite simply overkill and would be a massive unwarranted cost. A roadside cabinet or portable telecommunications hut will be sufficient in most cases. The OLT is then connected back to the aggregation switch where handovers are made to service providers.

You may be wondering about how I arrived at the ranges shown on that drawing. The range between the aggregation switch (BSA) and OLT can be whatever is supported by the installed optics, and then much longer in cases where DWDM is used to provide transport. I use <70 km as a rule of thumb for discussing concepts. The range between the OLT and the ONT is a factor of the split ratio, the GPON type, and number of connectors and splices. The typical example I use to get my <10 km rule of thumb is described below.

To start with we must know our power budget, which is the difference between the launch power and the recieve sensitivity as described in ITU-T G.984.2. For class B optics we have a 26 dB budget and for class C optics we have a 31 dB budget. This budget gets consumed as follows.

• 3 dB reserve for cable breaks and repairs.
• 0.5 dB for each pair of mated connectors. We have a connector at the port on the OLT and another at the port on the ONT. Ideally in between these everything is fusion spliced to reduce faults and maintenance, but I’ll allow for another connector at the NAP in case pre-connectorised drops are being used. 3 × 0.5 = 1.5 dB
• 0.05 dB for each fusion splice. Let’s allow for 10 fusion splices between the OLT and farthest ONT. 10 × 0.05 = 0.5 dB
• 3.5 dB for a every 1:2 split. Higher ratio splitters are just trees of binary splits, so the formula is log2(n) where n is the split ratio. Let’s allow for a typical 1:32 split ratio. log2(32) * 3.5 = 17.5 dB

At this point we’ve accumulated 22.5 dB, leaving us with around 4 dB for class B and 8 dB for class C. The downstream operates at 1490 nm and the upstream at 1310 nm wavelengths. In G.652 fibre these have a typical attenuation of around 0.2 dB per km and 0.3 dB per km respectively. Of course we need bidirectional communication to make things work so we’re bound by the higher 0.3 dB figure. From this we can conclude a range of around 10 km for class B and around 25 km for class C.

I’ve used a 1:32 split ratio above, but the Crown Fibre fact sheet for Whangarei tells us that they’ll be using a 1:24 split ratio. This doesn’t entirely make sense as splitters work as a binary tree. The fact sheet says, “GPON is split 1:24 enabling up to 24 customers to receive 100Mbps downstream and 50Mbps upstream,” from which I’m inferring that they’ve simply said 1:24 because 2.4 Gbps × 1:24 = 100 Mbps, and 100 Mbps is some magic number for Crown Fibre. Of course this also doesn’t make sense as some customers will elect to take lower speed services than others, and GPON’s dynamic bandwidth allocation (DBA) means that you oversubscribe the bandwidth, but I won’t digress down that path now.

One benefit of a smaller split ratio is less attenuation, which means we get a longer range. Remember the splitters are a binary tree, so in the case of a 1:24 split, we could have a range to 16 of the premises of up to 10 km as described above, but for 8 of the premises we gain an extra 3.5 dB because we drop one split, giving us an extra 10 km range to those 8 premises. Conceivably we could use such a splitter scheme for example as follows, allowing us to reach some more distant premises.

But there isn’t any difference between that and the following. The truth is the splitter scheme can vary to suit the specific situation.

This is where I’ll leave part 1 for today. In part 2 we’ll explore some conceptual topologies for Whangarei.