Examples of technical architecting at the BBC

These are some examples of the projects I've architected whilst at the BBC:

apps/ifl application suite (2001-2007)

I created a suite of highly reusable modperl applications which are used extensively across bbc.co.uk. They are designed to be easy to set up and configure by client-side developers requiring no central administration and share common features such as configuration formats, parameter names and templating language. They are completely reskinnable with templates (a choice of templating languages is available) and can be used as web services for example with AJAX or Flash clients using a variety of data formats such as XML, JSON or URL-encoded. For each application there is a user guide and example site for client-side developers, unit test and benchmark scripts, automated regression tests monitored via Nagios and performance graphing via RRD. All this is pulled together on an intranet page for each application which also dynamically queries the version numbers of the application released to each environment.

Gardening Calendar (2007 launch)

This is a personalised website and email system which allows different content to be delivered to different users depending on what they have in their garden and what needs doing at the time of year. Once launched, the system is designed to be run by journalists without intervention by technical staff and is designed to scale to hundreds of thousands of active users.

Recipe Search (2001-2007) and Plantfinder (2002-2007)

These are two of the BBC's most popular databases of content. Both have content management systems which are tailored to the workflows of the editorial teams (e.g. they get emails reminding them when they need to renew rights) and publish content to bbc.co.uk in a form which is optimised for high traffic searching and browsing. The searches combine metadata querying, free-text searching and some fuzzy matching to tolerate misspellings of tricky plant Latin names.

Blast showcase (2006-2007)

This is a database-driven site with automated ingest and processing of user-submitted images, audio and video. Users may also tag and rate content which is reflected in the ordering of the gallery and tag cloud. All the server side software components were built by my team including all the image and A/V processing.

FLIP (2001-2007)

I developed a content management system which is used to manage sites such as GCSE Bitesize, History and Lifestyle. The system consists of a desktop GUI application written in perl using WxWidgets which talks to a server application (also written in perl) using SOAP. Content is typically authored in XML and published via XSLT transformations, though the system is able to support arbitrary content types and transformations as required. I wrote much of the first version myself and managed its evolution through 6 years and 3 changes of lead developer.

Springwatch survey (2006)

Members of the public could fill in a form on the website or SMS in (in which case their location was automatically triangulated) when they first sighted one of six key species. Over 100,000 data points we delivered to the UK Phenology survey to help scientists get a better understanding of how Spring is changing and what needs to be done to help wildlife thrive in the UK.

Coast (2005)

An SMS service for coastal walks. The data pipeline architecture that had been used successfully on a number of similar projects was based on several cron-driven processes which had an end-to-end latency in the order of a couple of minutes which was too long for this service. I therefore designed a solution that reused many existing components but with a new daemon front-end to the data pipelines was able to turn SMS messages around in a few seconds.

Death in Sakkara (2005) and Death in Rome (2006)

These episodic web adventure games were promoted from the BBC TV series. In both cases a Flash front-end was developed by a 3rd party whilst I designed the server-side storage and email broadcast components that would interface with the Flash client and managed their development.

HTML Newsletter system (2005 - 2007)

At the end of 2006 this was sending over a quarter of a million emails a week at minimal cost by using spare overnight capacity on the BBC email relays. This involved designing and managing the development and optimisation of the mail generation and sending infrastructure as well as providing a user-friendly interface for editorial staff to author emails that render reliably in a very wide range of email clients. Examples: food, gardening, health, science + nature, blast, timewatch.

A Picture of Britain (2005)

A high-profile photography competition promoted on prime-time TV. Images uploaded from the website were automatically processed (transcoded, resized, stretched for TV pixels, gamma corrected etc) for delivery via bbc.co.uk, Interactive TV and big screens in public places around the UK. The top rated images were fed to a broadcast graphics system which was remotely controlled to play out possibly the first entirely user-generated TV channel with all content submission and selection done entirely by the public.

Fat Nation - The Big Challenge (2004)

This was a personalised fitness programme delivered by the web, email and SMS where the difficulty of tasks adapted over time depending on how well you did on previous ones. The web interface was built by a 3rd party whilst the back-end logic and components were designed and built by my team, so providing good quality documentation and a faithful integration test environment was a key part of this project. Another was developing a system that could be used for personalisation across platforms whilst being secure (e.g. not allowing someone to maliciously run up your mobile phone bill).

Chelsea Flower Show (2003, 2004, 2005, 2006)

The vote for the Nation's favourite garden over the last few years has spanned SMS, IVR (telephone), Web and Interative TV (Digital Cable and Digital Satellite). Fast turnarounds for announcing the results combined with some "imaginative" attempts to cheat have lead to increasingly sophisticated vote security measures being added over the years.

Psychology surveys and tests (2004-2006)

I developed a toolkit for capturing, filtering and collating data that can be set up by technical managers without needing programming skills. This has been applied to create numerous surveys on the science and nature site such as the SexID survey in which over half a million people participated generating a huge dataset for scientific analysis.

Brassed off britain (2004)

Members of the public could vote on their most annoying industry via the web, SMS, Digital Satellite, and IVR telephony. As votes came in during the TV show, demographic breakdowns were generated from the database and fed into a broadcast graphics system to generate images on live TV.

Test Your Pet (2004)

This involved close work with scientists, statisticians and television producers to provide advice on collecting data online as well as designing the solution and managing the build of the data collection service. Over 40,000 surveys were collected and analysed statistically which revealed some scientifically interesting results - notably that old dogs really are much less able to learn new tricks than their younger counterparts.

The Big Read (2003)

A TV call to action to nominate your favourite book generated over a quarter of a million free-text submissions (with more misspellings than you could ever dream of), all of which had to be totalled up in a few days by a couple of editorial staff. A number of software components were developed to help automate the task, stripping and cleaning data and allowing editorial staff to quickly combine groups of submissions that were different ways of expressing the same thing which then formed a rule for automatically grouping any further submissions. Once the nominaton phase was completed a vote followed for the Nation's favourite book which as well as multiple platforms involved measures to allow users without their own PC to cast votes from public (e.g. library) computers without opening up the vote to abuse by cheats.

Great Britons (2002)

This BBC1 TV series generated a level of vote traffic higher than anything that had been seen before in the BBC which overloaded all the existing data capture components. Between programmes in the series, I proposed and implemented an alternative approach for data capture that could recieve data as fast as the webservers can serve pages which is still in use today as a mechanism for handling large surges in data submission traffic after a prime-time TV programme. As well as attracting a lot of votes, this also attracted lots of people trying to rig the vote. I devised a number of tactics to filter the spoof votes out from the "real thing" which have subsequently been adopted broadly across the BBC.

TechNet (2001-2007)

I created the departmental technical intranet site which is now one of the most comprehensive technical resources in the BBC and is very highly regarded. All documentation is held in a CVS repository and automatically published to the webserver on check-in. Content is bare-bones XHTML and is skinned dynamically using a caching XSLT server I developed. Much of the documentation I originally wrote myself and is augmented with nightly feeds of infrastructure information, stats from access log digests, POD documentation from the software repository, NaturalDocs from the JS repository etc.

Database publishing toolkit (2001-2007)

I developed a reusable toolkit for template-driven publishing from relational-tabular data sources including databases, excel spreadsheets and delimited text files. This has been used to build dozens of publishing solutions including the Schools programme guide, Learning Zone schedules, Recipe Search and Plantfinder.