Skip to main content

Foreign, Commonwealth and Development Office: Harnessing the power of the cloud

How we built a bespoke data platform for the FCDO, with a cloud-based infrastructure that has enabled a step-change in the department's ​​advanced data science capability.

The Foreign, Commonwealth and Development Office (FCDO) relies on a vast wealth of data from across the globe to inform its decision-making. In a fast-changing world, the department’s Open Source Unit sought a tool equal to the task of aggregating, analysing and visualising data from a diversity of sources. Nothing that was available ‘off-the-shelf’ met its unique and exacting requirements.

We partnered with the FCDO to build a bespoke data platform capable of the rapid, automated ingestion of a wide range of data sources, including BBC Monitoring, diplomatic social media accounts, and economic indicator data. As the data sources are ingested, the platform normalises, processes, stores and indexes them, preparing them for easy interrogation by people ranging from non-technical civil servants to expert data scientists.

Laptop with insight platform screenshot

In a constantly shifting geopolitical landscape, the platform allows the FCDO to respond rapidly, drawing in new data to deliver the deep insights the department requires to make the right decisions in the face of world events. The innovative solution we delivered combines effortless configurability, frictionless usability and all the benefits of serverless architecture.

Cost-effective serverless architecture

The platform’s serverless architecture was designed to be secure from the outset, whilst allowing for easy expansion and development as the project progressed. As well as providing flexibility, the event-based nature of the pipeline design provides seamless scalability and resilience against failures.

The consumption-based pricing model for serverless technologies also makes this an exceptionally cost-effective design for the ingestion process.

The cloud-based infrastructure enabled a step-change in the FCDO’s ​​advanced data science capability. Modern ‘big data’ tooling, techniques and frameworks are now available to the department as a result, accelerating its data science initiatives.

The process of building, testing and deploying new features is made much easier thanks to this serverless architecture and its Continuous Integration/Continuous Deployment pipeline. In addition, the architecture allows different kinds of data processing to be decoupled from each other so that new forms of analysis can be added to the platform’s feature set with no impact on overall performance.

Easy extensibility and rapid data ingestion

The serverless architecture underpins the ease and speed of data ingestion and processing.  Data sources can be integrated in hours rather than weeks. Once ‘plugged in’, data sources remain separate but funnel down into common sections of code that meet the platform’s requirements. A small amount of configuration is required before the new data source goes live. None of this requires deep, specialist knowledge. After an hour’s training on the platform, FCDO users are able to add new data sources themselves, ready for analysis.

Data ingested into the platform is stored in a cloud-based Elasticsearch database which supports intuitive text-based searches, using simple parameters to filter data. The processing of the data includes information extraction techniques, including Named Entity Recognition—the identification of individuals, organisations and locations—and Sentiment Analysis. We met this requirement by implementing a free, Python-based open-source library for Natural Language Processing called spaCy.
 

Monitor with insight platform screenshot

Thanks to the speed of ingestion, the FCDO is able to get new data products off the ground quickly and build momentum within the department. For example, the Diplomatic & Development Performance Analysis programme was able to ingest a variety of data sources rapidly, including V-Dem democracy indicators, economic indicators, and the Global Terrorism Index. These enable the programme to assess regional development and make data-driven decisions about development programmes and broader diplomacy.

Socialising data to improve insights

To support the socialisation of data within the FCDO, we combined the expertise of our UX designers and front-end developers to develop a simple, search-based interface, with the resulting data presented in readily configurable dashboards implemented using Vue.js and D3. You can choose from a variety of dashboard components, including bar charts, line charts and word clouds, as well as choropleth maps shaded according to the range in which the aggregated statistic of interest falls. 

The framework we developed for the dashboarding tool makes it simple to add new template components in a matter of hours, with no significant development work required—everything from standard chart types to highly customised widgets for the display of complex data. In later iterations, we added time-series trend analysis capabilities to the platform.

Uniquely, our solution allows this trend analysis to be performed on an array of heterogeneous data sources, with the results overlaid on each other. Off-the-shelf business intelligence tools are incapable of performing this kind of bespoke, comparative analysis of multiple data sources within a single interface.

We designed the interface to support the widest range of requirements, from the most straightforward of search queries all the way to complex interrogation by experts. Our aim was to ‘hide the wiring’ from the user; regardless of the complexity and variety of the data sources ingested, the platform’s user-friendly interface remains the same. Importantly, the front-end and back-end are not tightly coupled in the architecture we designed, affording long-term flexibility to the FCDO in developing its robust new data infrastructure according to the department’s ever-evolving needs.