Some thoughts on data and smart analytics

I was asked to do some briefing notes about data and smart analytics for a strategy paper. I thought I’d share them here.

High-quality government data is important for:

• measuring government effectiveness
• holding government to account
• helping private, public and third sector organisations to innovate, using open data

“In a data-driven government, actionable information for all critical decisions is accessible when and where needed. The opportunities for better, smarter government through optimal
use of data are clear as are the challenges.” (Lutes)

A data-driven organization is one that understands the importance of data. It possesses a culture of using data to make all business decisions.

“…a data-driven government is one where, for all critical decisions, actionable information is available when and where needed. The benefits are almost incalculable.
A few examples include:

• Sounder governance and control
• Optimized fraud and error detection, mitigation and prevention
• Improved services based on insights gained from those being served
• Improved efficiency through intelligence networks, which can lead to reduced costs
• Improved public perception of the agency” (Lutes)

Data sources

• Transactional data – captured as part of business of delivering government services
• Interaction data – anonymised data about user’s interactions with government – digital analytics (GA), surveys, feedback, user research
• Datasets: Culture, Science, Finance, Statistics, Weather, Environment, Mapping

Shareability and standards

Many datasets can be published as Open Data and visualised as dashboards.

Data should be more easily shared between departments – to support a single user view and to provide canonical sources of data.

Requires standards and registers.

The analytics value pyramid

Government has loads of data, but it has traditionally been used for operational purposes, then stored, forgotten or deleted when legal limit of life is reached. Fraud and error detection has been a big driver for analysis in government. Some definitions (Gartner IT Glossary):

Descriptive analytics

Is the examination of data or content, usually manually performed, to answer the question “What happened?” (or What is happening?), characterised by traditional business intelligence (BI) and visualizations such as pie charts, bar charts, line graphs, tables, or generated narratives.

Diagnostic analytics

Is a  form of advanced analytics which examines data or content to answer the question “Why did it happen?”, and is characterized by techniques such as drill-down, data discovery, data mining and correlations.

Predictive analytics

Describes any approach to data mining with four attributes:

1. 1. An emphasis on prediction (rather than description, classification or clustering)
   2. Rapid analysis measured in hours or days (rather than the stereotypical months of traditional data mining)
   3. An emphasis on the business relevance of the resulting insights (no ivory tower analysis)
   4. (increasingly) An emphasis on ease of use, thus making the tools accessible to business users.

Prescriptive Analytics

Is a form of advanced analytics which examines data or content to answer the question “What should be done?” or “What can we do to make _______ happen?”, and is characterized by techniques such as graph analysis, simulation, complex event processing, neural networks, recommendation engines, heuristics, and machine learning.

 

As we move towards 2030, government needs to use descriptive and diagnostic analysis of data more effectively, but the real transformation is in using predictive and prescriptive.

Moving forward → DataOps (Data Operations)

Identify, combine, and manage multiple sources of data

• Know what data is available. Both the range of sources and the amount of data you can collect has increased by orders of magnitude
• It’s increasingly connected – transactions, interactions, and, increasingly, sensors are generating data
• Eliminate silos of data
• Tools for transforming data
• Build advanced analytics models for predicting and optimizing outcome

The most effective approach is to identify an opportunity or challenge and determine how an analytics model can help solve it. In other words, you don’t start with the data—at least at first—but with a problem.

Culture

A data-driven organisation is one that understands the importance of data. It possesses a culture of using data to make all business decisions.

Data professionals must understand what decisions their business users make, and give users the tools they need to make those decisions.

Data professionals need to beware of cognitive and unconscious bias in their analysis, tools they use and create and in their data story-telling.

Colleagues need upskilling to know what good looks like, to ask the right questions and challenge and to self-serve accurately.

DataOps definition: “DataOps is an automated, process-oriented methodology, used by analytic and data teams, to improve the quality and reduce the cycle time of data analytics. While DataOps began as a set of best practices, it has now matured to become a new and independent approach to data analytics.[1] DataOps applies to the entire data lifecycle[2] from data preparation to reporting, and recognizes the interconnected nature of the data analytics team and information technology operations” (Wikipedia)

Outcomes

• Single View of operational data, through shared and transformed data
• Single View of User, through shared and transformed data, subject to privacy, data protection and ethical requirements
• For all critical decisions, actionable information is available when and where needed.
• A few examples include: (some from Clark)
  • Sounder governance and control
  • Optimized fraud and error detection, mitigation and prevention
  • Improved services based on insights gained from those being served
  • Improved efficiency through intelligence networks, which can lead to reduced costs
  • Improved public perception of the agency
  • Data-led investment decisions

What is the remit of digital services in the context of a proliferation of data across society?

Shouldn’t we be just talking about government services, rather than digital services now?

Need to consider the original Government as a Platform principles – that government builds the data infrastructure so that third parties can provide the most relevant services (that may include a government service element) to users – dependent on context, situation, location etc. Third parties may well have a richer picture of users needs and context – eg having a baby.

How does this affect the “right place” for services to be delivered, in or out of government?

As above – need to consider the original Government as a Platform principles – that government builds the data infrastructure so that third parties can provide the most relevant services (that may include a government service element) to users – dependent on context, situation, location, life event etc. government.

Real time analytics into policy development, operation and iteration

Definition of Real-time analytics is the use of, or the capacity to use, data and related resources as soon as the data enters the system.  The adjective real-time refers to a level of computer responsiveness that a user senses as immediate or nearly immediate. Real-time analytics is also known as dynamic analysis, real-time analysis, real-time data integration and real-time intelligence. (SearchCRM).

For some applications – eg fraud detection, real-time is very valuable. But for others, less so; and scoping real time might add to costs and increase blockers. I’d recommend changing this to Actionable analysis informs policy development, operations and iteration

References:

Clark, Curtis  Data Driven Government – Supporting an Economic Vitality Vision…

Gartner IT Glossary

Lutes, Terry Data-driven government: Challenges and a path forward

SearchCRM Real time analytics

Thusoo, Ashish and Sen Sarma, Joydeep, Creating a Data-Driven Enterprise with DataOps, nicely summarised in a slidedeck by Nicky Zachariou

Wikipedia on DataOps