Not All Data Is The Same: Rules For Data Integrity.

February 28, 2021
Procurement

-Ian Mackill

Not all data is the same. It might have come from the same source, but how it gets treated is vital. If a data company doesn’t have good data hygiene practices things can get messy very quickly, making it hard to understand the data or undermining your valuable analysis.

These are our rules for ensuring data integrity:

  1. Always know where the data came from → We always record the precise source of every record, so that our users can always go back to the original source and validate each one of our records.
  2. Always know when the data was collected → We don’t just record where we got the data from, we also know when we collected the data, so if the source changes, we can change.
  3. Never overwrite source data → We know that some of our data needs to be improved, if a date is incomplete or a better category can be added, in doing this we always add to the data, rather than overwrite the underlying data.
  4. Generate metadata → Our clients want to be able to filter data on attributes that we create, the language of a record for instance. We augment our data with useful metadata every time we gather a record.
  5. Handle duplicates with sensitivity → We see a lot of duplicates and some records that look like duplicates but aren’t. So we don’t provide a binary ‘on’ or ‘off’ analysis of duplicates, we look at eight key attributes and then score these to provide a good understanding of whether something is a duplicate.
  6. Matching needs manual checks → Entity matching is incredibly hard to get right, algorithms can help, but in the end, every match that isn’t an exact match needs to be checked, to make sure that a match is correct. That’s what we do, because the details matter and if we get a contract award wrong, then it can impact investment decisions.
  7. Be ready to highlight anomalies → We wished that some of the records we gathered were better formed, had better information, or just had the data that they were supposed to have. We have to accept that this isn’t always the case. So where things aren’t right, we don’t shy away, we don’t pretend that everything is rosy, we tell our users where the problems are, and let them budget what’s best.

At Spend Network, we know that data quality matters. We won’t tell you things you want to hear just to get a sale, we’ll tell you what we know. We want to build partnerships, not future problems.

If you’d like to know more about our data or our research services, get in touch.

April 14, 2021

UK Government Procurement Under Pressure

Yesterday the FT published an article, £19bn of UK Covid-related contracts awarded without seeking rival bids. The report reviewed the value of...
April 14, 2021

Selling to procurement: No One Cares About Your Product

Selling to procurement professionals is something that most people find frustrating, mainly because they are highly resistant to direct sales. Why? Well,...
April 7, 2021

8 Reasons Why Procurement Doesn’t Need Blockchain.

Blockchain is fundamentally a database, but rather than a database where one item is allowed to replace another, each change to the...
March 25, 2021

Procurement Transparency Suffers Under Covid-19

Government publishing of procurement notices has fallen significantly following the global spread of Covid-19. The total number of tender notices published globally...
March 20, 2021

South Africa, Kenya lead the way on African transparency.

Both South Africa and Kenya lead the way in procurement transparency according to our data, South Africa and Kenya publish more tender...
March 16, 2021

Missing Data Is A Known Unknown

There is a famous quote about the fragility of knowledge by Donald Rumsfeld, the hawkish US Secretary of Defence during the Iraq...
March 4, 2021

NZ Government Pharmaceutical Procurement Review

The New Zealand Government is taking steps to improve its procurement of national medicine supply through a review process. The Pharmaceutical Management...
February 28, 2021

The Problem With Frameworks

-Ian Makgill In my last post, I covered off framework agreements, and the advantages of using them for both government and suppliers.In this...
February 28, 2021

Do Framework Agreements Have Value?

-Ian Makgill Framework agreements are like umbrella agreements, and are usually made with a group of providers to supply a set of...
February 28, 2021

NSW Aims To Reserve Procurement Budgets For SMEs

We're always pleased to see governments around the world improving their procurement processes, by broadening opportunities for all types of business to...
February 28, 2021

Creating Synergy Between Politics & Procurement.

The need for administrations to act at pace is often at odds with the processes and procedures needed for good procurement. If...
February 28, 2021

Canada Launches Green Procurement

We are always pleased to see governments taking steps towards better procurement practices. Recently, the Canadian Government took a step forward for...
February 4, 2021

Post Brexit Procurement – What Will Change?

With the Brexit transition period officially behind us, it's worth considering the potential impacts of Brexit on Government procurement into the future....
February 28, 2021

Where Next For Data Led Procurement in Europe? A Discussion.

The They Buy For You  Project (TBFY) concluded on 31 December 2020. To mark the occasion, we look back at three years...
February 28, 2021

Spending $400bn – A Demanding Task For Biden.

-Fiona Hunt As my colleague Ian wrote this week, governments are increasingly looking to procurement to deliver better social outcomes. President Biden...

Newsletter

Compelling research, insights and data directly into your inbox.

Recent media stories

Search