Colour Stripe
Mobile Colour Stripe

Making Data Great With Our Innovation Challenge Winners (Part 1)

In this post, we take a detailed look at the first two solutions to come out of the Make Data Great Innovation Challenge

The challenge, launched in November 2020, presented 5 challenge statements varying in focus from data discovery to data update notifications and personalisation. 

From many exciting submissions, 4 solutions were chosen to progress. Our first 2 solutions are now live on the Open Data Hub - and ready to make our data great!  

Data Update Notification Service by Arid Systems

One of the solutions comes from Arid Systems - lead by Bill Wood, Director - which is a start-up that works on solutions for making real-time data easily accessible and shared. 

“Our bread and butter is the so-called “Internet of Things”. It’s a tremendously exciting field and is leading to an explosion of real-time data which will become increasingly important to transport, along with almost every other aspect of our modern world”, says Bill. 

The solution provides Open Data Hub users with immediate notifications of updates to datasets. This will be of particular use to our regular users, as well as anyone who’s waiting for the latest release of an ongoing dataset such as Opal Tap-On/Tap-Off. 

“The service provides alerts in real-time. For example, an alert saying “the transport timetables (which may underpin your application) have just been updated, so come get them!” This unburdens the users of the data of that discovery process, and ensures they won’t be caught out with outdated data – allowing them to continue to focus their energies upon innovating with the data itself.”

“The core challenge we faced was knowing what had changed and how. Think about it this way: you are a keen follower of supermarket pricing, perhaps you buy for a large family. But how to stretch the budget? You can periodically check the weekly circulars, wander the aisles of your local stores, noting prices and comparing to your notes from last week. But the core challenge remains – when and where to check, and how often to do so?”

“That’s how things are with most open data sources. Take public transport timetables for example – or any open data that is subject to change. You incorporate this information into your application. Prior to the implementation of our Real-Time Service, like the shopping challenge, the only way to know data had been updated was to keep downloading it and comparing it to last time. But again, when and how often?”

So, how does the solution work? 

“We use cloud-based computing which is tightly coupled to the Open Data Hub’s data repositories. The current Open Data Hub infrastructure is Drupal-based and comes with various API’s for accessing the database which underpins it. We continuously scan the database and perform some fairly sophisticated analysis around the metadata to ascertain that something has changed, and how. We then translate that into understandable and meaningful indications for the users of that data.”

This is the first time Arid Systems have participated in a TfNSW Innovation Challenge, and saw Make Data Great as a win-win opportunity. 

“We were using the Open Data Hub as a source of real-time data to assist in the development of our solution, and came across some shortcomings. It seemed fairly obvious that if we were challenged, other app developers might be experiencing the same.” 

“Innovation is a messy business! If the process were straightforward, it would hardly be innovation. The biggest challenge for us was that while we understood the problem and had a sense for the solution, until we launched into it we really didn’t know what it would take, nor could we see the hurdles we would face. The effort turned out to be considerably greater than we envisaged, and the solution somewhat adapted, however looking back we feel that we have solved the fundamental challenge which we set out to address.”

“The team at Transport for NSW have been tremendously supportive of our work, and truly understand the challenges around innovation. For that we are thankful. We remain enormously impressed that they continue to push on the existing open data boundaries, and are so proactive in leading with a continuous, innovation-centred process of improvement.”

To set up your Data Update Notification Service, login to the Open Data Hub and under My Account you will find the Subscriptions option. By creating a subscription you can be notified whenever the dataset you are interested in is updated.

We have also added a new Data Updates page, which can be found under the Developers menu, that provides an overview of all updated data over the past month. A number of filters are available to narrow down what is shown. 

Connect with Arid Systems via their LinkedIn page

Odie Bot by Data Driven

The next Make Data Great solution is Odie - a chat bot created by Data Driven, who is a Microsoft Gold Partner specialising in advanced analytics consulting and building modern data platforms for its customers. 

“We deliver innovative data and AI solutions to help organisations build a data-driven culture and empower their business decisions with insights”, says Sofia Oropreza, Sales Director. 

“Entering the NSW Transport Innovation Challenge made sense for us as we have deep experience in the Government Transport domain and understand the challenges around finding useful data.”

“We enjoy solving business problems with technical solutions in the simplest way possible and thought that Odie Bot was an elegant solution to the problem and would be useful for Open Data Hub users.”

Odie is an embedded website chat bot that helps users easily find datasets, answer questions, and more. 

“Odie helps users find the data they are looking for in a quicker, friendlier way, whilst reducing support costs and effort. Odie is an interactive, conversational bot embedded in the Open Data Hub website acting as the first line of interaction between users and the business.”

“When a website user first opens the page, Odie welcomes them and helps them answer common support queries as well as surface lesser known datasets. More satisfied users = more engagement and more innovation.”

The Open Data and Innovation team will be able to continually train Odie to serve our customers more effectively. Odie understands misspelt words and similar phrases, improving the user experience for our customers. 

The chat bot is available 24/7, and improves discoverability of unknown datasets which encourages innovation as less popular datasets are surfaced.

“Odie uses Natural Language Understanding combined with fuzzy search logic to try and understand the user's request as closely as possible. This allows users to converse with Odie in a non-technical, conversational way to find the data solutions they want.”

“The bot leverages the Microsoft Bot Framework and Azure serverless PaaS services, with its "brain" built using Azure Cognitive services such as LUIS for natural language understanding to help better understand the user's intent, Azure QnA maker to create a conversational question-and-answer layer over existing data, and Azure Search for full text and filtered queries.”

“The Innovation Challenge process was easy to follow, and the Open Data Hub provided a good overview of the challenges they were trying to solve and the key areas to be addressed. In addition, the Open Data Hub team was always available during the project.”

Odie is now live on our website (desktop only), ready to answer your questions and help you find datasets. You can start getting help from the Odie chatbot by clicking on the speech bubble in the bottom left hand corner of the Open Data Hub.

Head over to the Data Driven website to learn more about what they do and read more customer success stories. 

We’ll be sharing our final two solutions in another blog post, so stay tuned via our Twitter page or mailing list!