Insights

Development of a Covid-19 data repository with Watson Knowledge Catalogue

Development of a Covid-19 data repository with Watson Knowledge Catalogue main image

Ana is a Data Scientist who is working on a challenge in Emergent Alliance and regularly searching for Covid-19 related datasets. She came across the work of Emergent Risk-Pulse Index challenge during a Town Hall event and recognized multiple points of relevance in terms of data and analysis to her current work. What if she could access a prepared Covid-19 data catalogue easily to explore and make use of the data in her project, and even add new datasets to the catalogue and share with others?

To support Ana and all data scientists in Emergent Alliance and to enhance collaboration within projects, we built a data repository for the Risk-Pulse Index project using Watson Knowledge Catalogue (WKC). This includes a categorized metadata repository that allows searching the assets, access, and reuse them via REST APIs or directly in an analytics project in IBM Cloud Pak for Data. Also, one can explore related assets assigned to metadata and their relationships.

Searching assets

When publishing data and assets (including models and notebooks) in WKC, we can assign tags and business terms that would allow finding related assets. Business term is the definition of key business information that is used in day-to-day business operations and analysis. They can be organized in hierarchies using type relationships as well as in high-level categories. In the current repository, we have created five data catalogues representing the categories of our assets: Health, Economy, Stringency, Mobility/Media, and Geospatial where currently over 60 data assets are published. As an example, some of the data assets in the Stringency category are assigned with ‘Stringency Index’ business term shown below.

A graphic describing the business terms

The figure above shows that ‘Stringency Index’ is a part (relationship) of ‘Covid19’ business term and belongs to the ‘Stringency’ category.

Similarly, we can assign Data Classes to the columns of a data set. Data classes describe the type of data contained in data assets, such as data fields or table columns, for example, city, account number, or credit card number. For example, ‘Oxford Stringency Index value’ is a data class related to ‘Stringency Index’.

We can also use matching methods to specify how to identify data class assignments automatically when profiling dataset columns. Data classes are necessary to mask data with policies.

Exploring content

Assigning business terms in the right way allows us to explore the related content. The table below shows a view on the asset type, name of the catalogue, and the last modified date.

A table showing the stringency index

Profiling data

After finding the related content, we can then view columns of a dataset and get general statistics via the ‘Profile’ feature. The profile of a data asset includes generated metadata and statistics about the textual content of the data. Thanks to this feature, we get information on data classes and decide whether a dataset is needed, then proceed with ‘Add to project’ in Cloud Pak for Data and use the dataset easily for further analysis.

A dashboard showing details on the data profile

Exploring relationships

To view the relationships between metadata and assets and to better understand how data is connected, we created a knowledge graph. In the interactive graph, we can expand nodes and display relationships in a category of interest. As shown in the image below, we created six sub-categories under “Covid19“ with each a set of business terms and datasets is associated.

A relationship graph showing the relationships between metadata and assets

Tracking data repository

Through a collection of Watson Data REST APIs associated with WKC, we created a script that is scheduled as a daily job to monitor and query data assets already published in WKC. This job enables to manage catalogues efficiently and detect data assets that are not yet categorized. A word cloud of metadata captured from catalogues is shown below.

A word cloud output

 

Main Takeaway

We used part of WKC capabilities to build a data asset repository of the use cases in the Emergent Risk-Pulse Index project. We showed how to get benefit from assigning metadata to assets and explore content and relationships through a knowledge graph. This work can be extended to build a data and asset ecosystem in Emergent Alliance that relies on standardized metadata and facilitates collaboration within data science teams. Future work can address automated metadata generation, data masking, and monitoring data quality.

Author

Mehrnoosh Vahdat is a Data Scientist with IBM Data Science & AI Elite team where she specializes in Data Science, Analytics platforms, and Machine Learning solutions.

Special thanks to Klaus Paul, Sarah Boufelja, Anthony Ayanwale, Shri Nishanth Rajendran, Álvaro Corrales Cano, Kareem Amin, Emma Tucker, and Alex Lavrov for their valuable contributions and support in building the data repository.

We are a team of data scientists from IBM’s Data Science & AI Elite Team, IBM’s Cloud Pak Acceleration Team, and Rolls-Royce’s R² Data Labs working on Regional Risk-Pulse Index: forecasting and simulation within Emergent Alliance. Have a look at our challenge statement!


Fatal error: Uncaught Error: Call to undefined function WP_Optimize() in /var/www/wp-content/plugins/wp-optimize/cache/file-based-page-cache-functions.php:178 Stack trace: #0 [internal function]: wpo_cache('<!doctype html>...', 9) #1 /var/www/wp-includes/functions.php(5107): ob_end_flush() #2 /var/www/wp-includes/class-wp-hook.php(303): wp_ob_end_flush_all('') #3 /var/www/wp-includes/class-wp-hook.php(327): WP_Hook->apply_filters(NULL, Array) #4 /var/www/wp-includes/plugin.php(470): WP_Hook->do_action(Array) #5 /var/www/wp-includes/load.php(1097): do_action('shutdown') #6 [internal function]: shutdown_action_hook() #7 {main} thrown in /var/www/wp-content/plugins/wp-optimize/cache/file-based-page-cache-functions.php on line 178