Executive Summary

The key numbers reveal the scope and depth of the six frameworks at a glance.

The AI development paradigm is shifting from Model-Centric to Data-Centric. As cutting-edge models become increasingly commoditized, the competitive edge now lies in the quality, richness, and integrity of data.

1.1Why Data Quality Matters

Problems inherent in data are not mere technical glitches. They are systemic risks that can lead to model failure, reputational damage, and regulatory violations.

• ▸ Social Bias: Latent biases embedded in data produce discriminatory outcomes
• ▸ Labeling Errors: Inaccurate annotations degrade model performance
• ▸ Data Drift: Shifting data distributions over time erode performance
• ▸ Ethical Blind Spots: Lack of ethics in data collection and use creates societal harm

The data quality journey begins with transparent and comprehensive documentation. Without clear information about how a dataset was created, its characteristics, and its limitations, quality cannot even be discussed.

2.1Datasheets for Datasets

Proposed by Gebru et al. in 2018, this concept drew inspiration from electronic component datasheets to create a standardized documentation framework for ML datasets. It represented a philosophical shift: redefining datasets not as objective raw materials, but as socio-technical artifacts shaped by human judgment.

The framework centers on five key areas of inquiry.

• ▸ Motivation: Who created it, and why?
• ▸ Composition: What data is included?
• ▸ Collection: How and where was the data gathered?
• ▸ Preprocessing: What cleaning or transformation was applied?
• ▸ Uses: What are the intended and prohibited use cases?

2.2Google Dataset Cards

Google evolved the academic Datasheets concept into a structured and flexible toolkit tailored for large-scale technology organizations. Through the Data Cards Playbook, it embeds transparency into organizational culture.

The framework is organized around four core modules.

• ▸ Ask: Define the scope and criteria for transparency
• ▸ Inspect: Generate metadata for the dataset
• ▸ Answer: Complete the card using standardized templates
• ▸ Audit: Evaluate the dataset's impact

Processing data at scale demands more than qualitative documentation. It requires quantitative and automated methodologies.

3.1IBM's Seven Data Quality Dimensions (DQAI)

IBM adapted traditional enterprise data quality management principles for the AI lifecycle, creating a measurable trustworthiness framework. It defines specific metrics for each of seven dimensions and supports automated measurement through software and APIs.

The seven quality dimensions are as follows.

3.2NVIDIA NeMo Curator

NeMo Curator reframes data quality not as a one-time validation step but as a continuous, automated pipeline problem. Optimized for processing the massive volumes of unstructured data used in deep learning, it is released under the CC BY 4.0 license.

Its core capabilities span four areas.

• ▸ Automation: End-to-end automation of data download, cleaning, and quality filtering
• ▸ Multimodal: Support for text, image, video, and other modalities
• ▸ Deduplication: Semantic deduplication and data mixing
• ▸ Synthetic Data: Generation of synthetic data to address identified weaknesses

The central philosophy behind NeMo Curator is the data flywheel. Model feedback drives data improvement, and improved data in turn enhances model performance, creating a virtuous cycle.

Data quality must be addressed beyond individual organizations, at the level of industry standardization and international governance. Technical tools alone are insufficient; community-driven benchmarking and international principles are essential.

4.1DataPerf (MLCommons)

DataPerf is an initiative to shift the ML community's competitive focus from model-centric to data-centric methods. It drives data-centric algorithm innovation through public leaderboards.

Its challenges cover four key areas.

• ▸ Dataset Selection: Identifying the optimal data subset
• ▸ Dataset Cleaning: Prioritizing noise and error removal
• ▸ Dataset Acquisition: Optimizing strategic data procurement
• ▸ Adversarial Examples: Uncovering model failure modes

4.2OECD.AI Principles

The OECD.AI principles establish the highest-level international standards for trustworthy AI. They function as an "ethical and legal API" that bridges technology with societal expectations.

The five value-based principles are as follows.

• 1. Inclusive Growth: Benefits must reach all members of society
• 2. Human-Centered Values: Respect human rights and prevent bias
• 3. Transparency: Data provenance and processing must be understandable
• 4. Robustness/Security: Defenses against adversarial attacks must be in place
• 5. Accountability: Clear lines of responsibility must be established

Each of the six frameworks brings its own philosophy and approach. The real power lies in using them together to build a comprehensive data quality management system. The comparison table below summarizes each framework's core focus, key outputs, and approach.

5.1A Five-Step Integration Strategy

To adopt these frameworks effectively within an organization, a top-down approach works best: start with macro-level principles and progressively move toward specific tools in five steps.

• 1. Top-Level Governance: Establish an AI ethics charter based on OECD principles
• 2. Transparency: Mandate documentation using Google Dataset Cards
• 3. Quantitative Measurement: Set a structured data baseline with IBM tools
• 4. Automation and Scale: Process large-scale unstructured data via NVIDIA pipelines
• 5. Performance and Innovation: Run internal DataPerf-style challenges

Moving beyond theoretical analysis, applying these frameworks in practice requires diagnosing the current state and building a phased roadmap for improvement.

6.1Data Quality Maturity Model

An organization's data quality management can be assessed across four maturity levels.

6.2Multi-Layered Data Quality Strategy

Starting from Why and progressing to How Well, the strategy is built across four layers.

At Pebblous, we are focused on solving data quality problems at the foundation of AI systems. The six frameworks analyzed in this report directly intersect with our core business areas.

7.1DataClinic: Data Quality Diagnosis in Practice

DataClinic is a service that diagnoses data distributions and automatically measures quality metrics. It implements IBM DQAI's seven dimensions (accuracy, completeness, consistency, and more) as practical tools, while reflecting Google Dataset Cards' documentation philosophy in its diagnostic reports. The starting point is showing clients exactly where their datasets stand in quantitative terms.

7.2AI-Ready Data: Quality Metric Automation and Governance

AI-Ready Data is a framework for determining whether data is prepared for AI training. It shares the automated pipeline philosophy of NVIDIA NeMo Curator, automating the full journey from data curation to quality verification. The transparency and accountability demanded by OECD.AI principles are embedded directly into operational processes.

7.3Data Greenhouse: Synthetic Data Quality Verification

Verifying synthetic data quality presents a new challenge. It requires auditing statistical fidelity against original distributions and confirming that the ethical questions raised by Datasheets for Datasets apply equally to synthetic data. Data Greenhouse holds strategic value as a tool that bridges this gap.

The six frameworks analyzed in this report demonstrate that data quality has evolved from simple technical preprocessing into a core strategic capability for building effective, trustworthy, and responsible AI.

To summarize each framework's role: Datasheets laid the foundation for responsible AI. Google Dataset Cards provide the bedrock for transparency and accountability. IBM DQAI measures technical soundness. NVIDIA NeMo Curator enables efficient management of data at scale. DataPerf drives data-centric innovation. And OECD.AI connects all of this to a societal context.

References directly related to the six core frameworks are highlighted in bold.

1. mlcommons/dataperf: Data Benchmarking - GitHub. https://github.com/mlcommons/dataperf
2. AI Ethics at IBM. IBM Data Ethics PDF
3. Beyond Accuracy: Redefining Data Quality Metrics for Ethical AI - ResearchGate. ResearchGate
4. Datasheets for Datasets - Morgan Klaus Scheuerman. morgan-klaus.com
5. Datasheets for Datasets - Microsoft Research. Microsoft PDF
6. Datasheets for Datasets - arXiv. arXiv:1803.09010
7. Datasheets for Datasets - ResearchGate. ResearchGate
8. User Guide - Data Cards Playbook - Google Research. Google Research
9. The Data Cards Playbook - Google Research. Google Research
10. Data Cards Playbook: Transparent Documentation for Responsible AI - Google for Developers. Google Developers
11. Data Quality in AI - IBM Research. IBM Research
12. Data Quality Tools & Solutions - IBM. IBM Solutions
13. What Is Data Quality Management? - IBM. IBM Think
14. What Is Data Quality? - IBM. IBM Think
15. Data Quality Dimensions - IBM. IBM Docs
16. The Six Primary Dimensions for Data Quality Assessment. SBCTC PDF
17. Data Quality for AI Tool: Exploratory Data Analysis on IBM API - ResearchGate. ResearchGate
18. NVIDIA AI Enterprise - Cloud-Native Software Platform. NVIDIA
19. NeMo Curator - NVIDIA Developer. NVIDIA Developer
20. NeMo - Build, Monitor, and Optimize AI Agents - NVIDIA. NVIDIA
21. Chat With Your Enterprise Data Through Open-Source AI-Q NVIDIA Blueprint. NVIDIA Blog
22. Benchmark Work - Benchmarks MLCommons. MLCommons
23. DataPerf. dataperf.org
24. AI Principles Overview - OECD.AI. OECD.AI
25. OECD AI Principles. OECD.AI
26. OECD AI Principles: Guardrails to Responsible AI Adoption - code4thought. code4thought
27. Working Group on Data Governance - OECD.AI. OECD.AI
28. Datasheets for Healthcare AI: A Framework for Transparency and Bias Mitigation - arXiv. arXiv
29. What Are the Key Metrics Used to Evaluate Vision-Language Models? - Milvus. Milvus
30. DDFAV: Remote Sensing Large Vision Language Models Dataset and Evaluation Benchmark - MDPI. MDPI

Download the PDF version of this report, which includes the complete analysis, detailed references, and supplementary materials. Feel free to share it within your organization as a learning resource.