Skip to content

open-metadata/OpenMetadataStandards

Folders and files

NameName
Last commit message
Last commit date

Latest commit

Β 

History

30 Commits
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 
Β 

Repository files navigation

OpenMetadata Standards

OpenMetadata Standards

The Foundation for Modern Metadata Management

Comprehensive schemas, ontologies, and specifications powering unified metadata across the data ecosystem

Deploy Docs License Documentation Slack GitHub Stars

Documentation | Getting Started | Examples | Community


🌟 What is OpenMetadata Standards?

OpenMetadata Standards is the official repository for comprehensive metadata schemas, ontologies, and specifications that power the OpenMetadata platform and ecosystem. It provides a unified, open-source framework for representing, validating, and exchanging metadata across the entire data landscape.

Built with open standards and designed for interoperability, OpenMetadata Standards enables organizations to:

  • πŸ“Š Unify metadata across databases, data warehouses, data lakes, ML models, dashboards, and APIs
  • πŸ” Discover and catalog data assets with rich, structured metadata
  • 🎯 Govern data with policies, classifications, and glossaries
  • πŸ”— Track lineage from source to consumption across multi-platform data pipelines
  • βœ… Ensure quality with comprehensive data validation and profiling schemas
  • 🌐 Build knowledge graphs using RDF/OWL ontologies and semantic web standards
  • πŸ”„ Integrate seamlessly with existing tools through JSON Schema, OpenAPI, and JSON-LD

πŸ“¦ What's Inside

🎯 JSON Schemas

700+ comprehensive schemas covering:

  • Entities: Tables, databases, topics, dashboards, pipelines, ML models, APIs
  • Types: Rich type system with custom properties
  • APIs: Complete OpenAPI specifications
  • Events: Change events and audit logs
  • Config: Service and integration configurations

🌐 RDF & Semantic Web

Semantic standards for linked data:

  • OWL Ontology: Full OpenMetadata ontology
  • SHACL Shapes: Validation constraints
  • JSON-LD Contexts: Semantic contexts
  • PROV-O: Provenance and lineage tracking
  • SKOS: Knowledge organization

πŸ“š Documentation

Comprehensive guides including:

  • Interactive schema documentation
  • API reference with 200+ endpoints
  • Real-world examples and patterns
  • Integration guides
  • Best practices

πŸ”§ Developer Tools

Resources for developers:

  • Schema validation examples
  • Code generation templates
  • Testing frameworks
  • Migration guides
  • SDK patterns

πŸš€ Key Capabilities

πŸ—‚οΈ Data Asset Schemas

Comprehensive metadata models for all data assets:

Databases Tables, Columns, Schemas, Views, Stored Procedures, Queries
Storage Containers, Directories, Files, Spreadsheets, Worksheets
Pipelines Data Pipelines, Tasks, Workflow Definitions, Scheduling
Messaging Topics, Message Schemas, Partitions, Consumer Groups
Dashboards Dashboards, Charts, Data Models, Reports
ML Models Models, Features, Hyperparameters, Metrics, Experiments
APIs Collections, Endpoints, Request/Response Schemas, Authentication
Search Search Indexes, Fields, Mappings

πŸ›οΈ Governance & Compliance

Enterprise-grade governance schemas:

  • πŸ“– Glossaries - Business terminology with hierarchies and relationships
  • 🏷️ Classifications & Tags - Automated and manual classification systems
  • πŸ“‹ Policies - Access control, data policies, and compliance rules
  • πŸ“ Metrics - Business metrics and KPI definitions
  • πŸ“ Data Contracts - Schema contracts and SLAs

βœ… Data Quality & Observability

Comprehensive quality and monitoring:

  • πŸ§ͺ Test Definitions - 50+ built-in test types (schema, freshness, completeness, etc.)
  • πŸ“Š Test Suites - Organized test collections with execution history
  • ⚠️ Incidents - Issue tracking and resolution workflows
  • πŸ“ˆ Data Profiling - Statistical profiles and distribution analysis
  • πŸ”” Alerts - Configurable alerting and notification rules

πŸ”— Lineage & Relationships

End-to-end data lineage tracking:

  • Column-level lineage with transformation logic
  • Cross-platform lineage across databases, pipelines, and dashboards
  • Manual lineage editing and augmentation
  • Impact analysis for upstream and downstream dependencies
  • Provenance tracking using W3C PROV-O standard

πŸ‘₯ Teams & Collaboration

People and organizational metadata:

  • Users & Teams - Hierarchical team structures
  • Roles & Policies - Fine-grained access control
  • Personas - User personas for targeted experiences
  • Domains - Business domain organization
  • Ownership - Asset ownership and stewardship

πŸ”Œ Service Integration

84+ connector schemas for data sources:

  • Databases: PostgreSQL, MySQL, Oracle, SQL Server, Snowflake, BigQuery, Redshift, etc.
  • Warehouses: Databricks, Synapse, Teradata, Vertica, etc.
  • Lakes: S3, GCS, Azure Data Lake, HDFS, etc.
  • Messaging: Kafka, Pulsar, RabbitMQ, Kinesis, etc.
  • Dashboards: Tableau, PowerBI, Looker, Superset, Metabase, etc.
  • Pipelines: Airflow, dbt, Dagster, Fivetran, etc.
  • ML Platforms: MLflow, SageMaker, Kubeflow, etc.

🎯 Use Cases

πŸ“š Data Catalogs

Build comprehensive, searchable data catalogs with rich metadata, automated discovery, and collaborative documentation.

πŸ›‘οΈ Data Governance

Implement enterprise governance with automated classification, policy enforcement, and compliance tracking.

πŸ” Data Discovery

Enable self-service data discovery with semantic search, recommendations, and popularity metrics.

πŸ“Š Data Quality

Define, monitor, and enforce data quality rules with automated testing and incident management.

πŸ”— Data Lineage

Track complete data lineage from source to consumption with column-level granularity and impact analysis.

🌐 Knowledge Graphs

Build semantic knowledge graphs using RDF, OWL, and SPARQL for advanced analytics and AI applications.

πŸ”„ Data Integration

Standardize metadata exchange across tools and platforms using open standards (JSON Schema, OpenAPI, RDF).

πŸ“‹ Compliance

Ensure regulatory compliance (GDPR, CCPA, HIPAA) with metadata-driven policies and audit trails.

πŸ€– ML Metadata

Manage ML model metadata including features, experiments, versioning, and deployment tracking.


πŸƒ Quick Start

πŸ“– Explore Documentation

Visit openmetadatastandards.org for comprehensive documentation, including:

πŸ’» Local Development

# Clone the repository
git clone https://github.com/open-metadata/OpenMetadataStandards.git
cd OpenMetadataStandards

# Install Python dependencies
pip install -r requirements.txt

# Serve documentation locally at http://localhost:8000
mkdocs serve

βœ… Validate Your Metadata

Python Example:

import json
import jsonschema

# Load the table schema
with open('schemas/entity/data/table.json') as f:
    table_schema = json.load(f)

# Load your table metadata
with open('my_table_metadata.json') as f:
    table_data = json.load(f)

# Validate
try:
    jsonschema.validate(instance=table_data, schema=table_schema)
    print("βœ“ Valid metadata!")
except jsonschema.ValidationError as e:
    print(f"βœ— Validation error: {e.message}")

JavaScript/TypeScript Example:

const Ajv = require('ajv');
const fs = require('fs');

const ajv = new Ajv();

// Load schema and data
const schema = JSON.parse(fs.readFileSync('schemas/entity/data/table.json'));
const data = JSON.parse(fs.readFileSync('my_table_metadata.json'));

// Validate
const validate = ajv.compile(schema);
const valid = validate(data);

if (valid) {
  console.log('βœ“ Valid metadata!');
} else {
  console.log('βœ— Validation errors:', validate.errors);
}

🌐 Use with OpenMetadata Platform

OpenMetadata Standards power the OpenMetadata platform:

# Try OpenMetadata with Docker
docker run -d -p 8585:8585 \
  --name openmetadata \
  openmetadata/server:latest

Visit docs.open-metadata.org for full installation guide.


πŸ“š Documentation Structure

OpenMetadataStandards/
β”œβ”€β”€ πŸ“ schemas/                    # 700+ JSON Schema files
β”‚   β”œβ”€β”€ entity/
β”‚   β”‚   β”œβ”€β”€ data/                 # Data entities (tables, topics, etc.)
β”‚   β”‚   β”œβ”€β”€ services/             # Service configurations
β”‚   β”‚   β”œβ”€β”€ governance/           # Governance entities (glossaries, policies)
β”‚   β”‚   β”œβ”€β”€ teams/                # Teams and users
β”‚   β”‚   └── ...
β”‚   β”œβ”€β”€ type/                     # Type system definitions
β”‚   β”œβ”€β”€ api/                      # API specifications
β”‚   β”œβ”€β”€ events/                   # Event schemas
β”‚   └── configuration/            # Configuration schemas
β”‚
β”œβ”€β”€ πŸ“ rdf/                        # RDF/OWL Ontologies
β”‚   β”œβ”€β”€ ontology/                 # OpenMetadata OWL ontology
β”‚   β”œβ”€β”€ shapes/                   # SHACL validation shapes
β”‚   └── contexts/                 # JSON-LD contexts
β”‚
β”œβ”€β”€ πŸ“ docs/                       # Documentation source (MkDocs)
β”‚   β”œβ”€β”€ getting-started/          # Tutorials and guides
β”‚   β”œβ”€β”€ data-assets/              # Entity documentation
β”‚   β”œβ”€β”€ governance/               # Governance docs
β”‚   β”œβ”€β”€ examples/                 # Usage examples
β”‚   └── reference/                # API reference
β”‚
└── πŸ“ examples/                   # Example metadata files
    β”œβ”€β”€ basic/                    # Simple examples
    β”œβ”€β”€ advanced/                 # Complex scenarios
    └── integration/              # Integration patterns

πŸ› οΈ Standards & Compliance

OpenMetadata Standards is built on industry-standard specifications:


JSON Schema
Draft 07 & 2020-12

RDF/OWL
W3C RDF 1.1 & OWL 2

SHACL
W3C Validation

JSON-LD
Linked Data

Additional Standards:

  • OpenAPI 3.0 - API specifications
  • PROV-O - W3C Provenance Ontology for lineage
  • SKOS - Simple Knowledge Organization System
  • DCAT - Data Catalog Vocabulary

🌍 Community & Support

πŸ’¬ Join Slack

Connect with the community, ask questions, and share ideas.

Slack

πŸ“– Read Docs

Comprehensive documentation with guides, examples, and API reference.

Docs

πŸ› Report Issues

Found a bug or have a feature request? Let us know!

Issues

🀝 Get Involved

We welcome contributions from the community! Here's how you can help:

  • ⭐ Star this repository to show your support
  • πŸ› Report bugs and suggest features via GitHub Issues
  • πŸ“ Improve documentation by submitting pull requests
  • πŸ’‘ Share use cases and examples in our community
  • πŸ—£οΈ Spread the word about OpenMetadata Standards

πŸ“ž Support Channels


🚒 Deployment & Hosting

This documentation is automatically deployed to GitHub Pages with every commit to the main branch.

🌐 Custom Domains

The documentation is accessible via multiple domains:

  • Primary: openmetadatastandards.org
  • Alternate: openmetadatastandards.com (redirects to .org)
  • Alternate: openmetadatastandard.com (redirects to .org)

🀝 Contributing

We love contributions! Whether you're fixing bugs, adding new schemas, improving documentation, or sharing examples, your help is appreciated.

πŸ“‹ Contribution Guidelines

  1. Fork the repository
  2. Create a feature branch (git checkout -b feature/amazing-feature)
  3. Make your changes in the appropriate directory:
    • Schemas: schemas/
    • Documentation: docs/
    • Examples: examples/
    • RDF/Ontologies: rdf/
  4. Test your changes locally:
    # Serve documentation locally
    mkdocs serve
    
    # Validate documentation (checks for broken links, 404s, build warnings)
    ./scripts/check-docs.sh
  5. Commit your changes (git commit -m 'Add amazing feature')
  6. Push to the branch (git push origin feature/amazing-feature)
  7. Open a Pull Request

πŸ“– Detailed Guides

βœ… Code of Conduct

Please read our Code of Conduct before contributing.


πŸ“Š Project Statistics

πŸ“ˆ Repository Stats

GitHub stars GitHub forks GitHub contributors GitHub last commit GitHub repo size

πŸ“¦ Content Statistics

  • 700+ JSON Schemas covering all major metadata entities
  • 200+ API Operations fully documented
  • 100+ Entity Types with comprehensive properties
  • 50+ Test Definitions for data quality
  • 84+ Service Connectors schemas
  • Full RDF/OWL Ontology with SHACL validation

πŸ“„ License

OpenMetadata Standards is licensed under the Apache License, Version 2.0.

Copyright 2021-2025 OpenMetadata

Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at

    http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.

See LICENSE for the full license text.


πŸ™ Acknowledgements

OpenMetadata Standards is built with love by the OpenMetadata Community and powered by:


MkDocs

Material for MkDocs

JSON Schema

W3C RDF/OWL

GitHub Pages

🌟 Special Thanks

Special thanks to all our contributors and the organizations using OpenMetadata Standards in production.


πŸ”— Related Projects

πŸš€ OpenMetadata Platform

The unified metadata platform powered by these standards.

OpenMetadata

πŸ“š OpenMetadata Docs

Complete documentation for the OpenMetadata platform.

Docs


πŸ’™ Built with love by the OpenMetadata Community

Website Slack Twitter LinkedIn

⭐ If you find OpenMetadata Standards useful, please star this repository! ⭐

About

No description, website, or topics provided.

Resources

License

Contributing

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages