Autonomous Data Stewardship with a Metadata Catalog Agent: Applying Agentic AI to Metadata at Scale

How an AI agent labels clinical, genomics and pharma metadata, maps it to CDISC and OMOP, flags PII, scores its own confidence, and hands only the uncertain cases to people.

Autonomous data stewardship: raw column names enriched by the metadata agent into a labeled, categorized, confidence-scored catalog entry

Ask a data scientist what clin_sig, gda_score or pt_dob means, and you'll often get an answer. Ask whether it's safe to use, and you'll get a pause.

Life sciences organizations sit on enormous amounts of data: clinical records, variant calls, protein annotations, drug-target references, trial registries. The problem is rarely having the data. It's knowing what each column actually contains, whether it matches an industry standard, and whether it carries patient identifiers that should never leave a controlled environment.

That knowledge usually lives in data dictionaries that are out of date, in the heads of a few data stewards, or nowhere at all. So every new analysis starts with the same detective work, and every audit starts with the same scramble.

You can't govern, share or trust data you can't describe.

At Clovertex, we built a Metadata Catalog Agent to do that describing automatically, and, just as importantly, to know when it isn't sure.

The problemWhy metadata falls behind

  • Too many columns, too few stewards. A single program can span hundreds of columns across clinical, genomics and pharma sources. Labeling them by hand never keeps up.
  • Every source speaks its own dialect. The same idea appears as clin_sig, clinical_significance and CLNSIG in different datasets.
  • Standards mapping is specialist work. Linking columns to CDISC or OMOP takes expertise that's scarce and expensive.
  • Privacy risk hides in plain sight. A column of birth dates or record identifiers looks harmless until it ends up somewhere it shouldn't.
  • Labels go stale. When an ETL job changes, yesterday's description may quietly stop being true.

The approachFrom raw columns to a governed catalog

How it works: clinical, genomics, protein, clinical research, drug and disease genomics sources flow through pre-flight discovery, agent enrichment on Amazon Bedrock, post-flight scoring and storage, and human review, with a fast path that reuses validated labels when source logic is unchanged
Machines do the discovery and labeling; people sign off on anything uncertain.

The agent works across a wide range of clinical, genomics and pharma reference data, including clinical EHR data (Synthea), genomics references (GnomAD, Ensembl), protein annotation (UniProt), clinical research (ClinicalTrials.gov), drug discovery and pharmacogenomics (ChEMBL, PharmGKB) and disease genomics (DisGeNET). Each column passes through four stages:

  1. Pre-flight. The pipeline scans Amazon S3 for data files, extracts column metadata, infers data types, classifies each column's domain and fetches any prior enrichment history.
  2. Agent enrichment. An AI agent powered by Amazon Bedrock studies each column using several sources of evidence: the source schema and ETL logic, pipeline run logs, existing canonical labels, and knowledge-base lookups against standards such as CDISC and OMOP.
  3. Post-flight. Confidence thresholds are applied, results are written to Amazon DynamoDB, summary reports go to S3 and metrics are published to CloudWatch.
  4. Human review. Results appear in a dashboard where data stewards approve, or edit and approve, anything the agent wasn't sure about.

The fast path

Each column's derivation logic in the source ETL script is fingerprinted. If the fingerprint hasn't changed since a label was validated, the pipeline simply reuses that validated label and skips the AI entirely. That keeps runs fast and cheap, and means a label is only re-examined when the data behind it actually changes.

Runs can be triggered on demand or on a schedule through AWS Lambda and EventBridge, with AWS Glue handling data movement.

What gets addedMore than a label

Governance is part of the label: canonical labels, standards mapping to CDISC or OMOP, PII flagging for HIPAA identifiers, human sign-off, source and lineage, and change detection via derivation fingerprints
Every enriched column carries what compliance teams, data stewards and analysts need to trust it.

For every column, the agent produces a complete, catalog-ready record:

Enriched label & description

A human-readable name and plain-language description, so clin_sig becomes "Clinical Significance".

Canonical label

A normalized identifier like CLINICAL_SIGNIFICANCE that stays consistent across every dataset.

Category & sub-category

For example Genomics → Variant Info, or Pharmacogenomics → Gene-Disease Association.

Standards mapping

A reference to the matching CDISC (CDISC.DOMAIN.VARIABLE) or OMOP (OMOP.table.field) variable.

PII flag

Columns that may contain HIPAA identifiers are flagged before anyone builds on them.

Source & lineage

Where the label came from, when it was last updated and its confidence history over time.

The key ideaAn agent that knows what it doesn't know

Most AI labeling tools give you an answer. This one also tells you how much to trust it. Every enrichment carries a confidence score from 0.00 to 1.00, computed from multiple sources of evidence, and that score decides what happens next.

Confidence scoring: high confidence at 0.70 and above is auto-validated, medium between 0.60 and 0.70 gets a spot-check, and low below 0.60 goes to manual review, with explained confidence reasoning for every score
Every label carries a confidence score, so human attention goes only where it's needed.

High · 0.70 and above

Auto-validated

Reliable labels go straight into the catalog, without a review step, unless someone chooses to override them.

Medium · 0.60 to 0.70

Spot-check

Probably right, but queued for a reviewer to confirm or edit.

Low · below 0.60

Manual review

High risk. A person decides the right label, guided by the agent's reasoning and evidence.

Crucially, the agent explains itself. A low-confidence label comes with reasoning such as: "No existing canonical labels found. Knowledge-base relevance below 75%. Label inferred from column name and domain context." A reviewer can see at a glance why the agent hesitated, rather than having to second-guess a black box.

Automation where it's confident. People where it isn't.

Human in the loopReview that respects people's time

Because high-confidence labels are validated automatically, the review queue contains only the columns that genuinely need a human: the medium and low tiers. Reviewers can filter that queue by data source, category or confidence tier, start with the riskiest items first, and approve a label as suggested or edit it before approving.

Every enriched column ends up in one of three clear states:

  • High Confidence: auto-validated by the agent
  • Pending: waiting for a person to review it
  • Overridden: approved by a reviewer, possibly with edits; the final, human-validated state

The outcomeA business catalog people actually use

The result is a searchable business catalog of every column, with its label, canonical name, category, data source, confidence and status. Anyone can inspect a column to see its description, standard mapping, confidence reasoning and lineage, or export a filtered view for downstream use. An enrichment summary shows overall confidence, quality by category and data source, and how much of the catalog is already mapped to industry standards: the gaps worth closing next.

Faster data understanding

Analysts find and understand data in minutes instead of chasing down data owners.

Privacy risk surfaced early

PII-flagged columns are visible before data is shared or joined.

Standards alignment

CDISC and OMOP mappings make submission work and cross-study analysis easier.

Audit-ready by default

Every label has a confidence score, a reason, a reviewer and a history.

What's nextMetadata as the foundation for AI

Well-described data is the foundation for everything else on the life sciences AI roadmap. The analytics agents in our precision medicine work depend on exactly this kind of semantic layer: plain-language descriptions of every table and field are what let an AI agent query data accurately instead of guessing.

The Metadata Catalog Agent makes that layer practical to build and keep current: AI does the labeling at scale, confidence scores route the hard cases to people, and every decision is recorded.

If your teams spend more time figuring out what data means than using it, or you're preparing for an audit and dreading the data dictionary, we'd be glad to show you the agent on your own datasets.

#Metadata#DataGovernance#CDISC#OMOP#HIPAA#LifeSciences#AgenticAI#AmazonBedrock#DataCatalog#Clovertex
Keep Reading

More from the Clovertex blog

Let's Talk

Know what your data means, and whether it’s safe to use

See the Metadata Catalog Agent label, map and risk-flag your own clinical and genomics datasets.