Epic remains one of the most widely deployed Electronic Health Record (EHR) systems across hospitals, clinics, and research networks. As healthcare organizations move toward value-based care and predictive analytics, the ability to analyze Epic’s operational data through modern cloud platforms such as Azure Databricks is essential. The goal is not only modernization. It is to make patient and operational information accessible, governed, and ready for clinical insight, operational excellence, and AI-driven outcomes.
Healthcare organizations are increasingly adopting cloud-based analytics by integrating Epic’s EHR platform with Azure Databricks. This approach delivers timely insights, supports better decision-making, scales analytics workloads, and maintains compliance with PHI and HIPAA requirements. Leaders are under growing pressure to move faster—to understand patient flow, reduce administrative burden, improve operations, and lay the groundwork for AI-driven initiatives. Epic contains the operational and clinical data that powers these insights, while Azure Databricks provides the scale and flexibility needed to analyze it—shifting analytics from static, retrospective reporting to real-time intelligence and continuous improvement.
This post explores the architectural patterns behind Epic-to-Databricks integration, focusing on secure connectivity, data access methods, and governance approaches that enable healthcare organizations to modernize analytics without compromising compliance.
Defining Epic data sources
Epic provides several ways to access clinical and operational data for analytics. Each source plays a different role in how organizations build reports, operational dashboards, and AI models. Understanding these sources is important because it helps teams decide which pipeline model best supports their business and analytical goals. Let’s explore a few below:
Clarity
Clarity is Epic’s relational reporting database, refreshed through scheduled extract, transform, and load (ETL) processes that pull data from Chronicles. Its SQL-friendly structure makes it a common foundation for historical, financial, and regulatory reporting. Clarity offers broad data coverage and supports detailed analysis when teams need complete clinical or revenue cycle information.
Caboodle
Caboodle is Epic’s enterprise data warehouse. It organizes curated data from Clarity into a simplified star schema that accelerates analytics and helps reduce development time. Caboodle is designed to support population health, business intelligence, and self-service reporting. It also helps reduce time to insight by giving analysts data that is easier to understand and ready for operational decision making.
FHIR APIs
Fast Healthcare Interoperability Resources (FHIR) APIs expose data directly from Epic Chronicles in near real time. This approach is especially valuable for operational workflows where data freshness matters, such as patient status updates, admissions and discharges, and clinical event monitoring. FHIR reduces reliance on nightly extract, transform, and load (ETL) refreshes and supports just-in-time analytics and AI-driven use cases.
Additionally, FHIR APIs have export endpoints for bulk exports against up-to-date information.
How these sources work together
Clarity delivers depth, Caboodle improves usability, and FHIR adds data freshness—so many organizations rely on a combination of all three to meet different analytics needs.
Caboodle typically supports dashboards and operational reporting, Clarity provides detailed and comprehensive data for advanced analysis, and FHIR enables real-time intelligence and AI readiness. Combined, these sources create a flexible and complete foundation for analytics in Azure Databricks.
Designing secure connectivity between Epic and Azure Databricks
Connecting Epic to Azure Databricks requires a secure and reliable network foundation. Epic environments often operate inside tightly controlled hospital data centers, and the clinical and operational data they contain is subject to HIPAA and HITRUST controls. As a result, connectivity between Epic and Azure must remain private, encrypted, and architected for consistent performance.
Most organizations use one of two primary connectivity patterns:
- Site to site VPN is often the first step because it is easier to provision and works well for development or smaller data volumes. As data movement increases, health systems adopt
- Many clients opt for Azure ExpressRoute. These dedicated circuits provide predictable throughput, lower latency, and private connectivity that does not traverse the public internet.
ExpressRoute is the most common pattern for Epic because it supports sustained data transfer from Clarity, Caboodle, and other operational systems.
Within Azure, networking is typically organized through a hub and spoke Virtual Network architecture. The hub contains shared resources such as firewalls, domain controllers, jump hosts, and connectivity gateways. Workloads that handle data, including Databricks and Azure Storage are deployed in isolated spokes. This design provides strong segmentation while still allowing controlled communication across the environment.
Figure 1: Example of Epic’s network security
Security controls are layered throughout the architecture. All traffic between Epic and Azure is encrypted in transit through VPN or ExpressRoute with private peering. Data is encrypted at rest in Azure Storage, Delta Lake, and any analytic cache layers. Private Link and VNet injection prevent PHI from flowing through public network paths. Network Security Groups and Azure Firewall enforce least privilege access between subnets and ensure that only required services can communicate across clinical, administrative, and analytics workloads.
This connectivity model allows healthcare organizations to scale analytics without increasing risk. It brings Epic data into a governed cloud environment while preserving privacy, meeting compliance requirements, and enabling the speed and flexibility needed for clinical operations, administrative reporting, and AI driven insights.
Integration patterns for Epic and Databricks
Throughput planning plays a critical role in selecting the right data access approach. Bulk data loads from Clarity or Caboodle can involve millions of records and require stable, sustained bandwidth over private connectivity. Near real-time use cases that rely on FHIR APIs demand low-latency compute to keep data current within Databricks. These factors directly shape both network design and data pipeline architecture.
The sections below outline the primary methods for accessing Epic data and how each supports different analytics and operational requirements.
Clarity and Caboodle data access
There are three different ways to connect to an external database in Azure Databricks. LakeFlow Connect and Query Federation are excellent methods for broadly sharing and replicating data from external databases, such as SQL Server or Oracle. Both of those methods actually rely on JDBC and Spark Connectors under the hood to build the service which is a traditional method.
This example demonstrates how to establish a JDBC connection to Caboodle from Azure Databricks.. For a copy of the code check out the following example at the link here.
# 1. JDBC connection details for Caboodle on Azure SQL
jdbc_hostname = ".database.windows.net"
jdbc_port = "1433"
jdbc_database = ""
jdbc_url = (
f"jdbc:sqlserver://{jdbc_hostname}:{jdbc_port};"
f"database={jdbc_database};"
"encrypt=true;"
"trustServerCertificate=false;"
"hostNameInCertificate=*.database.windows.net;"
"loginTimeout=30;"
)
# 2. Credentials pulled from Databricks secrets (replace with your scope and keys)
caboodle_user = dbutils.secrets.get("caboodle-scope", "caboodle-user")
caboodle_password = dbutils.secrets.get("caboodle-scope", "caboodle-password")
connection_properties = {
"user": caboodle_user,
"password": caboodle_password,
"driver": "com.microsoft.sqlserver.jdbc.SQLServerDriver"
}
# 3. Choose a Caboodle table or view to ingest
# Example: a dimension or fact table such as dbo.DimPatient or dbo.FactEncounter
caboodle_table = "dbo.DimPatient" # change to the table or view you need
# 4. Read from Caboodle into a Spark DataFrame
# https://learn.microsoft.com/en-us/azure/databricks/ingestion/lakeflow-connect/sql-server-pipeline#option-1-azure-databricks-ui
# https://docs.azure.cn/en-us/databricks/archive/connectors/sql-server # - this is archived but is what the other connectors leverage under the hood.
# https://docs.databricks.com/aws/en/query-federation/sql-server
caboodle_df = (
spark.read
.format("jdbc")
.option("url", jdbc_url)
.option("dbtable", caboodle_table)
.option("user", connection_properties["user"])
.option("password", connection_properties["password"])
.option("driver", connection_properties["driver"])
.load()
)
# 5. Write to Delta Lake for downstream analytics
# You can change the table name or path to match your bronze layer conventions
caboodle_df.write.saveAsTable("bronze_caboodle_dim_patient")
FHIR integration example
As previously discussed, FHIR APIs are designed to facilitate real-time access to healthcare data and are not optimized for large-scale data exports or high-volume workloads. Nonetheless, many organizations leverage supplemental export APIs to ingest data from source systems efficiently. In this section code examples will illustrate how to establish a connection to Epic on FHIR APIs (learn more about these APIs at the link here) as well as utilize the bulk export APIs available through Smart on FHIR at the link here.
The following code sample demonstrates how to extract data using the export APIs and subsequently load it into Databricks as files and tables, enabling advanced analytics and reporting capabilities. You can find the code at the link here.
# The SMART team runs a public Bulk FHIR server specifically for testing $export:
# Base URL: https://bulk-data.smarthealthit.org/fhir - bulk-data.smarthealthit.org
# Supports $export, Patient/$export, etc.
# Can be configured with different numbers of synthetic patients.
# No auth required by default → perfect for a demo.
# The export/poll logic can be used with EPIC Export functions
import time
import json
import requests
FHIR_BASE_URL = "https://bulk-data.smarthealthit.org/fhir"
# e.g. system-level export
EXPORT_URL = f"{FHIR_BASE_URL}/$export"
def start_bulk_export():
"""
Kick off a bulk $export job against the SMART Bulk Data sandbox.
No auth needed by default.
"""
headers = {
"Accept": "application/fhir+json",
"Prefer": "respond-async",
}
params = {
# What resource types you want:
"_type": "Patient,Encounter,Observation",
# "_since": "2020-01-01T00:00:00Z", # optional, for incremental exports
}
resp = requests.get(EXPORT_URL, headers=headers, params=params)
print("Kickoff status:", resp.status_code)
print("Kickoff headers:", resp.headers)
resp.raise_for_status()
status_url = resp.headers.get("Content-Location")
if not status_url:
raise ValueError("No Content-Location header returned – cannot poll export job.")
print("Status URL:", status_url)
return status_url
def poll_bulk_export(status_url: str, poll_interval: int = 5):
"""
Polls the status URL until the job completes.
Returns the export manifest JSON.
"""
headers = {
"Accept": "application/json",
}
while True:
resp = requests.get(status_url, headers=headers)
print("Poll status:", resp.status_code)
if resp.status_code == 202:
# Still working
print("Export still running...")
time.sleep(poll_interval)
continue
if resp.status_code == 200:
print("Export complete.")
manifest = resp.json()
print("Manifest:", json.dumps(manifest, indent=2))
return manifest
print("Export failed:")
print(resp.text)
resp.raise_for_status()
def download_bulk_files(manifest: dict, output_folder: str = "smart_bulk_output"):
"""
Downloads NDJSON files from the manifest.
SMART sandbox exposes them as unauthenticated HTTPS URLs.
"""
import os
os.makedirs(output_folder, exist_ok=True)
files = []
for entry in manifest.get("output", []):
resource_type = entry["type"]
url = entry["url"]
print(f"Downloading {resource_type} from {url}")
resp = requests.get(url, stream=True)
resp.raise_for_status()
local_path = os.path.join(output_folder, f"{resource_type}.ndjson")
with open(local_path, "wb") as f:
for chunk in resp.iter_content(chunk_size=8192):
f.write(chunk)
files.append(local_path)
print("Saved:", local_path)
return files
if __name__ == "__main__":
print("Starting SMART Bulk FHIR export demo...")
status_url = start_bulk_export()
manifest = poll_bulk_export(status_url)
files = download_bulk_files(manifest)
print("\nDownloaded NDJSON files:")
for f in files:
print(" -", f)
For direct authentication against the Epic on FHIR APIs, review the following code at the link here.
Data governance
Epic data contains Protected Health Information (PHI) and must be managed under HIPAA and HITRUST compliance frameworks. Security for an Epic-to-Databricks integration starts at the network layer and extends through data governance, encryption, and continuous monitoring.
Governance is handled within Databricks through Unity Catalog, which establishes a single control plane for access policies, lineage, and data classification. PHI datasets such as patient demographics or birthDate fields can be tagged, monitored, and governed under fine-grained role-based access controls RBAC. All activity is logged and made available to platform administrators for traceability.
Together, these Azure-native services create an end-to-end compliant architecture: network isolation through ExpressRoute and Private Link, encryption through Key Vault, and governance through Unity Catalog. This integrated approach secures PHI while empowering analytics and AI innovation in the healthcare cloud.
A path to scalable, governed healthcare analytics
Epic-to-Databricks integration is achievable through secure hybrid connectivity, FHIR-based interoperability, and lakehouse architecture. With proper governance, healthcare organizations can enable real-time, data-driven decision-making while maintaining patient privacy.
Unlock the full potential of your Epic data
Connect with our team to learn how to securely integrate Epic EHR with Azure Databricks, streamline analytics, and turn clinical and operational data into insights that drive better patient care and operational efficiency. Reach out at info@invisocorp.com to get started today.
