Microsoft has designed the entire Fabric architecture from the ground up to make it easy to adopt, set up, and use. One focal area is the way data is ingested, stored, and made available for consumption, so in this blog we’ll dig into new and existing methods to get data into Fabric OneLake.
With OneLake forming the central part of the entire Fabric architecture, all services can read and write directly to it, thus eliminating data redundancy, discouraging data silos, and making all the data available in a single unified lake.
Microsoft has also focused on data interoperability by adopting the Delta Parquet open data format for storage, which allows any service to tap into this data and run specific workloads, be it reporting, analytics, or any machine learning (ML) operations.
While OneLake and the interoperability and performance benefits of the open data format are advantageous, it’s very important that the process to initially get data into OneLake is well defined, easy to set up, and offers multiple pathways to pull in data. Microsoft has been quietly working on enhancing the existing data ingestion services as well as introducing new ways to ingest data into the OneLake.
Data ingestion options within Fabric
Fabric offers multiple data integration options, from the classic ETL/ELT services for scheduled or event-triggered data flows to other near real-time solutions. Below, we dig into these data integration options that are available within Microsoft Fabric, providing a description, use case, and updates for each:
- Data Factory pipelines
- Dataflows
- Spark notebooks
- Data workflow
- Eventstream
- Shortcuts
- Mirroring
- Direct uploads
- OneLake Explorer
Data Factory pipelines
About data pipelines:
- A data pipeline is an ETL/ELT tool that allows you to orchestrate and transform your data at scale.
- Data pipelines allow you to quickly transfer large volumes of data from a variety of sources using the 100+ connectors that are available within them.
- With control flow options, you can build workflow logic using loops and conditional flows.
- You can perform basic transformations on the data while ingesting it.
Use cases:
- Transfer large volumes of data quickly, which is convenient for scheduled data transfer activities and one-time data migrations.
- Useful for performing limited transformations on the data to be ingested.
- Trigger a wide variety of actions within or outside the Fabric environment such as calling a notebook, executing a (SP), running an Azure function, etc.
What’s new:
- Fabric Spark job definition activity is now available for data pipelines, allowing you to run a Fabric Spark job definition directly in your pipeline.
- Save time using the new ‘browse Azure’ functionality in Get Data to easily browse all your Azure resources and automatically connect to them without having to manually set up a connection.
- Easily get started importing data into your warehouse using the copy data assistant to guide you through the process.
Dataflow Gen2
About Dataflow Gen2:
- Dataflows provide a low-code interface for ingesting data from hundreds of data sources and transforming that data using over 300 available data transformations.
- You can load the resulting data into multiple destinations, such as Azure SQL databases and other supported platforms.
- Dataflows can be run manually, scheduled for refresh, or included as part of a data pipeline orchestration.
Use cases:
- Dataflows are useful when we need to perform advance transformations such as data cleansing, data aggregations, or custom transformations.
- If you have users who are familiar with Power Query (available with services such as Excel, Power BI, Power Platform, etc.), they can transform this data using the familiar no code/low code interface.
What’s new:
- With the introduction of fast copy, you can ingest terabytes of data with the easy experience of dataflows while leveraging the scalable backend of the pipeline’s copy activity.
- Destination connectors now allow data transfers to Fabric Lakehouse, Azure Data Explorer, Azure Synapse Analytics, and Azure SQL Database.
Spark notebooks
About Spark notebooks:
- Notebook jobs are a popular way to run data analysis and machine learning workflows on Fabric. They allow users to code using their preferred Apache Spark language for data analysis:
- PySpark (Python)
- Spark (Scala)
- Spark SQL
- SparkR
- Can be triggered by pipelines or a job scheduler, depending on the user’s needs.
- With the freedom to use hundreds of Spark libraries, users can consume and transform data from a variety of sources while performing numerous transformations.
Use cases:
- Connect to, consume, and parse data from a variety of complex API sources.
- Apart from data ingestion and transformation, Spark notebooks can be used for data profiling, data validation, and running ML workloads on the data.
- With access to any of the Spark libraries, you can perform activities that otherwise would not be possible with other tools such as Fabric data pipelines.
- Useful for processing large datasets.
What’s new:
- The Spark cluster in Fabric is much quicker to spin up and process data compared to earlier implementations.
Data workflow
About Data workflow:
- Fabric provides an integrated, cloud-based platform for developing, scheduling, and monitoringPython-based data workflows, articulated as directed acyclic graphs (DAGs) using Apache Airflow
Use cases:
- Airflow is one of the most popular ETL tools. If you have existing DAGS that you want to reuse in your Fabric environment, you can use the Data workflow feature to mount and monitor your existing Airflow DAGs.
Eventstreams
About Eventstreams:
- Eventsteam is a feature within Microsoft Fabric that allows you to capture, transform, and redirect real-time events to various destinations using a no-code experience.
- You can integrate your Eventstream with Azure Event Hubs, KQL database, and Lakehouse.
- Ingest data from databases using CDC, as well as popular sources such as Azure Event Hub, Amazon Kinesis, Kafka, Azure Blob storage events, etc.
Use case:
- With real-time data processing, Eventstream allows you to monitor critical systems such as stock markets and temperature-dependent machines.
- Directly integrate real-time event data with (an intelligent agent that automates responses to data conditions) for automatic alerting.
Shortcuts
About shortcuts:
- Shortcuts in Microsoft OneLake are objects that reference other internal or external cloud storage locations.
- Appearing as folders in OneLake, shortcuts allow any service with access to reference the data without physically moving or copying it.
- If you delete a shortcut, the target remains unaffected. If you move, rename, or delete a target path, the shortcut can break.
- As of now, you can create shortcuts to Azure Data Lake Storage (ADLS) Gen2, Amazon S3, Google Cloud Store, and Dataverse
Use cases:
- If your source data is in an S3 bucket on AWS, you can create a shortcut and reference the data in near real time from its existing location.
- Use shortcuts to avoid building ETL’s and storing redundant data for non-critical sources.
Mirroring (Preview)
About mirroring:
- Mirroring allows near real-time replication of external databases into OneLake Delta tables.
- Currently, mirroring supports Azure Cosmos DB, Azure SQL DB, and Snowflake sources.
Use cases:
- With mirroring, you don’t need to build and schedule ETL pipelines.
- Data is synced in near-real time between the source database and Fabric, allowing for time-critical analysis.
Direct uploads
About direct uploads:
- Direct uploads allow you to browse and upload a supported file or folder from your local system directly to OneLake using the Fabric interface.
Use case:
- Use the direct uploads feature for one-time or ad-hoc uploads for data analysis as it’s much quicker than building an ETL to ingest the data.
OneLake Explorer
About OneLake Explorer:
- One Lake Explorer is a Microsoft tool that can be installed on your system for syncing and browsing files stored within your One Lake.
- Use it to copy over supported local files directly to the desired folder within OneLake as you would to any of the drives on your system.
- You will only be able to see and interact with files for which you have permission.
Use case:
- You can use OneLake Explorer to upload files directly to the desired folder instead of authenticating and logging onto the portal each time.
Non-Fabric alternates
Apart from the Fabric ingestion tools documented above, you can also use newer connectors introduced in Azure Data Factory and other third-party services to export data to a Fabric OneLake/Lakehouse.
While Microsoft Fabric offers multiple options for data ingestion, it’s crucial to choose an ingestion mechanism that suits your existing architecture, is scalable, and provides the necessary performance.
We can guide you through selecting and implementing the ideal data ingestion mechanism for your organization. Contact us today to start your Fabric proof of concept!
