Welcome back to Part 2 of this 3-part series on optimizing Data Pipelines for historical loads. In the first two parts, we are introducing two technical patterns. Then in Part 3, we will bring everything together, covering an end-to-end design pattern. To recap, in Part 1 we covered how to parse a time interval (dd.hh:mm:ss) … Continue reading “Data Pipeline Performance Improvements Part 2: Creating an Array of JSONs”
Series Overview Welcome to this short series where we’ll be discussing the technical methods used to improve Data Pipeline Copy activity performance through parallelization by logically partitioning any source. Often, we see solutions leveraging a single Copy Activity to move large volumes of data. While this works great, you might face a scenario where you … Continue reading “Data Pipeline Performance Improvements Part 1: How to convert a time interval (dd.hh:mm:ss) into seconds”
Contents Scenario overview In this blog, we will act in the persona of an AVEVA customer who needs to retrieve operations data from AVEVA Data Hub into a Microsoft Fabric Lakehouse. Note: While this scenario is using AVEVA Data Hub, the concepts translate to interacting with any REST API that uses pagination. Considerations Design Pattern … Continue reading “Data Pipelines Tutorial: Ingest files into a Lakehouse from a REST API with pagination ft. AVEVA Data Hub”
