Overview
The Data Flow Destination Component is an SSIS Data Flow Component for exposing data from a data flow to the Data Flow Source component. It is similar in concept to the standard Microsoft DataReader Destination Component , but allows that data to be used as a source in the Data Flow Source component.
Since 2.3, the task can enable parallel processing of any source data Parallel Loop Task. The configuration parameters are provided in the standard Data Flow Component dialogs.
The user can choose how the rows will be distributed - balanced distribution or multicast. With balanced distribution, a row will be sent to just one Data Flow Source. With multicast distribution, a row will be sent to all connected Data Flow Source components. In this case, immediately after the Data Flow Source component, a Conditional Split transformation can be used to determine how to handle the incoming row - pass or not.
Quick Start
In this quick start we will configure parallel execution of DataFlow Destination and DataFlow Source by using Parallel Loop Task. Will be written multiple Parquet files in parallel.
Create Package Connection
Create Package Connection pointing the current package.
Create DataFlow Destination
Create Data Flow Task, create the desired source component and connect it with the new DataFlow Destination. Enter the number of the ConsumersCount. They should be equal to the planned parallel iterations.
Select input columns
Go to the input tab and select the desired input columns.
Foreach Loop Container
Create Foreach Loop Container and initialize iterations.
Data Flow Task
Place a Data Flow Task in the Foreach Loop Container.
Scoped variable
Create Foreach Loop Container scoped variable.
Map variable
Map the variable in Foreach Loop Editor.
Select connection in DataFlow Source
Place a DataFlow Source in the Data Flow Source Container task. Open the Editor and select the Package connection.
Select DataFlow Destination
Go to the Component Properties tab and select the DataFlowDestination property, select the DataFlow Destination in the selector.
Map columns
Go to the Column Mappings tab to map the columns.
Destination file name variable
Create string variable scoped in Foreach Loop Container and assign expression.
Parquet Destination
Place Parquet Destination and connect it with the DataFlow Source. Select the string variable as file name. Map the columns.
Parallel Loop
Place Parallel Loop in the package. Disable the Foreach Loop Container. Open the Parallel Loop editor. Select package connection. Select the Foreach Loop Container in the selector.
Sample
The sample with SQL Server 2012 format can be downloaded here.
Parameters
Configuration
Use the parameters below to configure the component.
Count of Parallel Client Components. When specified the component will wait the clients count to start processing.
Select how input data is handled. This parameter has the options listed in the following table.
Value Description BalancedDistribution Input data is evenly distributed to each consumer. Multicast Input data is passed to every consumer.
What's New
- New: Introduced ConsumersCount parameter.
- New: Introduced InputHandling parameter.
- New: Introduced component.
Related documentation
COZYROC SSIS+ Components Suite is free for testing in your development environment.
A licensed version can be deployed on-premises, on Azure-SSIS IR and on COZYROC Cloud.