JavaScript Task and JavaScript Component enable the automation of any custom logic in the Control Flow and the Data Flow of an SSIS package, via JavaScript scripting. Their purpose is similar to the standard Microsoft Script Task and Microsoft Script Component, but they aim to further simplify the development and maintenance of SSIS scripts.
Why JavaScript?
For too many years, SSIS users had no choice, but to automate various custom tasks with C# and VB.NET, which are both high-ceremony and somewhat difficult-to-learn programming languages. Although the Integration Services Expressions have also been available as a simpler-to-use alternative, their limitations and peculiarities have prevented them from becoming a preferred tool for such tasks.
JavaScript Task and JavaScript Component enable automating SSIS with an easy-to-learn, versatile, popular language that’s well suited for most scripting needs. You can still harness the power of the .NET Framework, but without the overhead of dealing with C# / VB.NET projects in the Microsoft Visual Studio Tools for Application (VSTA) IDE. When using JavaScript for SSIS scripting, you can expect to prepare your scripts in much less time. You will get the job done with less code and your scripts will be easier to comprehend and maintain.Efficient Coding
Unlike the standard Script Task and Script Component, viewing and editing the JavaScript code is done directly in a JavaScript Editor that's embedded in a dedicated “Script” tab. You no longer have to test your patience while waiting for a new instance of VSTA to load up, just to view/edit a simple script. You will have a full-featured JavaScript editor at your fingertips (the one included in Visual Studio) that has everything to make you productive - syntax highlighting, methods folding, breakpoints, etc.
Code Reusability
The JavaScript Task and JavaScript Component allow you to minimize code duplication across your SSIS projects by supporting easy code reuse. Instead of copying the same code in multiple JavaScript tasks or components, you can simply create a link to a “source” task/component instead. The source task/component can reside in any package of the project. With this setup, the runtime will automatically use the JavaScript code embedded in the source task/component during execution. After each code modification in the source task/component, you won’t need to do anything extra in order for all the dependent tasks/components to use the latest version of the script.
Parameterization
This separates the script logic from its input data. You can easily parameterize JavaScript Task and JavaScript Component within the script code. This enables viewing and specifying parameter values in a dedicated “Parameters” tab of the task/component editor. Specifying those values is done in a familiar Property Grid that has easy-to-use editors/selectors for various parameter types).
Debugging
To debug a JavaScript Task or JavaScript Component you must have VSTA installed. To enable debugging, you would need to set at least one breakpoint in the code. The debugger;
statement can also be used. You can have breakpoints in more than one JavaScript Task or JavaScript Component in your package. The only limitation is that if you have several JavaScript scripts that are executing simultaneously, you will only be able to debug the one that started first. Conditional breakpoints are not supported (i.e. they will be treated as unconditional).
Tip: Troubleshooting JavaScript scripts doesn't necessarily require debugging. Often it can be as effective to simply use alert()
or console.info()
statements for inspecting the intermediate results.
Upgrading
Upgrading of JavaScript Task and JavaScript Component to later versions of SQL Server would be seamless, as unlike the standard Script Task and Script Component, there is no early binding and no embedded CIL (Common Intermediate Language) in SSIS packages.
How Does It Work Under the Hood?
At the most basic level, scripting SSIS with JavaScript works by compiling the JavaScript (more precisely ECMAScript 5-compatible) code on-the-fly into .NET bytecode. Please note that the performance overhead of this is minimal, as the JavaScript code gets compiled only once and then exercised normally.
But there is a lot more to it... To be able to offer an easy-to-use API that makes the SSIS API look like it was originally designed to be used with JavaScript, we have developed wrappers around the SSIS Object Model and introduced additional objects and methods to improve the convenience for automating common scenarios. The end result is that JavaScript Task and JavaScript Component offer an API that makes it very easy and natural to manage SSIS entities like variables, parameters, connection managers, inputs, outputs, etc.
To understand better how the JavaScript scripting engine operates, it's worth noting that besides the execution of the main processing logic during runtime, there is a preliminary execution (a Global Scope execution), which is used for handling the script initialization and the processing of the parameter definitions. In design-time, it is triggered upon opening the Editor of the JavaScript Task or JavaScript Component and also upon detecting any change that can affect the script parameters. During runtime, it is run once while validating the task/component.
JavaScript Task
We will setup a parameterizable JavaScript Task to automate a simple scenario - checking if a file exists and saving the result in a boolean variable.
See the debugger in action
To start debugging, close the JavaScript Task Editor and execute the package. During the execution another instance of the Visual Studio will be launched and the debugger will stop at the first breakpoint.
After verifying that the script works as expected, close the VSTA instance.
JavaScript Component
In this quick-start we will setup two parameterizable JavaScript Components (a source and a transformation) to automate a simple scenario - generating dates and adding row counter. The source will generate rows with a date column, while the transformation will add another column with a row count.
Add parameters in the script
Let’s open the component editor and introduce two component parameters “StartDate” and “Count” by assigning them to component.parameters
. We will specify for them description
, type
and validation
. For validation we will use the library validation.js
where are defined the most frequently used validations.
Output column creation
Let’s create the output and the output column. As the columns are static, they will be created in component.reinitializeMetadata
function. We need one output and one column of type DT_DATE
. We make definition of the output and the output column. Helper functions will remove current when they are different and create new. To use helper functions the library component.metadata.js
needs to be included.
Summary
In summary, we have demonstrated how to setup two simple JavaScript Components: a source (having only an output) and a transformation (having an input and a synchronous output). Unlike the standard Script Component, please note, that both of them use a single method for processing the data component.processData(inputBuffer,outputs)
.
, , , Derived Column
In this Quick Start article, we will show you step by step in How-To create a simple package that uses Derived Column Plus transformation to create a new column and to modify an existing one. We will use the OLE DB Source component in the Data Flow to read the data from a table and write it to a different table using the OLE DB Destination component.
Setup Data Flow – OLE DB Source Component
- Click the Data Flow tab.
- Add the OLE DB Source component from the SSIS Toolbox, by dragging the component to the design surface.
- Double-Click on the OLE DB Source to open the Editor, configure the connection manager and select the source table from the list.
- Click OK button.
(See Figure-01)
Setup Data Flow – Derived Column Plus component
- Add the Derived Column Plus component from the SSIS Toolbox, by dragging the component to the design surface.
- Drag the connector arrow from OLE DB Source to the Derived Column Plus component.
- Double-Click on the Derived Column Plus to open the Editor.
- Create new derived column that replaces an existing input column:
4.1. Select "Add" button. In "Derived Column" drop down list select the existing input column. The derived column will have the same properties including Data Type, Length, Precision, Scale, Code Page and they cannot be edited.
4.2. Construct the derived column expression in JavaScript editor. The expression can contain any combination of variables, functions, operators, and columns from the transformation input. You can drag-drop them from the tree on the right side of the editor. Search functionality is provided for convenience.
4.3. Enter sample parameter values if needed. Real time expression evaluation is performed and can be seen in the Evaluation Result pane.
(See Figure-02)
Setup Data Flow – OLE DB Destination Component
- Add the OLE DB Destination Component from the SSIS Toolbox, by dragging the component to the design surface.
- Drag the connector arrow from Derived Column Plus to the OLE DB Destination Component.
- Double-Click on the OLE DB Destination to open the Editor, configure the connection manager and select the destination table.
- Click on the Mappings page to map derived and input columns to the destination columns. (See Figure-04)
- Click the OK button to return to the Data Flow.
- Click File from Menu and Save All.
- To run the package, Click Start Debugging on the menu bar, or press F5 key.
Note: After the package finishes running, press Shift+F5 to return to design mode.
Overview
JavaScript Task is an SSIS Control Flow Task that allows automation of any custom logic in the control flow. It is an alternative of Microsoft Script Task and COZYROC Script Task Plus. It is also a more flexible replacement of Microsoft Expression Task.
By choosing JavaScript for implementing custom control flow logic, you will significantly simplify the development and maintenance of your SSIS scripts. For more details about the benefits of JavaScript scripting, please look here.
Editor
The Editor Dialog contains the following tabs:
Parameters tab
The Parameters tab displays the parameters, defined in the script, and their current values. For more detailed information about defining and using script parameters see here.
Properties tab
Referencing code from another JavaScript Task
To avoid code duplication, the code of a JavaScript Task can be stored in another JavaScript Task. The properties under the Linked Task category allow referencing the source JavaScript Task:
- PackageConnection - allows choosing the package, where the source JavaScript Task is (i.e. the current package or an external one).
- JavaScriptTask - allows selecting the source JavaScript Task in the selected package.
If you have specified a source JavaScript Task, you can unlink it later by clicking the button Remove link below.
Customizing the execution and validation logic outside the script code
Under the Script category there are two properties:
- Expression - specifies JavaScript expression that will be executed. If not specified,
task.run
would be run. - Validation - specifies JavaScript expression that will be used to validate parameters. If not specified,
task.validate
would be used.
Usually, the execution expression is defined in the script code by implementingtask.run
. But having a separate Expression property provides additional opportunities for code reuse - e.g. two JavaScript Tasks might share a script, but they can execute different logic by configuring different expressions to run.
Script tab
The tab displays the JavaScript script code. It allows editing and adding breakpoints. The embedded editor is the one that comes with the particular version of Visual Studio. For detailed information how to automate SSIS tasks with JavaScript, please refer to the API Reference.
At the bottom of the dialog, there are two buttons that make it convenient to load the JavaScript code from a file and save the code to a file.
Quick Start
We will setup a parameterizable JavaScript Task to automate a simple scenario - checking if a file exists and saving the result in a boolean variable.
See the debugger in action
To start debugging, close the JavaScript Task Editor and execute the package. During the execution another instance of the Visual Studio will be launched and the debugger will stop at the first breakpoint.
After verifying that the script works as expected, close the VSTA instance.
Knowledge Base
- Where can I find the documentation for the JavaScript Task?
- Is there a way to set the parameters of a Javascript dynamically when using Javascript Task.
- Where can I find scripts for the JavaScript Task and the JavaScript Component?
- Microsoft Dataverse - A Template for Close Opportunities as WON or LOST
- Microsoft Dataverse - A Template to Cancel Sales Order
What's New
- Fixed: An ocassional error "The directory is not empty" in the post-execution cleanup (Thank you, Saikit).
- New: Commands to quickly Copy/Paste user parameters.
- Fixed: Incorrect variable replacement in JSON.
- Fixed: Disabled various destructive commands in the script editor.
- Fixed: Improved memory consumption and management.
- Fixed: Removed not needed exception popup dialog because the error is already reported in the log.
- New: Introduced task.
Related documentation
Overview
JavaScript Component enables automating of any custom logic in the data flow. It is an alternative of Microsoft Script Component and COZYROC Script Component Plus. You might also consider using it instead of Microsoft Derived Column Transformation if you need to deal with more complex expressions.
By choosing JavaScript for implementing custom data flow logic, you will significantly simplify the development and maintenance of your SSIS scripts. For more details about the benefits of JavaScript scripting, please look here.
Source, Destination or Transformation
The JavaScript Component can be used as a source, a destination or a transformation:
- When used as source, the component doesn't have any inputs
- When used as destination, the component doesn't have any outputs
- When used as transformation, the component can have one or more inputs and one or more outputs
The JavaScript Component is more flexible than the standard Script Component, as it:
- Can define more than one input
- Can define error outputs
- Supports columns with the same names in an input and its synchronous output.
Advanced features for configuring metadata
Configuring metadata via code
The component metadata can be intliialized and changed in component.reinitializeMetadata
function. It is invoked in several occasions: when the component needs new metadata, when the active tab in the Editor is changed, and when the "OK" button is clicked. Because of the multiple invocations, the code needs to check if the metadata already contains the desired changes (e.g. check if an input is already added and not to try to add it again).
Input columns mappings
The Editor's "Mappings" tab, which allows specifying mappings between virtual and external column, is hidden by default. It can be shown via code: component.showMappings = true;
. The external column cannot be created by the Editor, they must be created from the script. If the external columns do not depend on parameters, they should be created in component.reinitializeMetadata
function. Otherwise they should be created in the component.validate
function.
Parameterizing metadata changes
JavaScript Component supports parameter types which allow selection of an input, an input columns, an output, an output columns (see here) from the "Parameters" tab. Then in the component.validate
function, the component metadata can be modified, according the specified parameter values.
For example, a transformation script can have a parameter for selecting input virtual columns and after some conversion to output the values in an asynchronous output. Based on user selection, in the component.validate
function the selected columns can be marked as used and to create matching output columns.
Editor
The Editor Dialog contains the following tabs:
Parameters tab
The Parameters tab displays the parameters, defined in the script, and their current values. For more detailed information about defining and using script parameters see here.
Properties tab
Referencing code from another JavaScript Component
To avoid code duplication, the code of a JavaScript Component can be stored in another JavaScript Component. The properties under the Linked Component category allow referencing the source JavaScript Component:
- PackageConnection - allows choosing the package, where the source JavaScript Component is (i.e. the current package or an external one).
- JavaScriptComponent - allows selecting the source JavaScript Component in the selected package.
If you have specified a source JavaScript Component, you can unlink it later by clicking the button Remove link below.
Displaying information about component metadata
This page displays information about important component properties, its inputs and outputs.
Script tab
The tab displays the JavaScript script code. It allows editing and adding breakpoints. The embedded editor is the one that comes with the particular version of Visual Studio. For detailed information how to create SSIS data flow scripts with JavaScript, please refer to the scripting reference.
At the bottom of the dialog, there are two buttons that make it convenient to load the JavaScript code from a file and save the code to a file.
Mappings tab
This tab defines mappings between external input columns and input columns. It's hidden by default and can be shown by script code.
Quick Start
In this quick-start we will setup two parameterizable JavaScript Components (a source and a transformation) to automate a simple scenario - generating dates and adding row counter. The source will generate rows with a date column, while the transformation will add another column with a row count.
Add parameters in the script
Let’s open the component editor and introduce two component parameters “StartDate” and “Count” by assigning them to component.parameters
. We will specify for them description
, type
and validation
. For validation we will use the library validation.js
where are defined the most frequently used validations.
Output column creation
Let’s create the output and the output column. As the columns are static, they will be created in component.reinitializeMetadata
function. We need one output and one column of type DT_DATE
. We make definition of the output and the output column. Helper functions will remove current when they are different and create new. To use helper functions the library component.metadata.js
needs to be included.
Summary
In summary, we have demonstrated how to setup two simple JavaScript Components: a source (having only an output) and a transformation (having an input and a synchronous output). Unlike the standard Script Component, please note, that both of them use a single method for processing the data component.processData(inputBuffer,outputs)
.
Knowledge Base
- Where can I find the documentation for the JavaScript Component?
- COZYROC JavaScript component: Unable to type in KeyAttribute or ValueAttribute
- Where can I find scripts for the JavaScript Task and the JavaScript Component?
- Microsoft Dataverse - A Template for Close Opportunities as WON or LOST
- How to configure the Javascript Task or JavaScript Component
What's New
- New: Commands to quickly Copy/Paste user parameters.
- Fixed: Visual Studio crash after initial script selection.
- New: Introduced component.
Related documentation
Why parameterize tasks and components?
SSIS tasks and components usually take as input a set of values (or objects) and apply some custom logic to achieve a goal. Separating the inputs from the processing logic provides several benefits:
- Gain extra clarity about the script inputs and their type
- Enable script reuse - within an SSIS package or even across an entire SSIS project.
Without parameters, reusing the script logic and feeding in different inputs can only be achieved by relying on SSIS variables. This often requires a separate step to initialize their values before calling each instance of the task/component that uses this script. Although feasible, this approach makes SSIS packages harder to maintain and also has some limitations (e.g. executing in parallel several instances of the script with different inputs).
JavaScript Task and JavaScript Component offer a superior approach, by enabling script parameterization with an easy-to-use UI. The list of available parameters are defined in the JavaScript code. In design-time they appear within the “Parameters” tab of the task/component editor. There the parameters values are managed in a standard Property Grid, with an assistive UI which enables entering values, selecting items (variables, connections, etc.) from a list and more. You are also able to specify validation rules, so that the task/component can provide an immediate feedback if the inputs are invalid.
Defining parameters
Parameters are specified by initializing the parameters property of the task
or component
like this:
The list of all available parameter properties are:
Parameter description
Description allows providing an extra information displayed to the user for documentation purposes.
Parameter type
The parameter type determines what editor would be used in the "Parameters" Property Grid for specifying the value. The supported types, grouped by category, are:
- Primitive types:
string
,integer
,numeric
,boolean
,date
- Custom editors:
list
,text-editor
,regex
,guid
- Files and folders:
file-local
,folder-local
,file-remote
,folder-remote
- SSIS common entities:
variable
,connection
,task
,component
- SSIS Component entities:
input
,output
,column
NOTE: Some types (e.g. the file-related ones) allow parameterizing the editor with additional settings. In this case, the type is represented by an object: type: { id: "connection", connectionTypes: ["SFTP", "SSH"] }
Parameter value
The value that's initially specified to value
is the default value. The user can change it later, without modifying the script.
With the script logic, the user-specified value is accessed via the value
property of the parameter (e.g. parameters.MyParameter.value
). It's available both in the validation and execution phases.
Validating values
The validate
function is invoked whenever the user changes the parameter's value. It is invoked also when the script is recompiled - on Editor opening, when Editor tab is changed, when OK button is pressed. To indicate that the value is invalid, the function needs to return a string with the error message text. In validation.js
file there are several functions provided (e.g. validation.required
or validation.greaterThan
) for handling common validation scenarios .
Reacting to changes in parameter value
The afterEdit
function allows plugging in logic that handles a change of a parameter value. It is invoked only when the user changes the parameter's value. It's useful e.g. for updating component metadata in the JavaScript Component
Grouping and sorting parameters
Category is used for grouping parameters in the Parameters tab. A parameter is always displayed under a particular category. If category
is not specified, the parameter would go to the default Misc group. Groups are displayed in an alphabetical order.
Parameters can be sorted within their group by using the property order
. The allowed value are integers (negative numbers are also accepted). It's recommended not to have equal values within same group, as the paramters order would be random. If no order is specified explicitly, the name of the parameter would be used for sorting. Parameters without order
are positioned after those that have it specified.
Optional parameters
Parameters that are marked as optional (via the optional
property) are not required to have a value and by default they are not displayed in the Parameters tab. To make them visible, the user needs to click the Expand Options button.
Disabling parameters
Parameters are enabled by default, but can be disabled via the enabled
property. Disabled parameters are displayed as read-only. The enabled
property can have an initial false
value or it can have an assigned function to calculate the enabled state.
Disabled parameters are used in the following scenarios:
- Permanently disabled parameter - used for displaying read-only calculated values (the value can be recalculated e.g. in
validate
function). - Initially disabled parameter - can get enabled later
- Conditionally disabled parameter - the
enabled
value is a function that decides whether or not to make it readonly, based on other parameters' values.
Hiding parameters
Parameters are visible by default, but can be hidden via the visible
property. The visible
property can have an initial false
value or it can have an assigned function to calculate the visible state.
Invisible parameters are used in the following scenarios:
- Initially invisible parameter - can become visible later (e.g. in a
validate
function call). - Conditionally invisible parameter - the
visible
value is a function that decides whether or not to display it, based on other parameters' values.
Defining dependencies between parameters
A parameter can be configured to depend on other parameter(s). Such a dependency means that the current parameter will be disabled until all its prerequisite parameters have a value specified. Changing the value of a prerequisite parameter would lead to resetting the value of all its dependent parameters.
The property depends
is used for defining such dependencies. Its value can be a string (for a single prerequisite parameter), an array of strings (for multiple prerequisite parameters) or a function returning string or array of strings (for dynamically changing prerequisite parameters). Parameters which require a connection (e.g. file-remote
) are considered automatically dependent on the corresponding property.
Shorthand definition of parameters
All the properties of parameters are optional. The quickest way to define parameters is by specifying only their name and initial value (assuming the type can be inferred from the value):
Shorthand parameters can be only of type string, integer, numeric, boolean or date. This syntax doesn't allow specifying a description, nor a category, so the parameters would end up in the default Misc group.
Parameters with custom selectors/editors (common)
Selection from a List of Values
Parameters can specify a list of allowed values via:
- Providing a fixed list of values (e.g.
values: ["value 1", "value 2"]
) - Providing a function which returns a list of values. (e.g.
values: function() { return ["value 1", "value 2"] }
). NOTE: The function is invoked within the context of the main object (task
orcomponent
), i.e. theirthis
is set to this object.
When a parameter type is list
, the user will be able to choose a value (or optionally multiple values) from a dropdown list:
- The values of a list must be of the same type. Only primitive types are allowed: string, integer, numeric, boolean or date.
- In multiple-selection lists the returned
value
is always a string with comma-delimited values (i.e. even if the allowed values are integers, the selected ones would still be converted to strings). The delimiter is the string ",\n" (comma followed by the line feed), it allows usage of comma in values.
Here is an example of a function that retrieves the list of allowed values from a DB table (via an ADO.NET Connection):
NOTE: In the example above, the parameter Database will remain disabled until a connection manager is selected into the parameter ADONETConnection (due to the specified depends property). When the value of the master parameter ( i.e. ADONETConnection ) is changed, the values of all parameters depending on it would be cleared.
Selection of a Variable
When the parameter value is expected to be stored in a SSIS variable, you can use the variable
parameter type, which allows easy selection of a variable from a dropdown list. The name of the selected variable would be stored in the parameter value.
Supported variable types are string
(any text SSIS data type), integer
(any integer SSIS data type), numeric
(any floating SSIS data type), boolean
, date
and object
.
If variableTypes
is not specified, all available system and user variables will be listed. For that scenario, you can define the parameter using a simpler syntax:
Selection of a Connection Manager
You can use connection
type for selecting an available Connection Manager(s) in the package:
Tip: The IDs of the connection types can be seen in the Visual Studio "Add SSIS Connection Manager" dialog. The values in the column "Type" are the allowed values for the property connectionTypes
.
Text Editor (XML, SQL, JSON)
For parameters which value is expected to contain text that can't be comfortably displayed in a single line (e.g. an XML fragment or an SQL query), there is a syntax highlighting-capable editor provided via the text-editor
type, that supports XML, SQL, JSON, etc.
If such a parameter has avalidate
function, it will be executed upon pressing the Text Editor's OK button. Unlike other parameters, where the validation is executed after the value is set, this one allows catching invalid values early on to prevent applying them.
Regular Expression Editor
For parameters of regex
type, there is available a RegEx editor that allows testing the regular expression.
Guid Editor
For parameters of guid
type, there is inplace editor available which performs Guid validation.
Selection of Task/Component
Based on the provided in SSIS+ Package Connection there are special parameter types (task
and component
) that support selecting tasks or components from the SSIS package:
The connection
property is providing the information which other parameter (of type PackageConnection) should be used, and also implies dependency, thus the depends
property given afterwards is redundant. As with other properties, connection
can be a function which returns a string. Arrays cannot be used as value for this parameter.
The properties taskType
or componentType
are optional. If the type is not specified, all tasks/components in a package will be available for selection.
Filtering by component type is do by searching componentType
in the component property UserComponentTypeName (many components don't have such property) and in the property IdentificationString.
Filtering by task type is done by searching taskType
in the .NET type name of the task and in the task Name property.
Selection of Local File(s) and Folder
Selecting a file from the local file system is specified via the file-local
type:
- Property
multiple
is optional, if omitted single selection will be made. - Property
filter
is optional, if omitted no filter will be used.
When none of the optional properties are used, the following shorthand syntax can be used:
Selecting a folder from the local file system is specified via the folder-local
type
Selection of Remote File or Folder
To select remote files (file-remote
) or remote folder (folder-remote
), you would need a remote connection. "SFTP" and "SSH" connections can be used here (the implementation relies on having a connect
method that returns an object implementing CozyRoc.SqlServer.SSIS.IFtpClient
).
Filtering of Local and Remote Files
Since SSIS+ version 2.0, you can specify a filter for local files (file-filter-local
) or remote files (file-filter-remote
) via the following syntax:
JavaScript Component-specific parameters
Selection of an input
A parameter of type input
allows selection of one of the component inputs:
Selection of an output
A parameter of type output
allows selection of one of the component outputs. The selection is made from all regular or all error outputs of the component output collection. Whether the outputs include error outputs is determined from the property all
, default value is false
, which means only regular outputs are listed. Otherwise all outputs are listed - regular and error.
Selection of input or output columns
A parameter of type column
allows selection of input or output columns.
The property columnType
has the following possible values:
- For input columns: "all" - all columns from the parent output, "selected" - selected columns for read-only or read-write use, "external" - external metadata columns.
- For output columns: "all" - all configured columns into given output, "external" - external metadata columns.
The property containerName
is optional. If omitted, the first input or output will be used.
The property multiple
is optional. If omitted, only single selection will be allowed.
An example for selecting a virtual column:
After a virtual column is selected it should be marked as being used. This easily can be done in the afterEdit
function. In the example above, the selected virtual columns are marked for read-only use, the non-selected are marked as ignored.
An example for selecting an input column:
An example for selecting an output column:
Selection of external columns
When external
is specified for columnType
, first you need to call externalMetadataColumnCollection.isUsed = true
for the desired input or output (usually in afterEdit
function) like this:
The creation of external metadata columns can be done in the reinitializeMetadata
function, too.
External input column:
External output column:
Using default value with component parameters
Component parameters can have default value - a name of an input, an output or a column.
The default value can be used when multiple selection is allowed. In this case, the default value will be indicated only when a single selection is made. Also when the selection is cleared, the parameter value is reset to the default value.
Validation
Two levels of parameter validations are supported (local and global):
Single parameter validation
Parameters can have a validation function assigned to the validate
property. Its main purpose is to verify if the value is valid, and if not - to return an error message. When the value is valid, the function should return nothing.
Besides the validation, this function can be also used to change the value
, enabled
and visible
properties of any parameter, but such a usage is not recommended.
The validation function has a single parameter, containing the current value of the parameter. Within the function this
keyword refers to the corresponding task
or component
object.
Examples of validating a value:
In the following script the parameters FileVariable
and File
are hidden, based on the value of the IsFileVariable
.
Global validation (validating values of multiple parameters)
Whenever you need to verify the values of more than one parameter at once, you can use a validate
function to do so. The so-called global validation will be executed in multiple occasions:
- during the Validation phase of the task/component
- after opening of the Editor
- after changing the value of any parameter
The global validation is executed after the individual parameter validations. Similarly to the individual validation, it should return a string if an error is present. Within the function this
keyword refers to the corresponding task
or component
object.
Knowledge Base
Related documentation
This programming reference covers various aspects of how to use JavaScript to automate SSIS-related tasks. While some of the content is specific to JavaScript Task or JavaScript Component, most of it is applicable in the context of any JavaScript script.
Global scope objects and methods
Within the JavaScript code there are several objects and methods that can be used anywhere:
Global scope objects
JavaScript Task defines a task
object and JavaScript Component defines a component
object, which are available in the global scope of the corresponding script.
Besides this main object, there are several objects with convenience methods:
console = {...};
- it supports all standard methods of the console object + offering additional SSIS-specific methods likeprogress
.env = {...};
- used for getting and setting values of Environment Variables
Global scope functions
The following functions can be used anywhere in script code:
require = function(<additional_file_to_import>) { ; }
- imports a JavaScript module from a file located under<COZYROC SSIS+ install folder>\SSIS\JS\
(there are several popular JS files, already available there)alert = function(<message_box_content>) { ; }
- when in interactive mode, displays a message box with some text. In non-interactive mode, outputs the message to the execution log.sleep = function (<delay_in_milliseconds>) { ; }
- pauses the script for the specified number of milliseconds.getClrType = function (<fully_qualified_CLR_type_name>) { ; }
- gets a .NET type (more details below)
Accessing Parameters, Variables and Connections
The following properties are accessible for both JavaScript Task (via its task
object) and JavaScript Component (via its component
object):
Script Parameters
The script parameters are exposed via the parameters
property of the task/component. You can access them like this: component.parameters.MyParameter
For more details why and how to use parameters, see here.
SSIS Variables and Parameters
All SSIS Variables (User and System) and Parameters (Project and Package) are exposed via the variables
property of the task/component, which is a wrapper around a VariableDispenser
object. Variables are accessible only during the execution stage of the task / component.
If there are no variables/parameters with duplicated names, a short form can be used like this:
Changing a value is allowed only for the User Variables. System Variables, Project and Package Parameters are read-only. Attempting to set their value will throw an exception.
When there are duplicate names, the long form (with ::
) should be used:
Each variable is locked on-the-fly for read or for read/write, depending on the access of the .value
property. However, there are the following two functions exposed for a variable, which provide fine-grained control over locking and unlocking: lockShared()
and lockExclusive()
. They can be used like this:
More about variables locking:
- Explicit locking is used to speed up execution when a variable is used often within the script. Please note that locking a variable or a parameter is a relatively slow operation.
- The locking/unlocking mechanism is counter based, so on-the-fly locking/unlocking which happens inside
.value
will not result in additional underlying lock/unlock invocations, if an explicit call tolockShared()
(orlockExclusive()
) has been made already. - If you've used
lockShared()
on a variable and try to write in it, it'll succeed. However, as the name suggests, there is no guarantee for not having race conditions. If you want an exclusive lock just for this task/component, then uselockExclusive()
. Calling the latter function after another lock has been already made (i.e. call tolockShared()
orlockExclusive()
) will result in throwing a dead-lock prevention exception. - If a variable is locked exclusively and another task/component is trying to lock it, locking will wait until the exclusive lock is released.
- Explicit unlocking is optional, as all locked variables will be automatically unlocked after the execution of the script.
Various properties/methods for access to the underlying Microsoft.SqlServer.Dts.Runtime.Variable
object are available like readOnly
, description
, etc.
SSIS Connection Managers
Scripts can use SSIS Connection Managers both during the global scope execution and during expressions execution.
The currently supported Connection Managers include:
- Standard: ADO.NET, HTTP, FTP, FILE, FLATFILE
- COZYROC SSIS+: SFTP, SSH, REST, PACKAGE, EXCHANGE, IMAP
- Any other Connection Manager that is returning a string value as a result of calling
AcquireConnection
(just like FILE, FLATFILE )
Example of using an ADO.NET connection:
The result from a query is an array of objects. Array elements correspond to a table row, i.e. each row is represented by an object. The property names on this object start from 0 for the first column, and end with count of columns minus one.
A field from the query result can be accessed like this:
Example of using a REST connection:
Example of using a FILE connection:
FILE Connection Manager returns file name, so this connection doesn't needs closing. The file can be opened or closed.
.NET Interoperability
Using .NET types
The script can access any .NET types by using its type name, including namespace or assembly qualified name. The function getClrType
is used like this:
Calling a static method:
Creating an instance:
Creating a generic instance:
Once you have a reference to an object, getting and setting the properties is done like this:
There are a few limitations/gotchas to be aware of, related to calling .NET methods:
Out
andRef
arguments are not supported.- As JavaScript can convert automatically from a given type to another this might lead to an ambiguous call issues. For example, a C# class is having two methods with the same name, one has a
double
parameter, the other hasfloat
. If the method is called usinginteger
value, it would be ambiguous and an exception will be thrown.
Implementing .NET Interfaces
The script can create an .NET interfaces implementations. The function implementClrInterfaces
is used like this:
The object DataReaderEnumeratorImpl
contains the interface implementations. All calls to interface methods or properties are redirected to this class. It's not required for all interface members to be defined. When an undefined member is used, an 'NotImplementedException' is thrown. A JavaScript object can implement more than one interface, like in the above example.
The object DataReaderEnumeratorImpl
is from the Data Flow Enumerator Task script and looks like this:
The methods and properties of the JS object should have same names as the methods and properties of the .NET interface.
COM Interoperability
The script can create and use COM objects instances. Only COM objects implementing IDispatch
can be used. Events are supported, the connection point interface should also implement the IDispatch
interface.
The COM object type is loaded using getCOMType
function, like this:
The function getCOMType
accepts two parameters. The first one is mandatory and specify the COM object "Prog ID". The second parameter is the object locale and is optional:
The instance is created as usual:
Next the COM object is used like any other JS object:
Methods of the COM object are invoked like any other JS function. The COM object methods cannot be assigned to variables, though. The following is NOT allowed:
Proper calling is this:
Usage of regular properties:
Default indexer properties with numeric index can be used with square brackets:
Named indexer properties, indexer properties with more than one indexer and indexer properties with non-numeric index are accessed via functions. When getting a property value, the optional prefix "get_" can be used, but direct usage of the property name is also supported.
Using property getter:
When setting property the prefix "set_" is mandatory:
Event handlers can be attached by using "add_" prefix before the event name:
Only one handler for an event can be attached.
Removal of the handler is done by using "remove_" prefix before the event name:
The COM object can be released any time by calling the release
function:
All events handlers are removed before release.
A COM object instance has one special function, called changeLocale
to modify the locale of the object. This function is useful when the COM type is not obtained via getCOMType
.
JavaScript Task Execution
By default, the script execution starts with the task.run
method, but you can specify another function (or any valid JavaScript statement) in the task "Expression" property.
All SSIS task need to set an ExecutionResult
and JavaScript Task can return one the values below:
Explicitly returning a ScriptResults
in your scripts (in task.run
or in the Task Expression) is rarely necessary:
- If the script completes normally, then
DTSExecResult.Success
value is assumed. - If the script execution is aborted due to an exception a
DTSExecResult.Failure
is assumed. - The last two values that correspond to
DTSExecResult.Completion
orDTSExecResult.Canceled
are to be used only in scripts with very special requirements.
In addition, any value returned by task.run
(not necessarily ScriptResults
) will be saved in the task's "ExecutionValue". This would allow the task "ExecValueVariable" property value to be used by downstream logic.
JavaScript Component Execution
Processing input and output buffers
The component.processData
function is responsible for processing buffers of inputs and outputs.
The function has two arguments:
inputBuffer
- the currently processed input buffer (if any)outputs
- an object, which can contain multiple output buffers. A particular output buffer can be retrieved by the name of the corresponding output (e.g.outputs["Output 0"]
) or via the propertiessync
orasync
, which return arrays. Thesync
array contains all output buffers which are synchronous with the current input. Theasync
array contains all asynchronous output buffers.
The component.processData
function can be called multiple times per execution:
- When at least one asynchronous output is present, the first call of
component.processData
will be only with the asynchronous outputs and theinputBuffer
parameter will have anull
value. - Then
component.processData
is called once for each input. If this input has synchronous outputs, they will be present in theoutput
object. The asynchronous outputs are always present in theoutputs
object.
Columns values can be get and set by using the buffer functions get
and set
:
- The
get
function has one argument which is the original index of the column or the column name:inputBuffer.get(0)
orinputBuffer.get("RowId")
. - The
set
function has two arguments, the first is the original index of the column or its name, the second is the new value:outputBuffer.set(1, "1st Avenue")
oroutputBuffer.set("Street name", "1st Avenue").
A simple example of processing a synchronous output:
A simple example of processing an asynchronous output:
Pre and post execute
The two methods would be called before and after the calls to component.processData
:
component.preExecute = function ();
- executed during the component "pre execute" phase. Here you can acquire connections and lock variables.component.postExecute = function ();
- executed during the component "post-execute" phase. Here you can release connections and unlock variables, if necessary.
Reinitializing metadata
An advanced technique is to create or modify programatically the component metadata via the component.reinitializeMetadata
method. Implementing this method enables creating, deleting or altering inputs, outputs and columns.
An example with initialization of a transformation with an error output:
Metadata changes can be simplified by using the component.metadata.js
library. It contain helper functions for metadata manipulations. To use the library execute following line:
Function to change inputs is syncInputs
.
Synchronize SSIS component inputs with the specification in the provided inputData
array. Upon success, all inputs will have the properties, specified in inputData
. Extra inputs will be deleted.
The argument component
is the JavaScript Component object.
The argument inputData
is an array of objects with the following properties:
name
- input name (mandatory)description
- input description (optional, default: "")isUsedExternalCollection
- is the external metadata column collection used (optional, default: false)hasSideEffects
- does the component performs custom actions that are not visible to the data flow engine (optional, default: false)errorRowDisposition
- how to handle errors (optional, default: "DTSRowDisposition.RD_NotUsed")truncationRowDisposition
- how to handle truncated values (optional, default: "DTSRowDisposition.RD_NotUsed").
Function to change outputs is syncOutputs
.
Synchronize SSIS component outputs with the specification in the provided outputData
array. Upon success, all outputs will have the properties, specified in outputData
. Extra outputs will be deleted.
The argument component
is the JavaScript Component object.
The argument outputData
is an array of objects with the following properties:
name
- output name (mandatory)description
- output description (optional, default: "")isErrorOut
- is an error output (optional, default: false)hasSideEffects
- does the component performs custom actions that are not visible to the data flow engine (optional, default: false)synchronousInput
- name of an input that is synchronous to the output (optional, default: "")exclusionGroup
- ID of an exclusion group (optional, default: 0)isSorted
- is data in the output sorted (optional, default: false).
Function to change columns is syncColumns
.
Synchronize regular or external columns of an input or an output with the provided columnData
. Upon success, all columns will have the properties, specified in columnData
.
Checks if the current columns in the input/output matches the specification in columnData
. If not, the columns are recreated with the data in the supplied columnData
.
Automatically keeps input external columns mappings when a new input is connected and an input column in the new input has the same name as an input column from the old input. Creates mappings between columns with the same name.
The argument io
is either an input or an output for which to synchronize columns.
The argument columnData
is an array of objects with the following properties:
name
- column name (mandatory)dataType
- column SSIS data type (mandatory)length
- the length of the column (optional, default: 0)precision
- the number of digits contained in a column (optional, default: 0)scale
- the scale of the column (optional, default: 0)codePage
- the code page of the column (optional, default: 0)description
- column description (optional, default: "")
The argument isExternal
is a boolean, indicating which columns are synchronized - regular or external
(optional, default: true
for input columns, false
for output).
Returns true
if no difference were detected.
Function to use input columns with given names is useInputColumns
.
In the specified SSIS input set READONLY usage for the columns in the "columnNames" array and remove any columns, which names are not in the array.
Handling adding or deleting of input/output paths
If you need to handle changes in paths (i.e. connections from JavaScript component to other components), the following methods are available for inserting custom logic:
component.onInputPathAttached = function (inputID)
component.onInputPathDetached = function (inputID)
component.onOutputPathAttached = function (outputID)
Knowledge Base
Related documentation
-
ADO.NET Command Transformation
Execute SQL command for each row (like the standard OLE DB Command Transformation, but with ADO.NET).
-
AS2 Send Task
Send electronic documents (EDI, XML) using AS2 specification.
-
Checksum Transformation
Calculate CRC32 or SHA256 checksum for one or more input columns.
-
Column Compression Transformation
Compress or decompress columns.
-
Column Encryption Transformation
Encrypt or decrypt one or more input columns.
-
Column To Variable Destination
Store the values from the input columns into SSIS variable(s).
-
Conversion Transformation
Change data type, length, precision, scale of selected input columns.
-
Counter Transformation
Add a row counter column (e.g. can be used for surrogate keys).
-
Data Flow Enumerator Task
Process data flow result into control flow.
-
Data Record Source
Retrieve data from variable containing IDataRecord object.
-
Data Record To Variable Task
Extract field information from IDataRecord object into SSIS variables.
-
Data Verification Destination
Verify the output of an upstream component by comparing it against predefined data.
-
Dataset Destination
Load data in an in-memory ADO.NET DataSet object (similar to standard Recordset Destination).
-
Date Format Transformation
Reformat dates in text columns.
-
Date Source
Generate rows for a range of dates (by specifying a start date, an end date and an increment step).
-
Dictionary Destination
Loading input into a dictionary object.
-
Dynamics CRM Access Teams Destination
Add or remove a user to the auto created access team for the specified record.
-
Dynamics CRM Assign Destination
Assign record to a different owner.
-
Dynamics CRM Association Destination
Create associations between Dynamics CRM entities.
-
Dynamics CRM Attributes Source
Retrieve Dynamics CRM entity attributes metadata.
-
Dynamics CRM Audit Logs Query Transformation
Retrieve Dynamics CRM audit log records.
-
Dynamics CRM Bulk Delete Task
Perform Dynamics CRM bulk delete.
-
Dynamics CRM Convert Transformation
Convert Dynamics CRM entity record from one type to another.
-
Dynamics CRM Entity Changes Source
Retrieve Dynamics CRM entity changes records
-
Dynamics CRM Entity Source
Retrieve Dynamics CRM entity metadata.
-
Dynamics CRM Execute Workflow Destination
Execute Dynamics CRM workflows.
-
Dynamics CRM Incident State Destination
Update incident entity state.
-
Dynamics CRM Lookup Transformation
Perform lookup against Dynamics CRM.
-
Dynamics CRM Merge Destination
Merge two Dynamics CRM entity records into one
-
Dynamics CRM Opportunity State Destination
Update opportunity entity state.
-
Dynamics CRM OptionSet Transformation
Handle translation and creation of Dynamics CRM OptionSet values.
-
Dynamics CRM Qualify Lead Destination
Qualify a lead and create account, contact and opportunity records.
-
Dynamics CRM Quote To Sales Order Transformation
Convert Dynamics CRM quote to a sales order.
-
Dynamics CRM Relationship Destination
Create Dynamics CRM relationships between entities.
-
Dynamics CRM Route Destination
Route Dynamics CRM entity record to a queue.
-
Dynamics CRM Team Members Destination
Establish team members relationship.
-
Dynamics CRM User Access Destination
Establish user access permissions to Dynamics CRM entities.
-
Error Description Transformation
Extract additional information (column names and error descriptions) for error outputs in a data flow.
-
Excel Copy Worksheet
Copy a worksheet from one Excel file into another.
-
Excel Refresh Task
Refresh Excel connections and pivot tables.
-
Execute Salesforce DML Destination
Executes anonymous Salesforce Apex DML code.
-
Execute Salesforce DML Task
Executes anonymous Salesforce Apex DML code.
-
File Checksum Task
Calculate MD5 or SHA1 checksum for a file.
-
File Exists Task
Check if a specified file exists.
-
File Info List Task
Manage a list of SSIS+ IFileInfo objects (i.e. remote files on FTP, SFTP, S3, etc.).
-
File Info Source
Generate rows from IFileInfo list (i.e. remote files on FTP, SFTP, S3, etc.).
-
File Properties Source
Generate rows from files properties.
-
File Watcher Task
Detect changes in a monitored folder.
-
Find And Replace Task
"Find" or "Find & Replace" in a text file or a stream.
-
FTP File Exists Task
Check if a remote file exists (via SFTP or FTPS).
-
FTPS Task
Demonstrate FTP over SSL (FTPS).
-
Geocode Transformation
Retrieve via Google Geocoding and cache coordinates for input addresses.
-
Get Analysis Services Object Property Task
Get Analysis Services object property.
-
Get Environment Variable Task
Retrieve environment variable.
-
Get File List Task
Get files list.
-
Get FTP File List Task
Retrieve a list of remote files on an FTP server.
-
HTTP Upload Download Task
Upload or download a file over HTTP.
-
IMAP Task
Provide management functions for IMAP email server.
-
Init Task
Demonstrate SSIS script initialization setup.
-
Join Column Transformation
Combine multiple rows into single data flow row.
-
JSON to Variables Task
Bulk extract elements from JSON document into SSIS variables.
-
JSON Transformation
Retrieve data from a text column, containing an array of objects in JSON, XML or other text format.
-
Levenshtein Distance Transformation
Calculate difference between two strings using Levenshtein Distance.
-
Mail Info List Task
Manage IMailInfo list.
-
Mail Info Source
Generate rows for a IMailInfo list.
- Mail Task
-
Map Replace Transformation
Perform find-and-replace using dictionary object.
-
Measure Duration Transformation
Measures the processing time of the pipeline it is attached to.
-
Merge Setup Transformation
Merge temporary stage table using SQL 2008 MERGE (Transact-SQL) statement.
-
Microsoft Dataverse Lookup Transformation
Perform lookup against Microsoft Dataverse.
-
NetSuite File Download
Script task to download a file from NetSuite File Cabinet
-
NetSuite File Upload
Script task to upload a file to NetSuite File Cabinet
-
NULL Transformation
Setting and replacement of NULL values in the input columns.
-
OData Source
Consume data from an OData feed (similar to the standard OData Source).
-
OFTP Task
Send and receive EDI documents via ODETTE protocol.
-
Oracle Destination
Bulk-load data in Oracle table.
-
Package Configuration Task
Customize an SSIS package externally via configuring its properties in an Excel sheet.
-
PayPal Transactions Source
Retrieve PayPal transactions information.
-
Power BI Report Export Task
Exports Power BI report.
-
Profiler Trace Source
Load profile trace information.
-
Range Dictionary Destination
Create custom-crafted IDictionary object with representation of "many to one" relationships.
-
RegEx Extract Transformation
Extract matching groups from a given regular expression and output them as column values.
-
RegEx Match Transformation
Split the input based on matching with provided regular expression.
-
Reorder Columns Transformation
Perform columns reorder.
-
REST CSV Source
Retrieve data from CSV File.
-
REST Web Service Request Task
Send a request to a REST API endpoint.
-
REST Web Service Request Transformation
Execute REST web service request for each data flow row.
-
Row Repeat Transformation
Asynchronously output replicated rows based on repeat count, provided in a column.
-
RSS Source
Load information from RSS feed.
-
Salesforce Bulk Result Transformation
Return Salesforce bulk-load batch information.
-
Salesforce Fields Source
Generate rows from the Salesforce object fields metadata.
-
Salesforce Get Deleted Source
Retrieve Salesforce object deleted record identifiers.
-
Salesforce Get Updated Source
Retrieve Salesforce object updated record identifiers.
-
Salesforce Lookup Transformation
Perform lookup against Salesforce objects.
-
Salesforce Object Source
Retrieve Salesforce objects metadata.
-
Secure Tunnel Task
Establish a secure tunnel using SSH protocol.
-
Send Mail Task
Send SMTP email (supports HTML messages and attachments).
-
SFTP Connection Exists Task
Check if an SFTP connection can be established.
-
Soundex Transformation
Create Soundex representation of input column.
-
Split Column Transformation
Split single rows into multiple data flow rows.
-
SQL Job Agent Task
Provide basic manipulation operations for SQL Agent Jobs.
-
Table Update Destination
Easier to use alternative of the standard OLE DB Command transformation for amending table records.
-
Test Data Source
Define tabular input data for testing purposes.
-
Time Zone Transformation
Convert date/time columns from one timezone to another.
-
Timer Task
Wait specified amount of seconds before continuing.
-
Transform XML Task
Transform an XML document using an XSLT file.
-
Trash Destination
Easily terminate a data flow path (as a development aid)
-
Trim Transformation
Trim input columns.
-
UncompressZ Task
Decompress data compressed with the unix "compress" utility (LZC, a LZW variant).
-
Variable to File Task
Write string Variable contents to a File.
-
WMI Source
Load data from Windows Management Information (WMI).
-
XPath Extract Transformation
Extract information from a column with XML documents, using provided XPath expressions.
Derived Column
- New: Included
encodeBase64Bytes
anddecodeBase64Bytes
functions to work with DT_BYTES, DT_IMAGE columns. - Fixed: Many improvements and cleanup.
- New: Introduced component.
JavaScript Component
- New: Commands to quickly Copy/Paste user parameters.
- Fixed: Visual Studio crash after initial script selection.
- New: Introduced component.
JavaScript Task
- Fixed: An ocassional error "The directory is not empty" in the post-execution cleanup (Thank you, Saikit).
- New: Commands to quickly Copy/Paste user parameters.
- Fixed: Incorrect variable replacement in JSON.
- Fixed: Disabled various destructive commands in the script editor.
- Fixed: Improved memory consumption and management.
- Fixed: Removed not needed exception popup dialog because the error is already reported in the log.
- New: Introduced task.
Knowledge Base
- Where can I find scripts for the JavaScript Task and the JavaScript Component?
- Where can I find the documentation for the JavaScript Component?
- Where can I find the documentation for the JavaScript Task?
- How to configure the Javascript Task or JavaScript Component
- Microsoft Dataverse - A Template for Close Opportunities as WON or LOST
COZYROC SSIS+ Components Suite is free for testing in your development environment.
A licensed version can be deployed on-premises, on Azure-SSIS IR and on COZYROC Cloud.