SSIS :Parallelism in ssis

Advantage of Parallelism  in ssis


1.       Parallelism is ssis : Parallel Execution improves performance on Systems  that have multiple physical or logical processors. For this  SSIS uses two properties: MaxConcurrentExecutables and EngineThreads.

The MaxConcurrentExecutables property is a property of the package. This property defines how many tasks can run simultaneously; by specifying the maximum number of SSIS threads that can execute in parallel per package. The default value is -1, which equates to the number of physical or logical processors plus 2

The EngineThreads property is a property of each Data Flow task. This property defines how many threads the data flow engine can create and run in parallel

  Simple Usecase  , where I have implemented Parallelism.
I am taking simple work flow which I have implement in my requirement .
Need to call the web service  with the values in each row of the  FinalObjects  table.
For example : if the FinalObjects  table contains 50 rows we need to call the web service 50 times .

Normal implementation  :  we need to assign the values to the variables  from FinalObjects  table based on iterator value in for each loop and then call a web service task with parameters as  the above variables. And then Audit those values into another table .


The total process time for the above task is   2 min 46 sec and complete package takes  2 min 52 secs

Solution :
As  records are independent  and these tasks can be done separately ,so   I have used the parallel process to reduce the time of processing .  The default  value for MaxConcurrentExcutables is -1 which is enough for my Case as iam using only 2 parallel tasks .
1 st parallel task take care of 1st half of the records of finalobjects table and 2 nd parallel task take care of 2nd half of the records of FinalObjects table.

By making the process parallel , the total time taken for above task is  is  1.26 mins
And complete package is 1.33 mins

Conclusion :
If you have N number of parallel process  and total time taken by sequence process is M  then by implementing parallel process the total processing would be approx  M/N
 

Comments

Post a Comment

Popular posts from this blog

SSAS : A connection cannot be made to redirector. Ensure that 'SQL Browser' service is running. (Microsoft.AnalysisServices.AdomdClient). + Solution

Problem : have installed Analysis service on my server machine through remote login from my service account. I am able to open Analysis services from there. When I open SSMS from the same service account on my local machine, I am not able to connect to that Analysis service. It is giving me error by saying : "A connection cannot be made to redirector. Ensure that 'SQL Browser' service is running. (Microsoft.AnalysisServices.AdomdClient). A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connection host has failed to respond 10.245.192.111:2382(System)." Solution : Based on my research, the issue was caused by the Browser Service account was changed to the domain account. In your scenario, you can try to connect to the Analysis Service using <Servername>:2383 instead of instance name. Or you can change the Browser Service account to "Local System".
Read more

SSIS Error : Unable to infer the XSD from the XML file. The XML contains multiple namespaces

Image
Problem : When we have Multiple Namespaces in the Xml Source which we need to load the Data from   as we shown below Then we will face a issue in generating XSD for that XML Error : Unable to infer the XSD from the XML file. The XML contains multiple namespaces. (Microsoft Visual Studio) Solution : First we need to avoid multiple namespaces to transform your source file to a format that doesn’t refer to the namespaces. SSIS has an XML task that can do the transformation. 1) Add the XML task to an SSIS   Control Flow and edit it. 2) Change the OperationType property value to XSLT, the SourceType to File connection and the Source to your source file that has the problem. 3) Set the SaveOperationResult property to True and expand the OperationResult branch. Set DestinationType to File Connection and the Destination to a new XML file ( ModifiedInput.XML). 4) Add the below script to a new file and save it with an xslt file extension (Input.XSLT) <
Read more