bulk insert behavior - impact on the package's performance


i'm seeing an increase in run times between client development world , production.  on client, using production volumes, package runs in hour.  on production runs 80-90 minutes.  both environments using remote file servers data sources.  production has luxury of not having cross server joins development requires 1 , still runs faster.  input data source contains 2.5 million records can produce many times number of errors on error table , call duplicates unchanged records on duplicates table.

i've checked see if indexes didnt make prod, no such luck.

so i'm starting @ opportunities efficiency , explanations differences.

i've got stats 1 or both environments specific things , focused on bulk inserts @ moment.  bulk inserts being requested in package via "table or view - fast load option" no lock, no check constraint, no keep identity, no keep nulls, no rows/batch , no max insert commit size.

when watch graphics of client pkg running,  appears records being inserted 3 bulk inserts (import, error, work) tables data moving thru buffer.  i'm guessing isnt started until of data ready insert.  true?

my dba ran profiler , saw 44 minutes elapsed  (they overlap)  in 3 pkg bulk inserts.   error table shared amongst number of apps (i make own staging one) import , work owned app. 

might better performance using different option inserts import , work tables if records allowed "trickle" these tables  as buffer moving?  maybe with lock turned on?   assuming error table is dealt somehow so doesnt become critical path completion of pkg?      

 

validating data bulk inserted prior insertion doesn't seem efficient.  feasible disable check constraints, bulk insert data, validate bulk inserted data using t-sql?  realize may involve alot of "heavy lifting" database engine.  however, imagine faster attempting validate data using client-side code.

hth
duane douglas, mcad | http://www.ssisbi.com | please mark post(s) answered question.


SQL Server  >  SQL Server Integration Services



Comments

Post a Comment

Popular posts from this blog

Azure DocumentDB Owner resource does not exist

having same error microsoft.azure.documents.documentclientexception: message: {"errors":["owner resource not exist"]} , scenario. when deployed webapp azure , try document docdb throws error. docdb exists in azure , contains document looking for. weird local machine(running vs) works fine. i'm using same settings in azure , local. have , idea this. thanks Microsoft Azure  >  Azure Cosmos DB
Read more

job syspolicy_purge_history job fail in sqlserver 2008

Image
hello guys, when installed sqlserver,it automatically created 1 job name job syspolicy_purge_history. but failed @ step 1. the reason is: date                11/6/2010 2:00:00 am log                job history (syspolicy_purge_history) step id                1 server                mxosv-otsdb01 job name                syspolicy_purge_history step name                verify automation enabled. duration                00:00:00 sql severity                16 sql message id                34022 operator emailed                operator net sent                operator paged                retries attempted                0 message executed user: nt authority\system. policy automation turned off. [sqlstate 42000] (error 34022).  the step failed. have run statement: select msdb.dbo.fn_syspolicy_is_automation_enabled the returned valus 0. what can resolve it? or can delete job manually.   thanks in advance. if haven't things want,be
Read more

Trying to register with public marketplace error with 'Get-AzureStackStampInformation'

when trying register against public azure adfs azure stack error below. logs can found at: c:\maslogs\azurestack.azureregistration.2017-10-12.01-31-23.log  and  \\azs-ercs01\c$\maslogs error occurred checking stamp information: system.management.automation.remoteexception: term 'get-azurestackstampinformation' not recognized name of cmdlet, function, script file, or operable program. check spelling of name, or if path included, verify path correct , try again. @ c:\azurestack-tools-master\registration\registerwithazure.psm1:1106 char:5 +     throw "logs can found at: $global:azureregistrationlog  and  \ ... +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~     + categoryinfo          : operationstopped: (logs can fou... , try again.:string) [], runtimeexception     + fullyqualifiederrorid : logs can found at: c:\maslogs\azurestack.azureregistration.2017-10-12.01-31-23.log  a    nd  \\azs-ercs01\c$\maslogs error occurred chec
Read more