Azure Data Factory - Multiple activities in Pipeline execution order


i have 2 blob files copy azure sql tables. pipeline 2 activities:

{
    "name": "nutrientdatablobtoazuresqlpipeline",
    "properties": {
        "description": "copy nutrient data azure blob azure sql",
        "activities": [
            {
                "type": "copy",
                "typeproperties": {
                    "source": {
                        "type": "blobsource"
                    },
                    "sink": {
                        "type": "sqlsink",
                        "writebatchsize": 10000,
                        "writebatchtimeout": "60.00:00:00"
                    }
                },
                "inputs": [
                    {
                        "name": "foodgroupdescriptionsazureblob"
                    }
                ],
                "outputs": [
                    {
                        "name": "foodgroupdescriptionssqlazure"
                    }
                ],
                "policy": {
                    "timeout": "01:00:00",
                    "concurrency": 1,
                    "executionpriorityorder": "newestfirst"
                },
                "scheduler": {
                    "frequency": "minute",
                    "interval": 15
                },
                "name": "foodgroupdescriptions",
                "description": "#1 bulk import foodgroupdescriptions"
            },
            {
                "type": "copy",
                "typeproperties": {
                    "source": {
                        "type": "blobsource"
                    },
                    "sink": {
                        "type": "sqlsink",
                        "writebatchsize": 10000,
                        "writebatchtimeout": "60.00:00:00"
                    }
                },
                "inputs": [
                    {
                        "name": "fooddescriptionsazureblob"
                    }
                ],
                "outputs": [
                    {
                        "name": "fooddescriptionssqlazure"
                    }
                ],
                "policy": {
                    "timeout": "01:00:00",
                    "concurrency": 1,
                    "executionpriorityorder": "newestfirst"
                },
                "scheduler": {
                    "frequency": "minute",
                    "interval": 15
                },
                "name": "fooddescriptions",
                "description": "#2 bulk import fooddescriptions"
            }
        ],
        "start": "2015-07-14t00:00:00z",
        "end": "2015-07-14t00:00:00z",
        "ispaused": false,
        "hubname": "gymappdatafactory_hub",
        "pipelinemode": "scheduled"
    }
}

as understand, once first activity done, second starts. how execute pipeline both activities 2 blobs execute 1 after another(constrains in database), instead of going dataset slices , run manually? also pipelinemode how can set onetime only, instead of scheduled?

hi amel,

the following help:

ordered copy

it possible run multiple copy operations 1 after in sequential/ordered manner. have 2 copy activities in pipeline: ca1 , ca2 following input data output datasets.

ca1: input: dataset1 output dataset2

ca2: inputs: dataset2 output: dataset4

ca2 run if ca1 has run , dataset2 available.

in above example, ca2 can have different input, dataset3, need specify dataset2 input ca2 ca2 not run until ca1 completes. example:

ca1: input: dataset1 output dataset2

ca2: inputs: dataset3, dataset2 output: dataset4

when multiple inputs specified, first input dataset used copying data other datasets used dependencies. ca2 start executing when following conditions met:

  • ca2 has completed , dataset2 available. dataset not used when copying data dataset4. acts scheduling dependency ca2.
  • dataset3 available. dataset represents data copied destination.

the datasets in pipeline have shared use different datasets in 2 activities , not seem share dependency. hence scheduled in parallel.

thanks, harish




Microsoft Azure  >  Azure Data Factory



Comments

Post a Comment

Popular posts from this blog

Azure DocumentDB Owner resource does not exist

having same error microsoft.azure.documents.documentclientexception: message: {"errors":["owner resource not exist"]} , scenario. when deployed webapp azure , try document docdb throws error. docdb exists in azure , contains document looking for. weird local machine(running vs) works fine. i'm using same settings in azure , local. have , idea this. thanks Microsoft Azure  >  Azure Cosmos DB
Read more

job syspolicy_purge_history job fail in sqlserver 2008

Image
hello guys, when installed sqlserver,it automatically created 1 job name job syspolicy_purge_history. but failed @ step 1. the reason is: date                11/6/2010 2:00:00 am log                job history (syspolicy_purge_history) step id                1 server                mxosv-otsdb01 job name                syspolicy_purge_history step name                verify automation enabled. duration                00:00:00 sql severity                16 sql message id                34022 operator emailed                operator net sent                operator paged                retries attempted                0 message executed user: nt authority\system. policy automation turned off. [sqlstate 42000] (error 34022).  the step failed. have run statement: select msdb.dbo.fn_syspolicy_is_automation_enabled the returned valus 0. what can resolve it? or can delete job manually.   thanks in advance. if haven't things want,be
Read more

Trying to register with public marketplace error with 'Get-AzureStackStampInformation'

when trying register against public azure adfs azure stack error below. logs can found at: c:\maslogs\azurestack.azureregistration.2017-10-12.01-31-23.log  and  \\azs-ercs01\c$\maslogs error occurred checking stamp information: system.management.automation.remoteexception: term 'get-azurestackstampinformation' not recognized name of cmdlet, function, script file, or operable program. check spelling of name, or if path included, verify path correct , try again. @ c:\azurestack-tools-master\registration\registerwithazure.psm1:1106 char:5 +     throw "logs can found at: $global:azureregistrationlog  and  \ ... +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~     + categoryinfo          : operationstopped: (logs can fou... , try again.:string) [], runtimeexception     + fullyqualifiederrorid : logs can found at: c:\maslogs\azurestack.azureregistration.2017-10-12.01-31-23.log  a    nd  \\azs-ercs01\c$\maslogs error occurred chec
Read more