ADF Dataset Validation Validation Issues select count(1) from TABLENAME sql query / that is a problem


hi

adf v1

azure sql external dataset

{    "$schema": "http://datafactories.schema.management.azure.com/schemas/2015-09-01/microsoft.datafactory.table.json",    "name": "somename",    "properties": {      "type": "azuresqltable",      "linkedservicename": "azuresqldatabase",      "structure": [],      "typeproperties": {        "tablename": "schema.table"      },      "availability": {        "frequency": "day",        "interval": 1      },      "policy": {        //"validation": {        //  "minimumrows": 100        //}      },      "external": true    }  }

i thought validating rows on external sql dataset. defined validation rule. rows.

but instead of executing validation while copy (i use query source sqlreaderquery) seams "select count(1) schema.table" executed against database.

i have 2 issues that.

 - @ copy level not rows must read table - wrong

 - , more imporantly plain count(1) on rows might slow. in case fails timeout or fails overflow exception because table has more 2^31 rows.

does on of have recommendation me how validation @ copy level?

is functionality better in adf v2 ?

thanks input


http://www.hmayer.net/



Microsoft Azure  >  Azure Data Factory



Comments

Popular posts from this blog

Azure DocumentDB Owner resource does not exist

BizTalk Server 2013 Azure VM Log Shipping and HA for hosts

How to send non-standard Content-Type header ?