SQL SSIS, insert only new rows.

Friday, June 20, 2008 1:14 PM | Leave a reply »

SSIS workflow is really amazing when you have to add some automatism in your data flow processes. But sometimes is better to make same checks before start with a really dangerous BULK operation.

One of the most noise stuff I have to do is to launch insert of new records when I receive updates from web (web services, XML data). But when I do this I'm always afraid about duplicated rows.

Ho to check if a row exist, in SSIS?

SSIS Package contains a component called lookup. Very intuitive name. So if you need to insert only some new rows in a table you can build a workflow like this:

image

So we have three components:

  1. A OLEDB Datasource that reads data from the source table
  2. A LookUp component that check every row, if exists in the destination table. If doesn't it redirects the rows in the error event.
  3. A Destination SQL that receive only the new rows, the rows that are in error in the lookup component.

Easy and automatic.


Comments

  1. Gravatar OT says:

    Re : # re: SQL SSIS, insert only new rows.

    is this procedure fast? if i have around 500,000 rows in the destination table, will this workflow take more than a minute?

    hope to hear from you. thanks
  2. Gravatar raffaeu says:

    Re : # re: SQL SSIS, insert only new rows.

    Oh yeah this technique is awesome and really fast.
    Don't worry I process milion of rows without performance problems.
    :-D
  3. Gravatar DG says:

    Re : # re: SQL SSIS, insert only new rows.

    Could you explain how to properly use this setup?

    I have a flat file that I need to import and I want to insert the new rows only
    10x
  4. Gravatar Raffaeu says:

    Re : # re: SQL SSIS, insert only new rows.

    When you run the package, it will put in the error section all the rows not finded in the destination table. At that point, you should implement your technique to read and process those rows.
    Thank you.
  5. Gravatar apvzeynp says:

    Re : # apvzeynp

    apvzeynp
  6. Re : # levofloxacin

    levofloxacin
  7. Gravatar fbcqorms says:

    Re : # fbcqorms

    fbcqorms
  8. Gravatar tcswosio says:

    Re : # tcswosio

    tcswosio
  9. Gravatar rnhqdzik says:

    Re : # rnhqdzik

    rnhqdzik
  10. Gravatar wgasmixn says:

    Re : # wgasmixn

    wgasmixn
  11. Gravatar zgucaqxm says:

    Re : # zgucaqxm

    zgucaqxm
  12. Gravatar rtzorpxu says:

    Re : # rtzorpxu

    rtzorpxu
  13. Gravatar nxscrvwl says:

    Re : # nxscrvwl

    nxscrvwl
  14. Gravatar nhfyfgyk says:

    Re : # nhfyfgyk

    nhfyfgyk
  15. Gravatar tegevcml says:

    Re : # tegevcml

    tegevcml
  16. Gravatar cozaar says:

    Re : # cozaar

    cozaar
  17. Re : # ambien online

    ambien online
  18. Gravatar uuirypcy says:

    Re : # uuirypcy

    uuirypcy
  19. Gravatar gwjnosgc says:

    Re : # gwjnosgc

    gwjnosgc
  20. Gravatar adwwbzub says:

    Re : # adwwbzub

    adwwbzub
  21. Gravatar lzbxcllx says:

    Re : # lzbxcllx

    lzbxcllx
  22. Gravatar jgnhrmnp says:

    Re : # jgnhrmnp

    jgnhrmnp
  23. Re : # generic prevacid

    generic prevacid
  24. Gravatar amoxil says:

    Re : # amoxil

    amoxil
  25. Gravatar buy zoloft says:

    Re : # buy zoloft

    buy zoloft
  26. Gravatar allegra says:

    Re : # allegra

    allegra
  27. Re : # order vicodin

    order vicodin
  28. Re : # generic finasteride

    generic finasteride
  29. Re : # venlafaxine

    venlafaxine
  30. Gravatar uslebrww says:

    Re : # uslebrww

    uslebrww
  31. Gravatar buy ultram says:

    Re : # buy ultram

    buy ultram
  32. Re : # generic vicodin

    generic vicodin
  33. Re : # purchase viagra

    purchase viagra
  34. Re : # buy xenical

    buy xenical
  35. Gravatar ultracet says:

    Re : # ultracet

    ultracet
  36. Gravatar gfoycffz says:

    Re : # gfoycffz

    gfoycffz
  37. Re : # buy soma online

    buy soma online
  38. Gravatar darvon says:

    Re : # darvon

    darvon
  39. Gravatar buy nexium says:

    Re : # buy nexium

    buy nexium
  40. Re : # generic propecia

    generic propecia
  41. Re : # buy viagra online

    buy viagra online
  42. Gravatar celecoxib says:

    Re : # celecoxib

    celecoxib
  43. Gravatar levaquin says:

    Re : # levaquin

    levaquin
  44. Gravatar fioricet says:

    Re : # fioricet

    fioricet
  45. Gravatar lipitor says:

    Re : # lipitor

    lipitor
  46. Re : # order cialis

    order cialis
  47. Re : # buy adipex online

    buy adipex online
  48. Re : # cheap meridia

    cheap meridia
  49. Re : # order tramadol

    order tramadol
  50. Gravatar tenormin says:

    Re : # tenormin

    tenormin
  51. Re : # valium online

    valium online
  52. Gravatar zopiclone says:

    Re : # zopiclone

    zopiclone
  53. Re : # xenical online

    xenical online
  54. Re : # clopidogrel

    clopidogrel
  55. Re : # generic zocor

    generic zocor
  56. Gravatar cetirizine says:

    Re : # cetirizine

    cetirizine
  57. Gravatar sertraline says:

    Re : # sertraline

    sertraline
  58. Gravatar motrin says:

    Re : # motrin

    motrin
  59. Gravatar stilnox says:

    Re : # stilnox

    stilnox
  60. Re : # vicodin online

    vicodin online
  61. Gravatar citalopram says:

    Re : # citalopram

    citalopram
  62. Gravatar neurontin says:

    Re : # neurontin

    neurontin
  63. Gravatar pjftkugs says:

    Re : # pjftkugs

    pjftkugs
  64. Gravatar jkldwrum says:

    Re : # jkldwrum

    jkldwrum
  65. Gravatar norvasc says:

    Re : # norvasc

    norvasc
  66. Re : # order valium online

    order valium online
  67. Gravatar keflex says:

    Re : # keflex

    keflex
  68. Re : # buy carisoprodol online

    buy carisoprodol online
  69. Gravatar retin says:

    Re : # retin

    retin
  70. Gravatar aousejmm says:

    Re : # aousejmm

    aousejmm
  71. Gravatar bwwtfqko says:

    Re : # bwwtfqko

    bwwtfqko
  72. Gravatar ilcwovmz says:

    Re : # ilcwovmz

    ilcwovmz
  73. Gravatar prozac says:

    Re : # prozac

    prozac
  74. Gravatar propecia says:

    Re : # propecia

    propecia
  75. Gravatar bcynxxmm says:

    Re : # bcynxxmm

    bcynxxmm
  76. Gravatar ywpwmxgj says:

    Re : # ywpwmxgj

    ywpwmxgj
  77. Gravatar bamskukb says:

    Re : # bamskukb

    bamskukb
  78. Gravatar bhowjpzu says:

    Re : # bhowjpzu

    bhowjpzu
  79. Gravatar qkgklvan says:

    Re : # qkgklvan

    qkgklvan
  80. Gravatar skfrteix says:

    Re : # skfrteix

    skfrteix
  81. Re : # buy amoxicillin

    buy amoxicillin
  82. Gravatar naproxen says:

    Re : # naproxen

    naproxen
  83. Re : # propecia online

    propecia online
  84. Gravatar losartan says:

    Re : # losartan

    losartan
  85. Re : # generic nexium

    generic nexium
  86. Re : # diazepam online

    diazepam online
  87. Gravatar rfwmhnkc says:

    Re : # rfwmhnkc

    rfwmhnkc
  88. Gravatar Buy Viagra says:

    Re : # re: SQL SSIS, insert only new rows.

    great post thanks for sharing
  89. Re : # re: SQL SSIS, insert only new rows.

    I am new on integration services Please can you explain me in detail i.e. step by step that how to use Lookup Transform for inserting only new records , my scenario is that i am have a table with 5000000 Records in Destination DB and i just get a records from source and check are these records already existed in Destination Table or Not if No then i will transfer these records as a new records otherwise do nothing but problem is that i already have a primary key identity column in destination table so please tell me how can i achieve this through lookup transform ,Thanks
Comments have been closed on this topic.