Sqoop Import

[BGBOSS]

question to Hadoop Developers

What will  happen if there is a sqoop import failure on large data transfer from RDBMS to HDFS

will it persist some of the data onto HDFS before the the job fail occured

ee question interview lo adigadu ..I am not able to find the right answer in other forums

[Mod23isBhai]

15 minutes ago, BGBOSS said:

question to Hadoop Developers

What will  happen if there is a sqoop import failure on large data transfer from RDBMS to HDFS

will it persist some of the data onto HDFS before the the job fail occured

ee question interview lo adigadu ..I am not able to find the right answer in other forums

It depends on failure step

there are different steps of failure like for e.g. If it's fails before database connection it won't persist any data, other failure is after Database connection (. This has many failures steps) where data can persist but it all depends!!

[BGBOSS]

26 minutes ago, Mod23isBhai said:

It depends on failure step

there are different steps of failure like for e.g. If it's fails before database connection it won't persist any data, other failure is after Database connection (. This has many failures steps) where data can persist but it all depends!!

lets assume that this is related to a failure other than database connection...so in this case partial data persist avtundha HDFS lo ..

I know if the similar situation happens during export then partial data will be loaded into RDBMS and the workaround for this issue is to add a staging table...

can we go for similar kind of approach for imports as well

[vendetta]

12 minutes ago, BGBOSS said:

lets assume that this is related to a failure other than database connection...so in this case partial data persist avtundha HDFS lo ..

I know if the similar situation happens during export then partial data will be loaded into RDBMS and the workaround for this issue is to add a staging table...

can we go for similar kind of approach for imports as well

There is getting of partial data into hdfs , depending on  number of mappers failed.

[Bhai]

10 minutes ago, vendetta said:

There is getting of partial data into hdfs , depending on  number of mappers failed.

[Allarinaresh10]

1 hour ago, BGBOSS said:

question to Hadoop Developers

What will  happen if there is a sqoop import failure on large data transfer from RDBMS to HDFS

will it persist some of the data onto HDFS before the the job fail occured

ee question interview lo adigadu ..I am not able to find the right answer in other forums

Partial data will be in HDFS ..

Sqoop job untadi... where you can schedule this kind of multiple IMPORTS ... 

 

[BGBOSS]

19 minutes ago, vendetta said:

There is getting of partial data into hdfs , depending on  number of mappers failed.

Oh Ok...so if we are using 4 mappers and job failed after 2 mappers completing the task, we will have partial data in HDFS

so in this case if we want to initiate another instance of the same import, we need to clear the data in HDFS and restart the process?

[BGBOSS]

8 minutes ago, Allarinaresh10 said:

Partial data will be in HDFS ..

Sqoop job untadi... where you can schedule this kind of multiple IMPORTS ... 

 

thank you....yeah ofcourse we have sqoop jobs but my question is how to handle the partial data ...because if I initiate the import again,then it might result in duplicate records....sry if my question doesnt make proper sense...i am new to Hadoop ,trying to learn

[vendetta]

24 minutes ago, BGBOSS said:

thank you....yeah ofcourse we have sqoop jobs but my question is how to handle the partial data ...because if I initiate the import again,then it might result in duplicate records....sry if my question doesnt make proper sense...i am new to Hadoop ,trying to learn

yup duplicate records ani chupistundi , so specify only tables that you want to import , leda malli anni tables import cheyali ante  hdfs lo unna tables remove cheyalsinde lekapote files already exist ani error chupistune untundi

[vendetta]

27 minutes ago, BGBOSS said:

Oh Ok...so if we are using 4 mappers and job failed after 2 mappers completing the task, we will have partial data in HDFS

so in this case if we want to initiate another instance of the same import, we need to clear the data in HDFS and restart the process?

if we want to import all the tables , yup clear the data in hdfs,  or else specify particular tables do imports  , naku telsinantavarku inte  , alternative solution evaraina chepte good telsukunta

[Mod23isBhai]

1 minute ago, vendetta said:

if we want to import all the tables , yup clear the data in hdfs,  or else specify particular tables do imports  , naku telsinantavarku inte  , alternative solution evaraina chepte good telsukunta

 how can you clear all data ? Data anta delete chestha inka 10g antadu client erri fafa 

[vendetta]

2 minutes ago, Mod23isBhai said:

 how can you clear all data ? Data anta delete chestha inka 10g antadu client erri fafa 

cheyav uncle akda inko instance of sqoop import intiate cheyalante cheppa, inkem chestam

ledante enta data hdfs lo undo chuskuni  remaining import cheskodame 

[BGBOSS]

5 minutes ago, vendetta said:

cheyav uncle akda inko instance of sqoop import intiate cheyalante cheppa, inkem chestam

ledante enta data hdfs lo undo chuskuni  remaining import cheskodame 

I don't think this approach is recommended...Instead if we can use a staging table on sqoop import, we can achieve all or nothing load.. since I don't have any real time experience,I am not sure if this is can be done or not

[vendetta]

9 minutes ago, BGBOSS said:

I don't think this approach is recommended...Instead if we can use a staging table on sqoop import, we can achieve all or nothing load.. since I don't have any real time experience,I am not sure if this is can be done or not

staging tables ela ekda use chestam 

docs links emaina unte post it here

[Bhai]

32 minutes ago, Mod23isBhai said:

 how can you clear all data ? Data anta delete chestha inka 10g antadu client erri fafa