ETL interview questions

[Sarvapindi]

Basic... how do u find duplicate records

[vendettaa]

1 minute ago, mantis said:

ETL performance tuning ante....views,indexes create cheyadam laantivi untaaya

indexing undi  bucketing undi partitioning undi 

alage tez over mapreduce

spark job performance tuning ante , depends on volume of data and memory size , u configure  no of executors to be used , executor memory 

atlage depends on volume of data , persisting data inmemory or memory onto disk ala untay 

alage inka efficiently ante using broadcast variables, accumulators, broad cast joinsa

alage hive lo better to save data in orc file formats

e file formats concept untadi adoka vast 

[vendettaa]

1 minute ago, Sarvapindi said:

Basic... how do u find duplicate records

how to eliminate duplicate records

val streamingDf = spark.readStream. ...  // columns: guid, eventTime, ...

// Without watermark using guid column
streamingDf.dropDuplicates("guid")

[tables]

1 hour ago, ARYA said:

arey e lowda lo etl ni lite teskondi...

baa rude ga unnaru kada meeru.

[ARYA]

3 minutes ago, tables said:

baa rude ga unnaru kada meeru.

yes, malli govt form cheyyabothunnam aa matram G balupu undali brother..

[Sarvapindi]

Iga idiseyindri va ee post...

[bulleye]

connected or unconnected transformations..performance wise.

 

[vendettaa]

27 minutes ago, bulleye said:

connected or unconnected transformations..performance wise.

 

groupbykey vs reducebykey

java serializer vs kryo serializer

[Naaperushiva]

modeling questions kooda aduguthaaru...paina boys cheppinattu exception handling and performance tuning gurinchi kooda aduguthaaru..inka migilinavi nee tool related questions vuntaayi