help : raw table in hive

[kasi]

Title keka....kafka output to table

 

Actually question emo raw hiVe table to inko table

[mettastar]

6 minutes ago, kasi said:

Kaka antha avasaram kuda ledu....Schema create chesetappudu.....Adi stored as json antea chalu

 

Adi aa Schema ki folder create chesthundhi..... Just aa folder lo json file save chesthea...... Hive table ga read cheyochu

Yes but stored as json not working in all hive versions.. 

[kasi]

5 minutes ago, mettastar said:

Yes but stored as json not working in all hive versions.. 

CREATE TABLE my_table (field1 string, field2 int, field3 string, field4 double) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.JsonSerde' ;

 

 

we use parquet..... So json conf 100% thelidhu..... But should beer very similar as above

 

[mettastar]

23 minutes ago, kasi said:

CREATE TABLE my_table (field1 string, field2 int, field3 string, field4 double) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.JsonSerde' ;

 

 

we use parquet..... So json conf 100% thelidhu..... But should beer very similar as above

 

Yes, but in my case this did not work so i had to go for lateral view.. some of our old clusters are running hive 1.xx

[BGBOSS]

Thornydev blogspot ani undhi....very informative

If u have multi nested Json with arrays.and want to explode it into muliple rows..then u might need to look for positional explode(posexplode) combined with lateral view and get_json_object 

Use regex to replace [ 

 

[vendettaa]

1 hour ago, kasi said:

Title keka....kafka output to table

 

Actually question emo raw hiVe table to inko table

Kafka queue output ni raw ne antaru ,adi  ela untado teliynapdu ilage nilo nuvvu navvukuntav 

 

[vendettaa]

1 hour ago, mettastar said:

Yes, but in my case this did not work so i had to go for lateral view.. some of our old clusters are running hive 1.xx

Akda uncle Ki telsinde prapancham

 

[vendettaa]

2 hours ago, kasi said:

😂😂😂😂😂😂

Akka hive lo primary key lu unav.....

Btw asalu nuvvu em cheyali anukuntunav???

 

Naku nuvu em chepputhunavo 1% kuda ardham kala

Arey uncLe primary key generate chesi vere rdbms ki toyali which is a consumption Layer

kafka data which is raw (Json)landed into hadoop danni table chesi  prati record Ki unique ID undali which is going to be primary key  in consumption layer

 

[vendettaa]

5 hours ago, mettastar said:

https://stackoverflow.com/questions/11479247/how-do-you-make-a-hive-table-out-of-json-data

Read one with 25 votes.. i used similar in one of projects .. thats the easiest way i found .. no need to use any serdes 

Thanks anniyya

[fake_Bezawada]

8 hours ago, vendettaa said:

i have a raw table which contains json objects as records (kafka output)

and i have to convert it to table in hive ,

need to generate primary key for each record and this data is exported to rdbms

 

How to do that ? 

Help please

 

ah json ne avro files antaru 

hive lo external table create chesi structure itchey and serde emo avro ivvu

ee avro files ni  hdfs lo padey 

automatic ga data hive loki pothadi

[kasi]

7 hours ago, vendettaa said:

Kafka queue output ni raw ne antaru ,adi  ela untado teliynapdu ilage nilo nuvvu navvukuntav 

 

[kasi]

sarlea nee yedupu naku endhuku gani 

primary key column ni okati create chesi, table create cheseattapudu....RDBMS primary keys ki conatct chesi aa kotha column lo pettu 

 

and FYI google ani okati untundhi ........just chepthuna 

[mettastar]

6 hours ago, fake_Bezawada said:

ah json ne avro files antaru 

hive lo external table create chesi structure itchey and serde emo avro ivvu

ee avro files ni  hdfs lo padey 

automatic ga data hive loki pothadi

Avro veru json veru

[kasi]

3 hours ago, mettastar said:

Avro veru json veru

adhea kada 

[vendettaa]

On 2/27/2018 at 3:53 AM, fake_Bezawada said:

ah json ne avro files antaru 

hive lo external table create chesi structure itchey and serde emo avro ivvu

ee avro files ni  hdfs lo padey 

automatic ga data hive loki pothadi

Json veru avrò veru