80 GB files xml to CSV how to convert

[bleachreddy]

4 hours ago, phoenix_nebula said:

Option 1: use sax parser and streaming in java

Option 2: Apache tika 

Extension to Option 1:

Non-blocking I/O is there in Java8+, that comes handy.

Also use streams provided by Java8+, that are very good. Not having enough time now, but I remember I did this for at least a 20GB XML file.

 

[pencil]

80 gb xml file aa.. irs database lo kuda antha data undademoga 

[Ellen]

Use element tree, then pandas in python

[kevinUsa]

I used excel , 

IT took me around 12 hrs to do it

73 GB of  data is now 12 gb data. 

 

[kevinUsa]

2 hours ago, pencil said:

80 gb xml file aa.. irs database lo kuda antha data undademoga 

https://ia800107.us.archive.org/27/items/stackexchange/stackoverflow.com-Posts.7z

https://ia800107.us.archive.org/27/items/stackexchange/stackoverflow.com-Posts.7z

used  this data 

[kevinUsa]

well  I used python to convert xml to csv.

 than I used  formed a hd insights cluster ,

 converted the data from .csv to .gz than uploaded to azure blobs

than used pig to clean all the null values than  used Hcaster to moved the data from pig to  hive than permformed the queries