Remote Data Transfer between multiple Data lakes

[Ruskey]

Context : As part of a big data project, we have multiples data lakes (one by region): America, Asia and Europe. And we need to transfer data between them. Our data lakes are built around Hortonworks.

Constraint: Limitations in the WAN network because it is also used to transfer
Higher priority data (our data are a little less).

Volumetry: tens of gigs of data per day.

Need: What are the bests practices to garnish this transfer of data between the plates?
Do you have returns exp. For this kind of
need.

All documentation, contact, link, information will be welcome.

Thank you for your help

 

 

@Spartan @k2s @yomama @tennisluvr @loveindia Help!

[Katravelli]

there are several crawl tools that you can use for data transfer... we are using ucm 

[Ruskey]

Just now, Katravelli said:

there are several crawl tools that you can use for data transfer... we are using ucm 

Can you please elaborate 

[Katravelli]

Let me know if this helps 

 

https://www.hcltech.com/brochures/enterprise-content-management-portals/universal-content-migrator-service-offering

[former]

Hortonworks don't provide any tools for your use case ??

[Ruskey]

16 minutes ago, Katravelli said:

Let me know if this helps 

 

https://www.hcltech.com/brochures/enterprise-content-management-portals/universal-content-migrator-service-offering

Thx man will go through it 

[nako_job_kavali]

Try alteryx it is built for same purpose

[k2s]

2 hours ago, Ruskey said:

Context : As part of a big data project, we have multiples data lakes (one by region): America, Asia and Europe. And we need to transfer data between them. Our data lakes are built around Hortonworks.

Constraint: Limitations in the WAN network because it is also used to transfer
Higher priority data (our data are a little less).

Volumetry: tens of gigs of data per day.

Need: What are the bests practices to garnish this transfer of data between the plates?
Do you have returns exp. For this kind of
need.

All documentation, contact, link, information will be welcome.

Thank you for your help

 

 

@Spartan @k2s @yomama @tennisluvr @loveindia Help!

Azcopy is there no.. 

within cloud to cloud transfer is pretty fast..

[Ruskey]

Just now, k2s said:

Azcopy is there no.. 

within cloud to cloud transfer is pretty fast..

Thx let me research on that 

[k2s]

25 minutes ago, Ruskey said:

Thx let me research on that 

Azure data lake factory tools are exclusively for data copying . 

[k2s]

On 1/26/2017 at 7:51 AM, Ruskey said:

Context : As part of a big data project, we have multiples data lakes (one by region): America, Asia and Europe. And we need to transfer data between them. Our data lakes are built around Hortonworks.

Constraint: Limitations in the WAN network because it is also used to transfer
Higher priority data (our data are a little less).

Volumetry: tens of gigs of data per day.

Need: What are the bests practices to garnish this transfer of data between the plates?
Do you have returns exp. For this kind of
need.

All documentation, contact, link, information will be welcome.

Thank you for your help

 

 

@Spartan @k2s @yomama @tennisluvr @loveindia Help!

shame on you man.. question kuda original kadu kada.. LinkedIN lo Hadoop Users lo Ahmed Cheriat 8 days back esina question idi.. as it is copy paste chesi ikkada esinav... question anna ardam ayinda vay neeku aada ? edo help help ani sfamming posts esinav  

[k2s]

On 1/26/2017 at 10:41 AM, Ruskey said:

Thx let me research on that 

 panimalina pake pakodi 

[loveindia]

one of my friend did the same man... 

[k2s]

2 minutes ago, loveindia said:

one of my friend did the same man... 

like this a 

[loveindia]

1 minute ago, k2s said:

like this a 

aadu andariki cheppey answer adey andukey adey cheppa nenu kuda daddy man... diaper trash cheyakuna eeda em chestunnav...