Machine-learning clusters in Azure hijacked to mine cryptocurrency

[Spartan]

@k2s  @psycopk

Attackers recently hijacked powerful machine-learning clusters inside Microsoft’s Azure cloud-computing service so that they could mine cryptocurrency at the expense of the customers who rented them, the company said Wednesday.

The nodes, which were misconfigured by customers, made the perfect target for so-called cryptojacking schemes. Machine-learning tasks typically require vast amounts of computing resources. By redirecting them to perform the compute-intensive workloads required to mine digital coins, the attackers found a means to generate large amounts of currency at little or no cost.

The infected clusters were running Kubeflow, an open source framework for machine-learning applications in Kubernetes, which is itself an open source platform for deploying scalable applications across large numbers of computers. Microsoft said compromised clusters it discovered numbered in the “tens.” Many of them ran an image available from a public repository, ostensibly to save users the hassle of creating one themselves. Upon further inspection, Microsoft investigators discovered it contained code that surreptitiously mined the Monero cryptocurrency.

How was it done?

After finding the infected clusters, investigators turned their attention to how the machines were compromised. For security, the dashboard that allows administrators to control Kubeflow is, by default, accessible only through istio ingress, a gateway that’s typically located at the edge of the cluster network. The default setting prevents people on the Internet at large from accessing the dashboard and making unauthorized changes to the cluster.

In a post published Wednesday, Yossi Weizman, a security-research software engineer in the Azure Security Center, said that some users change the setting.

We believe that some users chose to do it for convenience,” Weizman wrote. “Without this action, accessing the dashboard requires tunneling through the Kubernetes API server and isn’t direct. By exposing the Service to the Internet, users can access the dashboard directly. However, this operation enables insecure access to the Kubeflow dashboard, which allows anyone to perform operations in Kubeflow, including deploying new containers in the cluster.”

Once attackers have access to the dashboard, they have multiple options for deploying backdoored containers in the cluster. For instance, attackers can create what’s known as a Jupyter Notebook server that runs on the cluster. They can then place a malicious image inside of the Jupyter Notebook. If a Jupyter Notebook is already installed, it can be maliciously modified.

Wednesday’s post provides several ways that users can check if their cluster has been compromised.

“Azure Security Center has detected multiple campaigns against Kubernetes clusters in the past that have a similar access vector: an exposed service to the Internet,” Weizman wrote. “However, this is the first time that we have identified an attack that targets Kubeflow environments specifically.”

[k2s]

4 minutes ago, Spartan said:

@k2s  @psycopk

Attackers recently hijacked powerful machine-learning clusters inside Microsoft’s Azure cloud-computing service so that they could mine cryptocurrency at the expense of the customers who rented them, the company said Wednesday.

The nodes, which were misconfigured by customers, made the perfect target for so-called cryptojacking schemes. Machine-learning tasks typically require vast amounts of computing resources. By redirecting them to perform the compute-intensive workloads required to mine digital coins, the attackers found a means to generate large amounts of currency at little or no cost.

The infected clusters were running Kubeflow, an open source framework for machine-learning applications in Kubernetes, which is itself an open source platform for deploying scalable applications across large numbers of computers. Microsoft said compromised clusters it discovered numbered in the “tens.” Many of them ran an image available from a public repository, ostensibly to save users the hassle of creating one themselves. Upon further inspection, Microsoft investigators discovered it contained code that surreptitiously mined the Monero cryptocurrency.

How was it done?

After finding the infected clusters, investigators turned their attention to how the machines were compromised. For security, the dashboard that allows administrators to control Kubeflow is, by default, accessible only through istio ingress, a gateway that’s typically located at the edge of the cluster network. The default setting prevents people on the Internet at large from accessing the dashboard and making unauthorized changes to the cluster.

In a post published Wednesday, Yossi Weizman, a security-research software engineer in the Azure Security Center, said that some users change the setting.

We believe that some users chose to do it for convenience,” Weizman wrote. “Without this action, accessing the dashboard requires tunneling through the Kubernetes API server and isn’t direct. By exposing the Service to the Internet, users can access the dashboard directly. However, this operation enables insecure access to the Kubeflow dashboard, which allows anyone to perform operations in Kubeflow, including deploying new containers in the cluster.”

Once attackers have access to the dashboard, they have multiple options for deploying backdoored containers in the cluster. For instance, attackers can create what’s known as a Jupyter Notebook server that runs on the cluster. They can then place a malicious image inside of the Jupyter Notebook. If a Jupyter Notebook is already installed, it can be maliciously modified.

Wednesday’s post provides several ways that users can check if their cluster has been compromised.

“Azure Security Center has detected multiple campaigns against Kubernetes clusters in the past that have a similar access vector: an exposed service to the Internet,” Weizman wrote. “However, this is the first time that we have identified an attack that targets Kubeflow environments specifically.”

nice.. 

[Spartan]

Just now, k2s said:

nice.. 

why are u allowing default configs on clusters and dashboard access..

[Mr Mirchi]

12 minutes ago, Spartan said:

why are u allowing default configs on clusters and dashboard access..

Jai aws

[k2s]

1 hour ago, Spartan said:

why are u allowing default configs on clusters and dashboar

1 hour ago, Spartan said:

why are u allowing default configs on clusters and dashboard access..

Too lazy man

[quickgun_murugun]

3 hours ago, Spartan said:

why are u allowing default configs on clusters and dashboard access..

lol

[Doravaru]

3 hours ago, Spartan said:

why are u allowing default configs on clusters and dashboard access..

@Spartan bhayya 

Azure nersukoneeki oka proper direction iyyu ekkada start cheyyali ela start cheyyali ...nenu ippudu DataAdmin/ BI Dev ga chestunna ... naa background ki azure lo a module baguntadi ? 
 

thanks 

[Spartan]

9 hours ago, Doravaru said:

@Spartan bhayya 

Azure nersukoneeki oka proper direction iyyu ekkada start cheyyali ela start cheyyali ...nenu ippudu DataAdmin/ BI Dev ga chestunna ... naa background ki azure lo a module baguntadi ? 
 

thanks 

@k2s  can help

[Doravaru]

On 6/12/2020 at 10:28 AM, Spartan said:

@k2s  can help

@k2s bro ... need your insights on this 

On 6/12/2020 at 12:53 AM, Doravaru said:

@Spartan bhayya 

Azure nersukoneeki oka proper direction iyyu ekkada start cheyyali ela start cheyyali ...nenu ippudu DataAdmin/ BI Dev ga chestunna ... naa background ki azure lo a module baguntadi ? 
 

thanks 

 

[Michaelbarbosa]

15 minutes ago, Doravaru said:

@k2s bro ... need your insights on this 

 

go for azure SQL that's what they say lots of DBA positions ask for Azure SQL too. 
 

[k2s]

9 hours ago, Doravaru said:

@k2s bro ... need your insights on this 

 

 

9 hours ago, Michaelbarbosa said:

go for azure SQL that's what they say lots of DBA positions ask for Azure SQL too. 
 

anthey anthey ... 

[quickgun_murugun]

7 minutes ago, k2s said:

 

anthey anthey ... 

@k2s thatha nuvvu Azure architect vaa?

[k2s]

12 minutes ago, quickgun_murugun said:

@k2s thatha nuvvu Azure architect vaa?

emo man.. teludu... certification chesthey Architect antaru antey.. then am double principal architect anukovachu... 

real job lo architect antey. am not even a support engg 

[quickgun_murugun]

25 minutes ago, k2s said:

emo man.. teludu... certification chesthey Architect antaru antey.. then am double principal architect anukovachu... 

real job lo architect antey. am not even a support engg 

Edayithey endi full paisal ostunnay kada adi saal 

[Scada]

On 6/12/2020 at 1:53 AM, Doravaru said:

@Spartan bhayya 

Azure nersukoneeki oka proper direction iyyu ekkada start cheyyali ela start cheyyali ...nenu ippudu DataAdmin/ BI Dev ga chestunna ... naa background ki azure lo a module baguntadi ? 
 

thanks 

@Doravaru  see AZ 203/204