calling all jjjjjjaaava and eewwwebservice experts

[fasak_vachadu]

I need to know is there any way I can check 100k urls in one shot ... rite now i am picking up urls from local .txt /excel/csv files and seding one by one to my method to check response code 200.

Also I need to get the response time for each one?? 

 

 

HttpURLConnection httpURLConnect=(HttpURLConnection)url.openConnection();

if(httpURLConnect.getResponseCode()==200){

syso(url and response message)

}

else{

syso( urls + response code

}

[fasak_vachadu]

evaru lera??

[Katara]

Why check once? One by one valla loss enti? You can check all once by spinng new thread for each

[fasak_vachadu]

43 minutes ago, Katara said:

Why check once? One by one valla loss enti? You can check all once by spinng new thread for each

time consuming .... taking almost a day to get results... 

[sri_india]

Threadpool /executor service

[Rushabhi]

2 hours ago, Katara said:

Why check once? One by one valla loss enti? You can check all once by spinng new thread for each

200 threads ki resources waste kadha. What if they increase? threadpools are the way to go.

[Katara]

22 minutes ago, Rushabhi said:

200 threads ki resources waste kadha. What if they increase? threadpools are the way to go.

He wants 200 parallel threads.

[fasak_vachadu]

Just now, Katara said:

He wants 200 parallel threads.

no no response code 200 ok is status code ..... 100k pina unnai chala time consuming at least minimize my time 

 

[Rushabhi]

28 minutes ago, fasak_vachadu said:

no no response code 200 ok is status code ..... 100k pina unnai chala time consuming at least minimize my time 

 

you got the answer man. use parallelism

[Rushabhi]

See this

https://samsaffron.com/archive/2012/06/07/testing-3-million-hyperlinks-lessons-learned

[karthikn]

4 hours ago, fasak_vachadu said:

I need to know is there any way I can check 100k urls in one shot ... rite now i am picking up urls from local .txt /excel/csv files and seding one by one to my method to check response code 200.

Also I need to get the response time for each one?? 

 

 

HttpURLConnection httpURLConnect=(HttpURLConnection)url.openConnection();

if(httpURLConnect.getResponseCode()==200){

syso(url and response message)

}

else{

syso( urls + response code

}

. Net lo Parallel.For untadhi and you can specify max parallelism.. Ee Kalam lo one by one chdck cheyadam and job takes 1 day Ante #cringe

[karthikn]

 

Nee urls anni you can process by page size let's say 5k or 10k. Check those parallely and go to next. SQL aithe staging table undedhi. You can update status there. Anni records kalipi SQL user defined table tho update all records records status at once

[Katara]

38 minutes ago, fasak_vachadu said:

no no response code 200 ok is status code ..... 100k pina unnai chala time consuming at least minimize my time 

 

Ok is that code a stand alone class or running on tomcat? If its stand alone , spinoff 100k threads in 1 line of code , should be fine

see this exact example you need

https://www.tutorialspoint.com/apache_httpclient/apache_httpclient_multiple_threads.htm

public class ClientMultiThreaded extends Thread {
   CloseableHttpClient httpClient;
   HttpGet httpget;
   int id;
 
   public ClientMultiThreaded(CloseableHttpClient httpClient, HttpGet httpget,
   int id) {
      this.httpClient = httpClient;
      this.httpget = httpget;
      this.id = id;
   }
   @Override
   public void run() {
      try{
         //Executing the request
         CloseableHttpResponse httpresponse = httpClient.execute(httpget);

         //Displaying the status of the request.
         System.out.println("status of thread "+id+":"+httpresponse.getStatusLine());

         //Retrieving the HttpEntity and displaying the no.of bytes read
         HttpEntity entity = httpresponse.getEntity();
         if (entity != null) {
            System.out.println("Bytes read by thread thread "+id+":
               "+EntityUtils.toByteArray(entity).length);
         }
      }catch(Exception e) {
         System.out.println(e.getMessage());
      }
   }

[Rushabhi]

7 minutes ago, Katara said:

Ok is that code a stand alone class or running on tomcat? If its stand alone , spinoff 100k threads in 1 line of code , should be fine

see this exact example you need

https://www.tutorialspoint.com/apache_httpclient/apache_httpclient_multiple_threads.htm

public class ClientMultiThreaded extends Thread {
   CloseableHttpClient httpClient;
   HttpGet httpget;
   int id;
 
   public ClientMultiThreaded(CloseableHttpClient httpClient, HttpGet httpget,
   int id) {
      this.httpClient = httpClient;
      this.httpget = httpget;
      this.id = id;
   }
   @Override
   public void run() {
      try{
         //Executing the request
         CloseableHttpResponse httpresponse = httpClient.execute(httpget);

         //Displaying the status of the request.
         System.out.println("status of thread "+id+":"+httpresponse.getStatusLine());

         //Retrieving the HttpEntity and displaying the no.of bytes read
         HttpEntity entity = httpresponse.getEntity();
         if (entity != null) {
            System.out.println("Bytes read by thread thread "+id+":
               "+EntityUtils.toByteArray(entity).length);
         }
      }catch(Exception e) {
         System.out.println(e.getMessage());
      }
   }

100k threads ante RAM space kooda choodali ga. Will quickly go to OOM errors

[fasak_vachadu]

44 minutes ago, Katara said:

Ok is that code a stand alone class or running on tomcat? If its stand alone , spinoff 100k threads in 1 line of code , should be fine

see this exact example you need

https://www.tutorialspoint.com/apache_httpclient/apache_httpclient_multiple_threads.htm

public class ClientMultiThreaded extends Thread {
   CloseableHttpClient httpClient;
   HttpGet httpget;
   int id;
 
   public ClientMultiThreaded(CloseableHttpClient httpClient, HttpGet httpget,
   int id) {
      this.httpClient = httpClient;
      this.httpget = httpget;
      this.id = id;
   }
   @Override
   public void run() {
      try{
         //Executing the request
         CloseableHttpResponse httpresponse = httpClient.execute(httpget);

         //Displaying the status of the request.
         System.out.println("status of thread "+id+":"+httpresponse.getStatusLine());

         //Retrieving the HttpEntity and displaying the no.of bytes read
         HttpEntity entity = httpresponse.getEntity();
         if (entity != null) {
            System.out.println("Bytes read by thread thread "+id+":
               "+EntityUtils.toByteArray(entity).length);
         }
      }catch(Exception e) {
         System.out.println(e.getMessage());
      }
   }

standalone bhaya..... running in my local .... will check this code and let you know