anybody stuck in Prod Issue - oncall support?

[cosmopolitan]

2 minutes ago, superhit3 said:

niku yem kavali scale up & down proportional to DB instances n pool gurinchi ...grafana, CPU utilization n gO Router perf ...nodes and services, their relocation etc...consumer count for topics n queues etc. inka bouncing stopping...response times per multiple foundations kondaru organization ani kuda antaru cloud matter ...abbo inka chalane vunnay......my turn comes every 3 weeks...appudu vucha naku. ante beginnig year lo naku yekam ga potty ye vachedi. now a lot better 

All rounder lekka unav ga bro 

[odhu_le_macha]

2 minutes ago, superhit3 said:

niku yem kavali scale up & down proportional to DB instances n pool gurinchi ...grafana, CPU utilization n gO Router perf ...nodes and services, their relocation etc...consumer count for topics n queues etc. inka bouncing stopping...response times per multiple foundations kondaru organization ani kuda antaru cloud matter ...abbo inka chalane vunnay......my turn comes every 3 weeks...appudu vucha naku. ante beginnig year lo naku yekam ga potty ye vachedi. now a lot better 

[JaiTDP]

2 minutes ago, superhit3 said:

prod server nu 3 days ....how? peddaga visibility and need leni app? not degrading bhayya ...just questioning how ani ante

Haha.. luckily it’s backend data loading server bro not the UI. The clients didn’t have visibility abt this but we notified saying there will be some delay in processing data. 

total server crashed , Microsoft vaadu kuda hands up annaadu after reviewing logs. Thella manager(chaala manchodu) pushing me on UAT stuff saying client was waiting where I am fighting with this prod server  !!  
 

Oka 1 month back up dhi vunte dhaanitho laaginchesaamu. 

[Jombo0108]

3 minutes ago, JaiTDP said:

Haha.. luckily it’s backend data loading server bro not the UI. The clients didn’t have visibility abt this but we notified saying there will be some delay in processing data. 

total server crashed , Microsoft vaadu kuda hands up annaadu after reviewing logs. Thella manager(chaala manchodu) pushing me on UAT stuff saying client was waiting where I am fighting with this prod server  !!  
 

 

 

 

 

 

 

 

 

 

 

 

 

 

 

Skip

 

 

Oka 1 month back up dhi vunte dhaanitho laaginchesaamu. 

@lokesh_rocks

[Jombo0108]

43 minutes ago, superhit3 said:

full time bhayya...that too mana DB lo kondaru hate chese QA nenu. dabbulu yentho ivvaru...anthe yentho lontha matrame......kani On-Call rotation doing. PCF clloud....instances...scaling...bouncing ...memory...grafana...cpu utiliztion...   appd dashboards anni chusi...DB connections yenni vunnay yenta yekkuvaku ika scale up cheyyachau yenta pcf space vundi....app logs...Grafana...AppD.... naa bonda....testing ku no link.

Same day they patched DB and OS...nodes nd services configuration lo gadabida chesaru...network path s issue...so, we r getting hit millions of evetns backlog

Financial domain ahh bro nuvvu work chesedi .. ma daggara kuda same protocol follow chestaru 

[JaiTDP]

4 minutes ago, Jombo0108 said:

@lokesh_rocks

https://gfycat.com/candidfirsthectorsdolphin

[superhit3]

6 minutes ago, Unixer said:

You can always UNDO patches Kno?

May take some time but might eliminate if patch related.. 

problem is not with Patches this afternoon -> evening -> night...they messed up with network paths resulted into huge backligs at queues...then issu with d onnections to process them

[superhit3]

11 minutes ago, JaiTDP said:

Haha.. luckily it’s backend data loading server bro not the UI. The clients didn’t have visibility abt this but we notified saying there will be some delay in processing data. 

total server crashed , Microsoft vaadu kuda hands up annaadu after reviewing logs. Thella manager(chaala manchodu) pushing me on UAT stuff saying client was waiting where I am fighting with this prod server  !!  
 

Oka 1 month back up dhi vunte dhaanitho laaginchesaamu. 

+  time sensitive data kademo ayithe

[superhit3]

15 minutes ago, cosmopolitan said:

All rounder lekka unav ga bro 

chesaru, thanks to them. But stress is high bro

[JaiTDP]

1 minute ago, superhit3 said:

+  time sensitive data kademo ayithe

ya kind of.

 

[cosmopolitan]

1 minute ago, superhit3 said:

chesaru, thanks to them. But stress is high bro

yeah.. weekend support full torture untadhi.. kani ee knowledge future lo help avuthundhi ..

[superhit3]

2 minutes ago, JaiTDP said:

ya kind of.

 

gotcha, u 1 lucky fellow

[superhit3]

21 minutes ago, cosmopolitan said:

yeah.. weekend support full torture untadhi.. kani ee knowledge future lo help avuthundhi ..

yemo, QA antente QA skills yemunaya ani amtrame looking....ivanni chesam ante aa*t care antaremoga?

[superhit3]

33 minutes ago, Jombo0108 said:

Financial domain ahh bro nuvvu work chesedi .. ma daggara kuda same protocol follow chestaru 

domain yendukule bhayya mana vallu domain septe alane anni dig chestaru. 

but, very time sensitive and live

[kathanayaka]

2 hours ago, superhit3 said:

full time bhayya...that too mana DB lo kondaru hate chese QA nenu. dabbulu yentho ivvaru...anthe yentho lontha matrame......kani On-Call rotation doing. PCF clloud....instances...scaling...bouncing ...memory...grafana...cpu utiliztion...   appd dashboards anni chusi...DB connections yenni vunnay yenta yekkuvaku ika scale up cheyyachau yenta pcf space vundi....app logs...Grafana...AppD.... naa bonda....testing ku no link.

Same day they patched DB and OS...nodes nd services configuration lo gadabida chesaru...network path s issue...so, we r getting hit millions of evetns backlog

orinee U tagalayya

docker kubernetes ECS EKS Serverless auto scaling inni tagaladdai kadara. 

inka "night 2 ki server crash ayyinid" story lu cheptav endira kuyya