messaging system

[Mr Mirchi]

 

System A   -> Kafka/some MQ service/NATs.  -> System B

 

System A sends data(new/updates) to the queue system, and then this queue system to system B. Queue system does not store the data, just a pass through system

 

system A requires user action to push the change to queue system.......okko saari oka 20 records bulk operation ga update chesam anuko DB lo...obivous ga user action kaadhu....ilanti situation lo meeru elaa handle chesaru to push  to system B through queue

 

 

[JackSeal]

In a situation where System A needs to push a bulk operation of around 20 records to System B through a queue system without requiring explicit user action, you can consider implementing an automated process. Here's a possible approach:

1. Monitor the database in System A for any changes or updates that meet your criteria for bulk operations.
2. Implement a background process or job scheduler that periodically checks for these changes, such as using a scheduled task or a cron job.
3. When the conditions for a bulk operation are met, gather the relevant records from the database.
4. Push these records as a batch to the queue system, ensuring that you maintain the order and integrity of the data.
5. System B, subscribed to the queue system, will receive the batch of records and process them accordingly.

By automating this process, you eliminate the need for explicit user action and ensure that the relevant changes are efficiently propagated from System A to System B through the queue system.

[cameraman]

use some stored procedures on system A DB to push data on any DB changes .

or use kakfa connect

 

[Mr Mirchi]

21 minutes ago, JackSeal said:

In a situation where System A needs to push a bulk operation of around 20 records to System B through a queue system without requiring explicit user action, you can consider implementing an automated process. Here's a possible approach:

1. Monitor the database in System A for any changes or updates that meet your criteria for bulk operations.
2. Implement a background process or job scheduler that periodically checks for these changes, such as using a scheduled task or a cron job.
3. When the conditions for a bulk operation are met, gather the relevant records from the database.
4. Push these records as a batch to the queue system, ensuring that you maintain the order and integrity of the data.
5. System B, subscribed to the queue system, will receive the batch of records and process them accordingly.

By automating this process, you eliminate the need for explicit user action and ensure that the relevant changes are efficiently propagated from System A to System B through the queue system.

i suggested this - 2 options

option 1: forget abt user action, have a trigger on a table/column and insert the record id into a sync table from trigger code., just record id , and cron job will pull the data using SP and sends to the queue.

option 2 : since DB changes are done rarely in prod, just have one column on the table and never update it from the code, but do update while making changes to the db and have a trigger on this column only and follow the option 1. , kind of dirty solution., but run the job manually since data is updated manually...you know when to run it.

 

 

offshore team..ignored this when they built thisls kind of queue flow....when I asked them, they said "we did not this abt it"  and they are happily making backend fixes without telling sys B(my system)

[Mr Mirchi]

15 minutes ago, cameraman said:

use some stored procedures on system A DB to push data on any DB changes .

or use kakfa connect

 

oh...kafka connect source to connect system A db to kafka as changes happen in the backend?

[tables]

29 minutes ago, JackSeal said:

In a situation where System A needs to push a bulk operation of around 20 records to System B through a queue system without requiring explicit user action, you can consider implementing an automated process. Here's a possible approach:

1. Monitor the database in System A for any changes or updates that meet your criteria for bulk operations.
2. Implement a background process or job scheduler that periodically checks for these changes, such as using a scheduled task or a cron job.
3. When the conditions for a bulk operation are met, gather the relevant records from the database.
4. Push these records as a batch to the queue system, ensuring that you maintain the order and integrity of the data.
5. System B, subscribed to the queue system, will receive the batch of records and process them accordingly.

By automating this process, you eliminate the need for explicit user action and ensure that the relevant changes are efficiently propagated from System A to System B through the queue system.

chatGPT response lekka undeee

[hunkyfunky2]

33 minutes ago, Mr Mirchi said:

 

System A   -> Kafka/some MQ service/NATs.  -> System B

 

System A sends data(new/updates) to the queue system, and then this queue system to system B. Queue system does not store the data, just a pass through system

 

system A requires user action to push the change to queue system.......okko saari oka 20 records bulk operation ga update chesam anuko DB lo...obivous ga user action kaadhu....ilanti situation lo meeru elaa handle chesaru to push  to system B through queue

 

 

Kafka connect can pull the data from amy database. Look into it

[Mr Mirchi]

28 minutes ago, hunkyfunky2 said:

Kafka connect can pull the data from amy database. Look into it

Look into ledhule….they used NATS jet stream