mettastar Posted February 7, 2020 Report Posted February 7, 2020 meru vaade query engine (db) enti to serve users ? high concurrency and low latency (sub second) unna db ethukuthunna.. any suggestions from your experience ? TPS - 400 (400 requests per second support chesavi kaavali or close to that) Latency - sub seconds to less than 5 secs okay should be able to handle TBs of data SQL support unte manchidhi Quote
kasi Posted February 7, 2020 Report Posted February 7, 2020 kaka......asalu ni req enti, nuvu nosql db kosam looking aa??? do you know CAP theorem ???? if not check it out...... basic architecture iythea raw data ---> Spark/Hadoop(etl)--->nosql db ---> UX for users to generate reports Quote
mettastar Posted February 7, 2020 Author Report Posted February 7, 2020 1 minute ago, kasi said: kaka......asalu ni req enti, nuvu nosql db kosam looking aa??? do you know CAP theorem ???? if not check it out...... basic architecture iythea raw data ---> Spark/Hadoop(etl)--->nosql db ---> UX for users to generate reports Elasticsearch vaduthunnam kaani bulk loading chesthunte sasthundi adi. As it does indexing. Kylin ani okati undi explore chesa kaani adi olap cube based .. but ma data emo continues updates untayi historical data kuda .. rebuilding cubes is not easy adi kuda ruled out for now. Any other nosql db in specific ? NoSqls tho problem endhante non key columns meda sorting or join btw other tables not easy run time lo Quote
kasi Posted February 7, 2020 Report Posted February 7, 2020 22 minutes ago, mettastar said: Elasticsearch vaduthunnam kaani bulk loading chesthunte sasthundi adi. As it does indexing. Kylin ani okati undi explore chesa kaani adi olap cube based .. but ma data emo continues updates untayi historical data kuda .. rebuilding cubes is not easy adi kuda ruled out for now. Any other nosql db in specific ? NoSqls tho problem endhante non key columns meda sorting or join btw other tables not easy run time lo did you try increasing the resources.....scaling the server??? ES is distributed, so i think you can try increasing the sever resources Quote
Spartan Posted February 7, 2020 Report Posted February 7, 2020 27 minutes ago, mettastar said: Elasticsearch vaduthunnam kaani bulk loading chesthunte sasthundi adi. As it does indexing. Kylin ani okati undi explore chesa kaani adi olap cube based .. but ma data emo continues updates untayi historical data kuda .. rebuilding cubes is not easy adi kuda ruled out for now. Any other nosql db in specific ? NoSqls tho problem endhante non key columns meda sorting or join btw other tables not easy run time lo did u explore DocumentDB... Quote
mettastar Posted February 7, 2020 Author Report Posted February 7, 2020 11 minutes ago, Spartan said: did u explore DocumentDB... ninnane chadiva but use cases lo analytical ki vadachu ani mention cheyaledu .. bulk load ela untado chudali Quote
phoeniix Posted February 7, 2020 Report Posted February 7, 2020 5 minutes ago, mettastar said: ninnane chadiva but use cases lo analytical ki vadachu ani mention cheyaledu .. bulk load ela untado chudali elastic search enduku vadutunnaru....reports kosama kibana through generate cheyadam kosam Quote
phoeniix Posted February 7, 2020 Report Posted February 7, 2020 If not nosql you may consider Thoughspot BI tool but it is not an open source. Quote
phoeniix Posted February 7, 2020 Report Posted February 7, 2020 MariaDB too as a consumption layer . Quote
phoeniix Posted February 7, 2020 Report Posted February 7, 2020 57 minutes ago, kasi said: kaka......asalu ni req enti, nuvu nosql db kosam looking aa??? do you know CAP theorem ???? if not check it out...... basic architecture iythea raw data ---> Spark/Hadoop(etl)--->nosql db ---> UX for users to generate reports typical architecture. Quote
Spartan Posted February 7, 2020 Report Posted February 7, 2020 20 minutes ago, mettastar said: ninnane chadiva but use cases lo analytical ki vadachu ani mention cheyaledu .. bulk load ela untado chudali https://segment.com/blog/choosing-a-database-for-analytics/ Quote
Sarvapindi Posted February 7, 2020 Report Posted February 7, 2020 Data engineers more of development related.. Quote
dewarist Posted February 7, 2020 Report Posted February 7, 2020 Apache Cassandra. A row partitioned distributed NO SQL data store. We use it for storing inventory from thousands of locations and serve inventory lookup queries What kind of data are you storing? Quote
mettastar Posted February 7, 2020 Author Report Posted February 7, 2020 31 minutes ago, dewarist said: Apache Cassandra. A row partitioned distributed NO SQL data store. We use it for storing inventory from thousands of locations and serve inventory lookup queries What kind of data are you storing? Sales invent forecast click stream customer behavior Quote
mettastar Posted February 7, 2020 Author Report Posted February 7, 2020 33 minutes ago, dewarist said: Apache Cassandra. A row partitioned distributed NO SQL data store. We use it for storing inventory from thousands of locations and serve inventory lookup queries What kind of data are you storing? Volume ey size lo untadi bro? And size of cluster? Query latency? Does it support ansi sql? Quote
.thumb.jpg.ce42bb16ec102475a78c2087ce269c7d.jpg)
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.