Tech Tips, Tricks & Trivia

Scylla and Cassandra are both open-source, distributed NoSQL databases designed to handle large amounts of data across many commodity servers. They are often compared due to their similar data models and use cases. Scylla offers the horizontal scale-out and fault-tolerance of Apache Cassandra but delivers 10X the throughput. Scylla achieves the same throughput with a 3-node cluster as a 30-node Cassandra cluster.   This comparison chart from a ScyllaDB datasheet   [PDF] highlights the difference in features:

Key points from the ScyllaDB technical paper  SQL to NoSQL: Architecture Differences and Considerations for Migration  [PDF] -  While RDBMSs are not fit for distributed deployments, non-relational databases are designed specifically to support such topologies. A number of alternative non-relational database systems have been proposed, including Google’s Bigtable (2006) and Amazon’s Dynamo (2007). The papers for these projects paved the way for Cassandra (2008) and MongoDB (2009).  Despite their origins in a long-forgotten technology cycle, relational SQL databases are by no means ‘legacy’ technology. Some SQL databases, notably PostgreSQL and MySQL, have experienced a recent resurgence in popularity. A new generation of NewSQL databases, notably Google Spanner and CockroachDB, leverage SQL as a query language and offer a distributed architecture similar to that of NoSQL databases yet provide full transactional support. Relational databases are defined in part by...

Excerpts from the article " Scalable Efficient Big Data Pipeline Architecture " by Satish Chandra Gupta The big data pipeline is the railroad on which the heavy wagons of ML run. A data pipeline stitches together the end-to-end operation consisting of collecting the data, transforming it into insights, training a model, delivering insights, applying the model whenever and wherever the action needs to be taken to achieve the business goal. There are 5 stages in the big data pipeline: 🔹 Collect - Collect data from internal & external sources 🔹 Ingest - Ingest data through batch jobs and streams 🔹 Store - Store in Data Lake and/or Warehouse 🔹 Compute - Compute analytics aggregations and ML features 🔹 Use - Use it in dashboards, data science, and ML Typical serverless architectures of big data pipelines on Amazon Web Services, Microsoft Azure, and Google Cloud Platform (GCP) are shown below.