Data Processing Pipelines A data processing pipeline is a series of interconnected steps or stages to transform, enrich, and analyze data. These steps can include tasks such as extracting data from various sources, cleaning and filtering the data, transforming it
The Hadoop Ecosystem is a framework and suite of tools that tackle the many challenges in dealing with big data. Although Hadoop has been on the decline for some time, there are organizations like LinkedIn where it has become a
Storage used to be simple. A single large expensive drive (SLED) was all that was needed; then the industry moved onto redundant arrays of inexpensive disks (RAID) – still relatively straightforward. Once hard drives began to be replaced by solid-state
Spotify, the Freemium streaming music company, has over 200 million users with 96 million of those paying customers with around 40 million tracks available. Spotify has recently added podcasts to its streaming platform. These numbers make it the world’s biggest
Cockroach Labs, the company behind CockroachDB, the global SQL database for building worldwide cloud services, yesterday announced Managed CockroachDB and the latest release of its mainstay CockroachDB 2.1. Businesses are increasingly migrating their workloads to the cloud. However, traditional database
The Apache Software Foundation has announced that Pulsar has graduated to become its latest Top Level Project. Apache explains Pulsar as a “next-generation, Open Source distributed publish-and-subscribe messaging system designed for scalability, flexibility, and no data loss.” By graduating Pulsar
This post offers an overview of the Raft algorithm used for implementing distributed consensus. Within the post, we’ll look at what consensus and distributed consensus means, why Raft was developed and give an overview of how it works, along with
ACIDRain attacks are a new kind of cyberattack on database-backed applications, in which a cybercriminal systematically exploits concurrency-related vulnerabilities that lead to the violation of an application integrity constraint. This might be for instance, vouchers that should only be spent
Apache Kafka, the open source distributed streaming platform, has become increasingly popular in the DevOps community, experiencing a meteoric rise in popularity, particularly within the developer and engineer community, over the last seven years. It was initially released in 2011.
Twitter’s Hadoop System is at the core of its data platform, providing vast storage for analytics of its user actions and allowing its infrastructure to scale whenever needed. Twitter’s large Hadoop clusters number among the world’s largest; its Hadoop file
RAID, or Redundant Array of Independent Disks, is a way to spread data across a set of drives to prevent the loss of one drive causing permanent data loss. RAID works in two different ways: (i) a complete mirror image
According to a new survey from data Artisans interviewing Apache Flink users, the majority of surveyed businesses are planning on deploying more applications powered by Apache Flink software in the year ahead. Apache Flink is an open-source stream processing framework
Founded in 2011, Redis is one of the most innovative companies around, encompassing both the open-source multipurpose database as well as Redis Enterprise, which enhances Redis Open Source with high availability, linear scaling, and deployment via public and private clouds, on-premises,
