15 open-source projects that quietly run the internet

The internet you interact with every day — streaming video, sending encrypted messages, loading web pages in milliseconds — runs on a vast and largely invisible layer of software that nobody sells. This software is open-source: its code is publicly available, freely distributed, and maintained by communities of developers who are often unpaid volunteers, or engineers whose employers contribute their time because the projects are too important to leave unfunded.

The scale of what this ecosystem supports is hard to overstate. The majority of the world's web servers run on open-source software. The encryption protecting your banking session was designed in public, not in a corporate lab. The operating system powering the server delivering your email is almost certainly Linux — which, despite being one of the most consequential software projects in history, was started in 1991 by a Finnish computer science student posting to a Usenet newsgroup.

What makes this story interesting is not just the software itself, but how it got here. Open-source infrastructure wasn't built by a coordinated plan. It accumulated through decades of individual decisions by engineers who wanted to solve a problem and share the solution. Over time, those solutions were adopted, improved, and embedded so deeply into the internet's architecture that replacing them would be enormously difficult — which creates both a resilience and a fragility. When a critical open-source project is well-maintained, the whole internet benefits. When it isn't, as a 2014 vulnerability in OpenSSL called Heartbleed demonstrated, the consequences can be worldwide.

The projects on this list were selected because they operate below the visible layer of the web — not the apps users open, but the infrastructure those apps depend on. Some are databases. Some are web servers. Some are programming languages that have become so universal they're practically utilities. A few are build tools or container systems that changed how software is deployed. Together they form a portrait of how the internet actually works: not as a collection of polished products, but as a living, evolving stack of shared code.

Credit:   Lukas / Unsplash

The operating system that runs most of the internet began as a side project. In August 1991, Linus Torvalds, then a 21-year-old student at the University of Helsinki, posted a message to the Usenet group comp.os.minix announcing he was working on a free operating system — "just a hobby," he wrote, "won't be big and professional like gnu." That disclaimer has aged poorly. Linux now powers the vast majority of the world's web servers, virtually all of the world's supercomputers, the Android operating system, and a substantial portion of the cloud infrastructure that underlies modern digital life.

Linux is a kernel — the core of an operating system that manages hardware resources and allows software to run. It is not a complete operating system on its own; distributions like Ubuntu, Debian, Red Hat, and Fedora package the Linux kernel with additional software to make it usable. But the kernel is the foundation, and it has proved durable beyond almost any reasonable expectation.

The reasons for Linux's dominance in server environments are both technical and economic. It is free to use, which matters at the scale of running thousands of servers. It is stable and reliable over long uptimes. It is highly configurable — engineers can strip it down to run on tiny embedded devices or tune it for high-performance computing workloads. And because its source code is open, organizations can audit it, patch it, and contribute improvements, rather than waiting for a vendor to fix a problem.

The Linux kernel development process is one of the most sophisticated software engineering operations in the world. Thousands of developers contribute code, which is reviewed through a rigorous patch submission process before being merged. Torvalds himself continues to oversee kernel development, though the project now involves major technology companies — Google $GOOGL, Intel $INTC, Red Hat, IBM $IBM, and others — whose engineers contribute substantial portions of the code.

Linux's influence extends far beyond servers. The Android platform, which runs on the majority of the world's smartphones, is built on the Linux kernel. Chromebooks run a Linux-based operating system. Consumer electronics, routers, smart TVs, and industrial control systems frequently run embedded Linux distributions. The project Torvalds described as a hobby in 1991 has become the most widely deployed operating system in history.

Credit: Brett Sayles / Pexels 

Before Nginx, before cloud load balancers, before the sprawling infrastructure of modern web hosting, there was Apache. The Apache HTTP Server was the dominant web server software for most of the internet's commercial history, and it remains in widespread use today. Understanding it means understanding how the early web was built.

Apache emerged in 1995 from a collection of patches applied to the NCSA HTTPd server, one of the first web servers ever written. A group of webmasters began exchanging modifications — "patches" — to fix bugs and add features the original software lacked. The name Apache is sometimes said to derive from "a patchy server," though the Apache Software Foundation has not officially confirmed this etymology. Regardless of origin, the project grew quickly, and by 1996 Apache had become the most popular web server on the internet, a position it held for more than 20 years.

Apache's architecture is built around modules — discrete components that can be enabled or disabled to add functionality. An administrator can load a module to enable SSL encryption, another to handle URL rewriting, another to support server-side scripting with PHP. This modular design made Apache enormously flexible, and it became the default choice for running dynamic websites during the era when LAMP (Linux, Apache, MySQL, PHP) was the standard web stack for most web applications.

The server's configuration system, based on a file called httpd.conf and a concept called .htaccess files, gave individual webmasters significant control over their server behavior. The .htaccess file in particular became ubiquitous — WordPress installations, for instance, rely on it for clean URL routing.

Apache's market share declined significantly after Nginx introduced an event-driven architecture that handled high concurrency more efficiently. Apache's traditional process-per-connection model consumed more memory under heavy load. But Apache never disappeared — it remained dominant in shared hosting environments and is still widely used in configurations that favor its flexibility and mature documentation over raw throughput. The Apache Software Foundation, the non-profit that oversees the project, also became the institutional home for dozens of other open-source projects, giving Apache an influence that extends well beyond its HTTP server.

Credit: panumas nikhomkhai / Pexels 

Nginx (pronounced "engine-x") was created to solve a specific problem: in the early 2000s, web servers based on Apache's architecture struggled to handle tens of thousands of simultaneous connections efficiently. Igor Sysoev, a Russian software engineer, began developing Nginx in 2002 to address this, and released it publicly in 2004. The software introduced an event-driven, asynchronous architecture that handled concurrent connections with far less memory overhead than the traditional process-per-request model.

The problem Sysoev was addressing had a name: the C10K problem, formulated in 1999 by engineer Dan Kegel, which asked how a web server could handle 10,000 simultaneous clients. Apache's default model assigned each connection a thread or process, which became expensive at scale. Nginx instead used a single-threaded event loop — similar to how Node.js would later be designed — that could handle many connections within a single process.

Nginx's efficiency made it the preferred choice for high-traffic sites. It is particularly well-suited to serving static files, acting as a reverse proxy in front of application servers, and handling SSL termination. Major websites use Nginx as the front layer of their infrastructure, offloading work from application servers by having Nginx handle static assets and route dynamic requests to backend services.

Beyond raw web serving, Nginx became essential as a reverse proxy and load balancer. In a typical modern deployment, Nginx sits in front of a cluster of application servers — Node.js, Python, or Ruby applications — distributing incoming requests and providing a consistent interface to the outside world. It handles connection queuing, request buffering, and can be configured to return cached responses for common requests, significantly reducing load on application servers.

Nginx, Inc., the company Sysoev founded around the project, was acquired by F5 Networks $FFIV in 2019 for around $670 million. The open-source project continues under that ownership, with a commercial version called Nginx Plus offering additional features. The base open-source version remains widely used and actively developed. By most measures of web server usage, Nginx and Apache together account for a substantial majority of active sites, with Nginx having surpassed Apache in market share among high-traffic sites.

Credit: panumas nikhomkhai / Pexels 

Relational databases are the workhorses of the internet's data layer, and MySQL became the default choice for a generation of web developers. Created by Swedish and Finnish developers Michael Widenius and David Axmark in 1995, MySQL offered a fast, reliable, and free alternative to commercial database systems like Oracle $ORCL. It became the "M" in the LAMP stack and powered much of the early web's dynamic content.

MySQL's design priorities were pragmatic: speed and simplicity over strict standards compliance. Early versions of MySQL omitted features like stored procedures, triggers, and foreign key constraints in favor of raw query performance. This made it well-suited to read-heavy web applications — online forums, e-commerce catalogs, content management systems — where the ability to retrieve data quickly mattered more than complex transactional integrity.

The database became the backbone of some of the internet's most visited sites. Wikipedia runs on MariaDB, a MySQL fork. Facebook $META used MySQL extensively before developing customizations and eventually Cassandra for certain workloads. WordPress, which powers a substantial share of all websites, uses MySQL as its default database. The relationship between WordPress and MySQL illustrates how deeply this database embedded itself in the web's infrastructure: millions of websites depend on it without their operators necessarily knowing its name.

Sun Microsystems acquired MySQL AB, the company behind the database, in 2008 for approximately $1 billion. When Oracle acquired Sun in 2010, concern spread through the open-source community about MySQL's future under a company that also sold a competing proprietary database. Those concerns led to the creation of MariaDB, a community fork initiated by Widenius himself, intended to ensure a fully open development path continued.

MySQL remains actively developed under Oracle and is still one of the most widely deployed databases in the world. Its longevity reflects the enormous installed base built during the LAMP era — applications don't switch databases easily, and MySQL's performance and compatibility have continued to improve.

Credit: Fez Brook / Pexels 

PostgreSQL is the database that developers reach for when they need MySQL's accessibility and Oracle $ORCL's correctness. It is a fully featured relational database that has been in continuous development since 1986, making it one of the longest-lived active open-source projects. Its reputation for standards compliance, data integrity, and advanced feature support has made it the preferred choice for applications where correctness matters as much as speed.

The project descends from a research database called POSTGRES developed at the University of California, Berkeley by Michael Stonebraker and his students. When Berkeley's funding for the project ended, a group of developers rewrote the query engine to support SQL and released the result as PostgreSQL in 1996. Development has continued under a non-profit called the PostgreSQL Global Development Group, with no single company controlling the project.

PostgreSQL's feature set is extensive. It supports complex data types — JSON, arrays, geometric data, full-text search — that relational databases traditionally required extensions to handle. It has robust support for transactions with proper ACID compliance, meaning data integrity is guaranteed even when operations fail partway through. Its support for complex queries, window functions, and common table expressions makes it capable of analytical workloads that MySQL historically handled poorly.

One distinguishing aspect of PostgreSQL's architecture is its extensibility. Users can define their own data types, operators, and index types. The PostGIS extension adds full geographic information system capabilities to PostgreSQL, making it the database underlying much of the world's geospatial software. The TimescaleDB extension transforms PostgreSQL into a time-series database. This extensibility means PostgreSQL has become the foundation for specialized databases that inherit its reliability while serving domain-specific needs.

PostgreSQL's rise in popularity accelerated significantly in the 2010s as cloud providers — Amazon $AMZN, Google $GOOGL, Microsoft $MSFT — offered managed PostgreSQL services, eliminating the operational burden of running the database yourself. For new projects, PostgreSQL has become the default choice among developers who prioritize reliability over historical familiarity with MySQL.

Credit: William Warby / Pexels 

Redis occupies an unusual position in the database world: it is a data store that lives entirely in memory, designed to be fast above almost everything else. Created by Italian developer Salvatore Sanfilippo in 2009, Redis started as a way to improve the performance of a real-time web analytics tool he was building. The core insight was that for certain types of data — counters, session information, leaderboards, caches — the latency of reading from disk was unnecessary. If you put the data in RAM, operations that would take milliseconds on a disk-based database can happen in microseconds.

The name stands for Remote Dictionary Server, which describes its fundamental model: a server that stores data as key-value pairs. But Redis extended this model with a set of data structures — strings, lists, sets, sorted sets, hashes, bitmaps, and more — each with its own set of operations. A sorted set, for instance, allows you to store a collection of elements ordered by a numerical score, which makes implementing a leaderboard or a priority queue trivial.

Redis became the near-universal solution for caching in web applications. When a web server needs to display a page that requires several database queries, it can store the result in Redis after the first request. Subsequent requests retrieve the cached result in microseconds rather than querying the database again. This pattern — application talks to Redis first, falls back to database if cache is cold — reduced database load across the industry and made web applications substantially more responsive.

Beyond caching, Redis became the standard infrastructure for rate limiting, session storage, pub/sub messaging, and real-time data pipelines. Its atomic operations — which guarantee that certain operations complete without interruption — made it reliable for distributed systems that needed to coordinate without race conditions.

Redis Labs made a controversial licensing change in 2024, moving away from an open-source license to a more restrictive model. This prompted the Linux Foundation to fork the project under the name Valkey, which has since attracted support from major cloud providers. The episode illustrates the complex relationship between open-source projects and the commercial entities that often develop them.

Credit: Markus Spiske / Pexels 

curl is among the most widely installed pieces of software on earth, and most of the people who use it don't know it by name. It is a command-line tool and library for transferring data over networks, and it ships by default with almost every version of Linux, macOS, and Windows. When a developer writes a script that fetches data from an API, or when a piece of embedded firmware checks a server for an update, curl is often the tool doing the actual work.

The project was created by Swedish developer Daniel Stenberg in 1998, starting as a simple tool to download currency exchange rates. Stenberg has maintained it ever since — an extraordinary span of continuous involvement that represents one of the longest-running sole-maintainer projects in open-source history. Despite its ubiquity, curl operated for much of its history with minimal funding, a situation that drew attention during broader discussions about open-source sustainability.

curl supports an enormous range of protocols: HTTP, HTTPS, FTP, SFTP, SCP, SMTP, and dozens of others. This breadth made it the standard tool for network transfers across different contexts. Its -v (verbose) flag, which shows every step of a network transaction, made it invaluable for debugging. Its ability to set headers, send POST data, handle cookies, follow redirects, and authenticate with various schemes meant it could interact with virtually any network service.

The libcurl library — the embeddable version of curl that developers can include in their own software — has been incorporated into thousands of applications. It is present in routers, printers, cars, medical devices, and satellites. The curl project's security advisories receive attention from hardware manufacturers who ship embedded devices containing the library, because a vulnerability in curl is a vulnerability in all of them.

Stenberg has written extensively about the experience of maintaining critical infrastructure as an individual, including the financial precarity that characterized curl's first two decades. His situation became a reference point in discussions about how the open-source ecosystem fails to adequately support the individuals maintaining its most critical components.

Credit:   FlyD / Unsplash

OpenSSL is the library that encrypts the internet. When a browser establishes a secure connection to a website — the padlock icon in the address bar — the underlying cryptographic operations are often performed by OpenSSL or one of its derivatives. It implements the TLS (Transport Layer Security) protocol, which protects data in transit across the internet, and provides a wide set of cryptographic functions that applications depend on for secure operations.

The project began in 1998 as a fork of SSLeay, a library developed by Eric Young and Tim Hudson. It became the dominant open-source implementation of SSL/TLS protocols and was incorporated into web servers, databases, VPN software, email clients, and thousands of other applications. Because cryptographic code is difficult to write correctly and even more difficult to audit, the open-source community largely consolidated around OpenSSL rather than each project implementing its own.

That consolidation created a single point of failure. In April 2014, researchers disclosed the Heartbleed vulnerability — a bug in OpenSSL's implementation of the TLS heartbeat extension that allowed an attacker to read arbitrary memory from a server running a vulnerable version. The flaw affected a staggering proportion of HTTPS servers worldwide. It required emergency patches from operating system vendors, web server operators, certificate authorities, and hardware manufacturers simultaneously. Private keys had potentially been exposed, meaning SSL certificates had to be revoked and reissued at scale.

The Heartbleed disclosure revealed that OpenSSL, despite being one of the most critical pieces of security infrastructure on the internet, was maintained by a tiny team with minimal funding. At the time, the project had roughly one full-time employee. The Core Infrastructure Initiative, funded by major technology companies, was formed in response to provide financial support to critical open-source security projects.

OpenSSL has since undergone significant security improvements, code audits, and development process changes. The project now receives more resources than it did before Heartbleed. But the episode remains the clearest illustration of what can go wrong when the internet concentrates critical dependency on an underfunded open-source project.

Credit:   Chris Ried / Unsplash

Git changed how software is made. Before Git, version control systems — software that tracks changes to code over time — were centralized: all developers shared a single repository on a central server. If you wanted to branch, merge, or work offline, the tools were cumbersome. Git introduced a distributed model in which every developer has a complete copy of the repository, enabling workflows that would have been impractical before.

Linus Torvalds created Git in 2005 to manage Linux kernel development after the community's previous version control tool, BitKeeper, became unavailable. Torvalds wrote the initial version in about two weeks, with a focus on speed and data integrity. Git uses a content-addressable storage model — every file and commit is identified by a cryptographic hash of its contents — which means corruption or tampering is detectable. This design choice was intentional: Torvalds wanted a system that made it impossible to undetectably alter history.

Git's distributed design meant that operations like branching and merging, which had been painful in centralized systems, became fast and cheap. A developer can create a branch in milliseconds, work on it for days, and merge it back with a complete, auditable history. This workflow enabled the open-source collaboration model that platforms like GitHub and GitLab later built businesses around.

The branching and merging capabilities shaped not just individual projects but entire development methodologies. The "Git flow" branching strategy, published in 2010 by Vincent Driessen, became widely adopted in software teams. Continuous integration systems, which automatically test code when changes are proposed, are built on the assumption that Git-style pull requests exist. The entire modern software development practice of code review emerged from workflows that Git made practical.

GitHub, launched in 2008, made Git accessible to a far wider audience by providing a web interface, issue tracking, and social features. Microsoft $MSFT acquired GitHub in 2018 for $7.5 billion, a price that reflected how central the platform had become to software development globally. But GitHub is a product built on top of Git — the version control system itself remains an open-source project that Torvalds initiated and the community has extended.

Credit:  Tom Fisk / Pexels 

Kubernetes is container orchestration software — a system for deploying, scaling, and managing applications packaged as containers across clusters of machines. It was developed by Google $GOOGL engineers and released as open-source in 2014, based on internal Google systems called Borg and Omega that the company had used to manage its own workloads for years. The release of Kubernetes shifted how the industry thought about deploying software at scale.

A container, in the software sense, is a lightweight package that includes an application and everything it needs to run — its dependencies, configuration, and runtime — isolated from the host system. Docker popularized containers and made them practical for development teams. Kubernetes solved the harder problem of what to do with containers at scale: how do you deploy 500 of them, ensure they restart when they fail, balance load across them, roll out updates without downtime, and manage their networking?

Kubernetes introduced a declarative model for infrastructure: instead of specifying the steps to deploy software, you describe the desired state — "I want three instances of this service running" — and Kubernetes figures out how to achieve and maintain that state. If a node fails, Kubernetes reschedules the affected containers elsewhere. If traffic spikes, Kubernetes can scale up automatically. This model made operations more reliable and reduced the manual toil of managing servers.

The project was donated to the Cloud Native Computing Foundation in 2016, removing it from direct Google control and making it a vendor-neutral community project. This governance decision was important: it reassured competitors and users that no single company could dominate the project's direction. The CNCF became the home for a broad ecosystem of cloud-native tools built around Kubernetes.

Kubernetes transformed cloud computing by making workloads more portable. Because Kubernetes runs consistently across different cloud providers, organizations gained the ability to run applications without being fully locked into one vendor's platform. All three major cloud providers — Amazon $AMZN, Google, and Microsoft $MSFT — offer managed Kubernetes services, which speaks to how central the project became to cloud infrastructure despite originating at one of those providers' competitors.

Credit: Markus Winkler / Pexels

Debian is a Linux distribution — a complete operating system built on the Linux kernel with a package management system, a set of included software, and a set of policies governing how software is maintained. It is not the most visible Linux distribution; Ubuntu, which is built on Debian, gets more attention. But Debian is foundational to a large portion of the Linux ecosystem and has been developed entirely by volunteers since Ian Murdock announced the project in 1993.

What distinguishes Debian is its governance model and its commitment to free software principles. The project is run democratically: a Debian Project Leader is elected annually by the project's developers, and significant policy decisions are made through voting. The Debian Social Contract and Debian Free Software Guidelines, published in 1997, articulated principles about software freedom that became influential throughout the open-source movement. The Open Source Definition, which defines what "open source" means, was derived directly from the Debian Free Software Guidelines.

Debian's package management system, built around APT (Advanced Package Tool) and the .deb package format, was one of the first to solve the "dependency hell" problem — the challenge of installing software that requires other software to be installed first, which in turn requires other software. Debian's dependency resolution made it possible to install complex software with a single command, automatically fetching and installing everything needed. This approach became the model for package management across Linux distributions.

The distribution also became the foundation for a large number of derivative distributions. Ubuntu, which became the dominant desktop and cloud Linux distribution, is directly derived from Debian. Raspberry Pi OS, used on the popular single-board computer, is Debian-based. Many Docker container images use Debian or Debian-derived base images.

Debian's release cycle is deliberately slow — new stable releases come every two years, with a lengthy freeze period for testing. This conservatism makes it a reliable choice for servers where stability matters more than having the latest software versions. The project's longevity and stability, maintained entirely by volunteers across three decades, is an unusual achievement in software.

Credit: Nemuel Sereti / Pexels 

Python is a programming language, but it belongs on this list because it has become one of the primary languages of internet infrastructure, data science, and automation. Created by Dutch computer scientist Guido van Rossum and first released in 1991, Python was designed with readability as a central goal — its syntax enforces indentation to structure code, which van Rossum believed made programs easier to understand. That design decision made Python approachable enough to spread far beyond its original audience.

Python's role in internet infrastructure takes several forms. It is the language of Django and Flask, two web frameworks that power thousands of web applications. Instagram's server infrastructure was built on Django. Dropbox was built in Python. The language is used heavily at Google $GOOGL, Spotify $SPOT, Reddit $RDDT, and YouTube. When developers need to write a script to process data, automate a task, or prototype a service quickly, Python is frequently the choice.

The language's dominance in data science and machine learning is equally significant. NumPy, Pandas, Scikit-learn, TensorFlow, and PyTorch — the primary libraries of scientific computing and machine learning — are all Python libraries. The emergence of machine learning as a practical technology in the 2010s drove Python adoption dramatically: researchers and engineers who needed to work with these tools had to work in Python. By the early 2020s, Python had become the most widely used programming language in the world by several measures.

Python is managed by the Python Software Foundation, a non-profit organization that holds the intellectual property for the language and supports the community. Van Rossum served as what the community called "Benevolent Dictator for Life" — the final arbiter of language design decisions — until he stepped down from the role in 2018. Language governance is now handled by a steering council.

One characteristic of Python's ecosystem is the central role of PyPI (the Python Package Index), a repository of over 500,000 packages that Python developers can install with a single command. PyPI is the distribution infrastructure for the Python ecosystem — a critical piece of internet infrastructure that, like many others, depends significantly on volunteer effort and relatively modest funding.

Credit: Christina Morillo / Pexels 

Ansible is automation software for managing computer infrastructure. It was created by Michael DeHaan and first released in 2012, solving a problem that had grown more acute as organizations deployed more servers: how do you configure thousands of machines consistently, apply updates across all of them, and document what's installed where? Ansible approached this problem with a design philosophy centered on simplicity — it uses a human-readable format called YAML to describe desired system states, requires no software installed on the machines being managed, and connects using ordinary SSH.

The "no agent" design was a significant departure from earlier configuration management tools like Puppet and Chef, which required installing and maintaining agent software on every managed machine. Ansible's agentless approach reduced the operational overhead of adoption significantly — you could point it at existing servers without modifying them first. This made it accessible to teams that found the complexity of agent-based tools a barrier.

Ansible's playbooks — the YAML files that describe automation tasks — became the primary way many organizations documented their infrastructure as code. Rather than relying on institutional knowledge about how servers were configured, teams could read their Ansible playbooks to understand exactly what state each system was supposed to be in. This "infrastructure as code" approach made environments reproducible and auditable in a way that manual configuration never could be.

Red Hat acquired Ansible in 2015 for approximately $150 million, integrating it into its enterprise product portfolio while keeping the core project open-source. The acquisition reflected how important automation had become to enterprise IT operations. Red Hat subsequently became part of IBM $IBM, which acquired it in 2019.

Ansible's role in the modern internet is largely invisible: it runs in the background during deployments, keeping servers configured correctly and enabling teams to manage hundreds or thousands of machines with the same effort that once required physical access to each one. When a cloud provider updates the configuration of its servers, or when a company rolls out a security patch across its entire fleet, the mechanism is often something like Ansible operating quietly in the background.

Credit: Tima Miroshnichenko / Pexels 

Elasticsearch is a search engine built on top of Apache Lucene, designed to make full-text search fast, scalable, and accessible over a simple API. It was created by Shay Banon, first released in 2010, and has since become the most widely used search engine in enterprise and web applications. When a website's search bar returns results in milliseconds, or when a log management system lets engineers search through billions of log entries, Elasticsearch is often the engine underneath.

The software's design centers on distributed architecture from the start. An Elasticsearch cluster spreads data across multiple nodes, automatically handling replication and load balancing. Queries are processed in parallel across shards, enabling search performance at scales that single-machine databases cannot match. Adding nodes to a cluster scales capacity horizontally — a design that aligned well with the cloud infrastructure patterns that became standard in the 2010s.

Beyond search, Elasticsearch became the foundation of the ELK stack — Elasticsearch, Logstash (a data pipeline tool), and Kibana (a visualization interface). This combination became the dominant open-source solution for log aggregation and analysis. Operators of web infrastructure need to collect logs from hundreds of services, index them for fast lookup, and visualize patterns to debug problems. The ELK stack provided this capability with a coherent interface and strong ecosystem support.

Elasticsearch is maintained by Elastic, the company Banon founded. Like Redis, Elastic changed its licensing terms in 2021, moving from the Apache License to a more restrictive proprietary-adjacent license, citing concerns about cloud providers offering Elasticsearch as a service without contributing to its development. Amazon $AMZN responded by creating OpenSearch, a fork of the last Apache-licensed version of Elasticsearch, which continues as a separate open-source project.

The question of how to sustain commercially viable businesses around open-source software — while preventing cloud providers from capturing value without contributing — has become one of the defining tensions of the modern open-source era. Elasticsearch and Redis are two of the most prominent examples of projects where that tension led to visible forks and governance disputes.

Credit:  Luke Chesser / Unsplash

Prometheus is a monitoring system and time-series database created at SoundCloud and released as open-source in 2012. It was designed to solve a specific problem: how do you monitor modern distributed applications, where a single service might consist of dozens of microservices running across hundreds of containers, with no fixed IP addresses and services spinning up and down continuously? The monitoring tools of the previous decade assumed a relatively static infrastructure that could be configured manually. Prometheus assumed infrastructure was ephemeral and needed to discover and track services automatically.

The project's approach to data collection, called the pull model, was distinctive. Rather than having services push metrics to a central server, Prometheus regularly polls target services for their metrics over HTTP. Services expose a metrics endpoint — typically at /metrics — and Prometheus scrapes it on a schedule. This design made it easy to add monitoring to any service that could expose an HTTP endpoint, and it gave operators a clear picture of what was being monitored at any given time.

Prometheus introduced a powerful query language called PromQL, which allows operators to express complex queries about time-series data. An operator can ask for the 95th percentile of request latency across all instances of a service over the past five minutes, or track the rate of errors compared to successful requests. These queries feed dashboards and alerting rules that tell engineers when something is wrong before users notice.

Prometheus was the second project to join the Cloud Native Computing Foundation in 2016, after Kubernetes. Its inclusion reflected how central monitoring had become to cloud-native operations. A large ecosystem of "exporters" grew up around Prometheus — software components that translate metrics from databases, message queues, hardware, and other systems into the Prometheus format, making it possible to monitor nearly any component of an infrastructure stack with a consistent query language.

Grafana, a visualization tool that integrates tightly with Prometheus, became the standard way to display Prometheus data. Together they represent the de facto observability stack for cloud-native infrastructure, used across the industry to monitor the health of the systems that serve modern web applications.