Author: Daniel Blando Date: July 2025 Status: Proposed Background Distributors use a token-based ring to shard data across ingesters. Each ingester owns random tokens (32-bit numbers) in a hash ring. For each incoming series, the distributor: Hashes the series labels to get a hash value Finds the primary ingester (smallest token > hash value) When replication is enabled, selects additional replicas by moving clockwise around the ring Ensures replicas are distributed across different availabil...| Cortex
Author: Harry John Date: June 2025 Status: Proposed Overview Background Cortex currently implements distributed query execution by rewriting queries into multiple subqueries, handled through middlewares in the query-frontend. These split queries are scheduled via the query-scheduler, evaluated by queriers, and merged back in the query-frontend. This proposal introduces a new distributed query execution model based on Thanos PromQL engine. Terminology Logical plan - Parsed PromQL expression re...| Cortex – Proposals
Author: Bogdan Stancu Date: June 2025 Status: Proposed Overview This proposal outlines the design for a new API endpoint that will allow users to modify their current limits in Cortex. Currently, overrides can only be changed by administrators modifying the runtime configuration file and waiting for it to be reloaded. Problem Currently, when users need limit adjustments, they must: Manually editing the runtime configuration file Coordinate with users to verify the changes Potentially repeatin...| Cortex
Author: Alan Protasio, Ben Ye Date: April 2025 Status: Proposed Background Since the introduction of Block Storage in Cortex, TSDB format and Store Gateway is the de-facto way to query long term data on object storage. However, it presents several significant challenges: TSDB Format Limitations TSDB format, while efficient for write-heavy workloads on local SSDs, is not designed for object storage: Index relies heavily on random reads to serve queries, where each random read becomes a request...| Cortex
The horizontally scalable, highly available, multi-tenant, long term storage solution for Prometheus and OpenTelemetry Metrics.| Cortex
Author: Doğukan Teber Date: March 2023 Status: Proposed Overview If you run Cortex for multiple tenants you need to identify your tenants every time they send metrics or query them. This is needed to ensure that metrics can be ingested and queried separately from each other. For this purpose, the Cortex microservices require you to pass a Header called X-Scope-OrgID. The Cortex k8s manifests suggest deploying an NGINX server inside of each tenant which acts as a reverse proxy. Its sole purpo...| Cortex – Documentation
Author: Marco Pracucci Date: November 2020 Status: draft Introduction Queriers and store-gateways, at startup and periodically while running, need to scan the entire object store bucket to find tenants and blocks for each tenants. For each discovered block, they need to fetch the meta.json file and check for the existence of deletion-mark.json (used to signal a block is “marked for deletion”). The meta.json file is immutable and gets cached, but deletion-mark.json can be created at any ti...| Cortex – Documentation
Author: Marco Pracucci Date: March 2020 Status: accepted Problem In Cortex, when using the experimental blocks storage, each querier internally runs the Thanos BucketStore. This means that each querier has a full view over all blocks in the long-term storage and all blocks index headers are loaded in each querier memory. The querier memory usage linearly increase with number and size of all blocks in the storage, imposing a scalability limit to the blocks storage. In this proposal we want to ...| Cortex – Documentation
Author: Christian Simon Date: October 2020 Status: Accepted Overview This document aims to describe how to implement the ability to allow queries to cover data from more than a single Cortex tenant. Reasoning Adopting a tenancy model within an organization with each tenant representing a department comes with the disadvantage that it will prevent queries from spanning multiple departments. This proposal tries to overcome those limitations. Alternatives considered Aggregation in PromQL API cli...| Cortex – Documentation
Author: Peter Stibrany Date: November 2020 Status: Accepted Deletion of tenant data Problem When a tenant is deleted from the external system that controls access to Cortex, we want to clean up tenants data from Cortex as well. Background When using blocks storage, Cortex stores tenant’s data in several places: object store for long-term storage of blocks, ingesters disks for short-term storage. Ingesters eventually upload data to long-term storage, various caches: query-frontend, chunks, i...| Cortex – Documentation
Author: Jay Batra Date: March 2020 Status: proposal Problem In Cortex, currently, we are missing versioning of documentation. The idea is to have version documentation just like Prometheus.Prometheus. Documentation is the main source of information for current contributors and first-timers. A properly versioned documentation will help everyone to have a proper place to look for answers before flagging it in the community. In this proposal, we want to solve this. In particular, we want to: Ver...| Cortex – Documentation
Author: @annanay25 Reviewers: @jtlisi, @pstibrany, @cyriltovena, @pracucci Date: April 2020 Status: Accepted Overview Cortex uses modules to start and operate services with dependencies. Inter-service dependencies are specified in a map and passed to a module manager which ensures that they are initialised in the right order of dependencies. While this works really well, the implementation is tied in specifically to the Cortex struct and is not flexible for use with other projects like Loki, ...| Cortex – Documentation
Author: @jtlisi Reviewers: @pracucci, @pstibrany, @khaines, @gouthamve Date: March 2020 Status: Accepted Overview The purpose of this design document is to propose a set of standards that should be the basis of the Cortex HTTP API. This document will outline the current state of the Cortex http api and describe limitations that result from the current approach. It will also outline a set of paradigms on how http routes should be created within Cortex. Current Design As things currently stand,...| Cortex – Documentation
Author: Roy Chiang Date: May 2021 Status: Proposed --- Introduction As a part of pushing Cortex’s scaling capability at AWS, we have done performance testing with Cortex and found the compactor to be one of the main limiting factors for higher active timeseries limit per tenant. The documentation Compactor describes the responsibilities of a compactor, and this proposal focuses on the limitations of the current compactor architecture. In the current architecture, compactor has simple shardi...| Cortex – Documentation
Author: Allenzhli Date: January 2021 Status: Accepted, Implemented in PR #3879. Retention of tenant data Problem Metric data is growing over time per-tenant, at the same time, the value of data decreases. We want to have a retention policy like prometheus does. In Cortex, data retention is typically achieved via a bucket policy. However, this has two main issues: Not every backend storage support bucket policies Bucket policies don’t easily allow a per-tenant custom retention Background ten...| Cortex – Documentation
Author: Daniel Blando Date: August 2022 Status: Proposed Background Cortex implements a ring structure to share information of registered pods for each service. The data stored and used by the ring need to be implemented via Codec interface. Currently, the only supported Codec to the ring is Desc. Desc is a proto.Message with a list of instances descriptions. It is used to store the data for each pod and saved on a supported KV store. Currently, Cortex supports memberlist, consul and etcd as ...| Cortex – Documentation
Author: Soon-Ping Phang Date: June 2022 Status: Deprecated --- Introduction This proposal is deprecated in favor of the new proposal This proposal consolidates multiple existing PRs from the AWS team working on this feature, as well as future work needed to complete support. The hope is that a more holistic view will make for more productive discussion and review of the individual changes, as well as provide better tracking of overall progress. The original issue is #4435. Problem Rulers in C...| Cortex – Documentation
Author: Anand Rajagopal Date: Aug 2024 Status: Proposed --- Problem Rulers in Cortex currently run with a replication factor of 1, wherein each RuleGroup is assigned to exactly 1 ruler. This lack of redundancy creates the following risks: Rule group evaluation Missed evaluations due to a ruler outage, possibly caused by a deployment, noisy neighbour, hardware failure, etc. Missed evaluations due to a ruler brownout due to other tenant rule groups sharing the same ruler (noisy neighbour) API I...| Cortex – Documentation
Author: Rees Dooley Date: November 2021 Status: Accepted Overview This document aims to describe how to implement the ability to allow rules to cover data from more than a single Cortex tenant, here after referred to as federated rules. Since currently rules are owned by, query data from and save resulting series in the same tenant, this document aims to provide clear delineation of who owns a federated rule, what tenants the federated rule queries data from and where the federated rule saves...| Cortex – Documentation
Author: Josh Abreu Date: December 2020 Status: Accepted Context and Background The Cortex Alertmanager at its current state supports high-availability by using the same technique as the upstream Prometheus Alertmanager: we gossip silences and notifications between replicas to achieve eventual consistency. This allows us to tolerate machine failure without interruption of service. The caveat here is traffic between the ruler and Alertmanagers must not be load balanced as alerts themselves are ...| Cortex – Documentation
Author: @pracucci, @tomwilkie, @pstibrany Reviewers: Date: August 2020 Status: Accepted, implemented in PR #3090 Shuffle sharding and zone awareness Background Cortex shards the received series across all available ingesters. In a multi-tenant cluster, each tenant series are sharded across all ingesters. This allows to horizontally scale the series across the pool of ingesters but also suffers some issues: Given every tenant writes series to all ingesters, there’s no isolation between tenan...| Cortex – Documentation
Author: @pracucci, @tomwilkie, @pstibrany Reviewers: Date: August 2020 Status: Proposed, partially implemented Background Cortex currently supports sharding of tenants to a subset of the ingesters on the write path PR. This feature is called “subring”, because it computes a subset of nodes registered to the hash ring. The aim of this feature is to improve isolation between tenants and reduce the number of tenants impacted by an outage. This approach is similar to the techniques described ...| Cortex – Documentation
Author: @gotjosh Reviewers: @gouthamve, @pracucci Date: March 2020 Status: Accepted Problem Statement Prometheus holds metric metadata alongside the contents of a scrape. This metadata (HELP, TYPE, UNIT and METRIC_NAME) enables some Prometheus API endpoints to output the metadata for integrations (e.g. Grafana) to consume it. At the moment of writing, Cortex does not support the api/v1/metadata endpoint that Prometheus implements as metadata was never propagated via remote write. Recent work ...| Cortex – Documentation
Author: Ilan Gofman Date: June 2021 Status: Proposal Problem Currently, Cortex only implements a time series deletion API for chunk storage. We present a design for implementing time series deletion with block storage. We would like to have the same API for deleting series as currently implemented in Prometheus and in Cortex with chunk storage. This can be very important for users to have as confidential or accidental data might have been incorrectly pushed and needs to be removed. As well as...| Cortex – Documentation
Author: @roystchiang Reviewers: Date: August 2022 Status: Proposed Timeseries Partitioning in Compactor Introduction The compactor is a crucial component in Cortex responsible for deduplication of replicated data, and merging blocks across multiple time intervals together. This proposal will not go into great depth with why the compactor is necessary, but aims to focus on how to scale the compactor as a tenant grows within a Cortex cluster. Problem and Requirements Cortex introduced horizonta...| Cortex – Documentation
Cortex can be configured using a YAML file - specified using the -config.file flag - or CLI flags. In case you combine both, CLI flags take precedence over the YAML config file. The current configuration of any Cortex component can be seen by visiting the /config HTTP path. Passwords are filtered out of this endpoint. Reference To specify which configuration file to load, pass the -config.file flag at the command line.| Cortex
Gojek launched in 2010 as a call center for booking motorcycle taxi rides in Indonesia. Today, the startup is a decacorn serving millions of users across Southeast Asia with its mobile wallet, GoPay, and 20+ products on its super app. Want to order dinner? Book a massage? Buy movie tickets? You can do all of that with the Gojek app. The company’s mission is to solve everyday challenges with technology innovation.| Cortex
Author: @pstibrany Reviewers: Date: June 2020 Status: Replaced with migration guide (now removed from this site). Warning Suggestions from this proposal were implemented, but general procedure outlined here doesn’t quite work in Kubernetes environment. Please see chunks to blocks migration guide (now removed from this site) instead. Introduction This short document describes the first step in full migration of the Cortex cluster from using chunks storage to using blocks storage, specificall...| Cortex
Author: Joe Elliott Date: April 2020 Status: Proposed Overview This document aims to describe the role that the Cortex Query Frontend plays in running multitenant Cortex at scale. It also describes the challenges of horizontally scaling the query frontend component and includes several recommendations and options for creating a reliably scalable query-frontend. Finally, we conclude with a discussion of the overall philosophy of the changes and propose an alternative.| Cortex
General Notes Cortex has evolved over several years, and the command-line options sometimes reflect this heritage. In some cases the default value for options is not the recommended value, and in some cases names do not reflect the true meaning. We do intend to clean this up, but it requires a lot of care to avoid breaking existing installations. In the meantime we regret the inconvenience. Duration arguments should be specified with a unit like 5s or 3h.| Cortex
The query auditor is a tool bundled in the Cortex repository, but not included in Docker images – this must be built from source. It’s primarily useful for those developing Cortex, but can be helpful to operators as well during certain scenarios (backend migrations come to mind). How it works The query-audit tool performs a set of queries against two backends that expose the Prometheus read API. This is generally the query-frontend component of two Cortex deployments.| Cortex
REWE digital builds the technology that drives the e-commerce, app, and food pickup and delivery services for one of Germany’s largest grocery chains, REWE. Like other companies involved in the food supply chain, REWE has seen demand spike during the Covid-19 pandemic. Thanks to its adoption of Cortex last year, the monitoring team has been able to ensure stability at growing scale. The REWE digital subsidiary was started in 2014 to advance its parent company’s digital transformation, so ...| Cortex
Buoyant, the creator of Linkerd, has had a close relationship with Prometheus and related technologies for years. As of today, that relationship now includes Cortex. Linkerd is an open source service mesh for Kubernetes, and part of the CNCF, along with Prometheus and Cortex. Linkerd provides three key classes of features: observability, reliability, and security—all without requiring any changes to your code. That first pillar, observability, has motivated deep Prometheus integration, and ...| Cortex
You can use the Cortex query frontend with any Prometheus-API compatible service, including Prometheus and Thanos. Use this config file to get the benefits of query parallelisation and caching. # Disable the requirement that every request to Cortex has a # X-Scope-OrgID header. `fake` will be substituted in instead. auth_enabled: false # We only want to run the query-frontend module. target: query-frontend # We don't want the usual /api/prom prefix. http_prefix: server: http_listen_port: 9091...| Cortex
The query-tee is a standalone service which can be used for testing purposes to compare the query performances of 2+ backend systems (ie. Cortex clusters) ingesting the same exact series. This service exposes Prometheus-compatible read API endpoints and, for each received request, performs the request against all backends tracking the response time of each backend and then sends back to the client one of the received responses. How to run it You can run query-tee in two ways:| Cortex
Requests mirroring (or shadowing) is a technique you can use to mirror requests from a primary Cortex cluster to a secondary one. For example, requests mirroring can be used when you need to setup a testing Cortex cluster receiving the same series ingested by a primary one without having control over Prometheus remote write config (if you do, then configuring two remote write entries in Prometheus would be the preferred option).| Cortex
Cortex consists of multiple horizontally scalable microservices. Each microservice uses the most appropriate technique for horizontal scaling; most are stateless and can handle requests for any users while some (namely the ingesters) are semi-stateful and depend on consistent hashing. This document provides a basic overview of Cortex’s architecture. The following diagram does not include all the Cortex services, but does represent a typical deployment topology. The role of Prometheus Promet...| Cortex