Cortex supports data replication for different services. By default, data is transparently replicated across the whole pool of service instances, regardless of whether these instances are all running within the same availability zone (or data center, or rack) or in different ones. It is completely possible that all the replicas for the given data are held within the same availability zone, even if the Cortex cluster spans multiple zones. Storing multiple replicas for a given data within the s...| Cortex
Cortex can be configured using a YAML file - specified using the -config.file flag - or CLI flags. In case you combine both, CLI flags take precedence over the YAML config file. The current configuration of any Cortex component can be seen by visiting the /config HTTP path. Passwords are filtered out of this endpoint. Reference To specify which configuration file to load, pass the -config.file flag at the command line.| Cortex
General Notes Cortex has evolved over several years, and the command-line options sometimes reflect this heritage. In some cases the default value for options is not the recommended value, and in some cases names do not reflect the true meaning. We do intend to clean this up, but it requires a lot of care to avoid breaking existing installations. In the meantime we regret the inconvenience. Duration arguments should be specified with a unit like 5s or 3h.| Cortex
The querier service handles queries using the PromQL query language. This document dives into the storage-specific details of the querier service. The general architecture documentation applies too. The querier is stateless. How it works The querier needs to have an almost up-to-date view over the entire storage bucket, in order to find the right blocks to lookup at query time. The querier can keep the bucket view updated in to two different ways:| Cortex
The compactor is an service which is responsible to: Compact multiple blocks of a given tenant into a single optimized larger block. This helps to reduce storage costs (deduplication, index size reduction), and increase query speed (querying fewer blocks is faster). Keep the per-tenant bucket index updated. The bucket index is used by queriers, store-gateways and rulers to discover new blocks in the storage. The compactor is stateless. How compaction works The blocks compaction has two main b...| Cortex
This page shares some tips and things to take in consideration when setting up a production Cortex cluster based on the blocks storage. Ingester Ensure a high number of max open file descriptors The ingester stores received series into per-tenant TSDB blocks. Both TSDB WAL, head and compacted blocks are composed by a relatively large number of files which gets loaded via mmap. This means that the ingester keeps file descriptors open for TSDB WAL segments, chunk files and compacted blocks whic...| Cortex
In order to query series inside blocks from object storage, the store-gateway has to know certain initial info from each block index. In order to achieve so, the store-gateway builds an index-header for each block and stores it on local disk; such index-header is built by downloading specific pieces of original block’s index and storing them on local disk. Index header is then used by store-gateway at query time. Store-gateways build the index-header with specific sections of the block’s ...| Cortex
The bucket index is a per-tenant file containing the list of blocks and block deletion marks in the storage. The bucket index itself is stored in the backend object storage, is periodically updated by the compactor, and used by queriers, store-gateways and rulers to discover blocks in the storage. The bucket index usage is optional and can be enabled via -blocks-storage.bucket-store.bucket-index.enabled=true (or its respective YAML config option). Benefits The querier, store-gateway and ruler...| Cortex
Cortex consists of multiple horizontally scalable microservices. Each microservice uses the most appropriate technique for horizontal scaling; most are stateless and can handle requests for any users while some (namely the ingesters) are semi-stateful and depend on consistent hashing. This document provides a basic overview of Cortex’s architecture. The following diagram does not include all the Cortex services, but does represent a typical deployment topology. The role of Prometheus Promet...| Cortex
A StatefulSet runs a group of Pods, and maintains a sticky identity for each of those Pods. This is useful for managing applications that need persistent storage or a stable, unique network identity.| Kubernetes
Expose an application running in your cluster behind a single outward-facing endpoint, even when the workload is split across multiple backends.| Kubernetes