We have identified and fixed an additional scenario that may cause upgrade failures when moving from etcd v3.5 to v3.6. This post contains details, the fix, and additional workarounds. Please refer to issue 20793 to get detailed technical information. Issue In a previous post — How to Prevent a Common Failure when Upgrading etcd v3.5 to v3.6 — we described an upgrade issue affecting etcd versions in v3.5.1-v3.5.19. That issue was addressed in v3.5.20. However, a follow-up investigation re...| etcd
This is a post from the CNCF blog which we are sharing with our community as well. As a critical component of many production systems, including Kubernetes, the etcd project’s first priority is reliability. Ensuring consistency and data safety requires our project contributors to continuously improve testing methodologies. In this article, we will describe how we used advanced simulation testing to uncover subtle bugs, validate the robustness of our releases, and increase our confidence in ...| etcd
Table of Contents Introduction Security Features Migration to v3store Downgrade Feature Gates Livez/readyz checks v3discovery Performance Memory Throughput Breaking changes Old Binaries Are Incompatible with New Schema Versions Peer Endpoints No Longer Serve Client Requests Clear boundary between etcdctl and etcdutl Testing Critical bug fixes Upgrade issue Platforms Dependencies Dependency Bumping Guide Core Dependency Updates grpc-gateway@v2 grpc-ecosystem/go-grpc-middleware/providers/promet...| etcd
There is a common issue 19557 in the etcd v3.5 to v3.6 upgrade that may cause the upgrade process to fail. You can find detailed information and related discussions in the issue. TL; DR Users are required to first upgrade to etcd v3.5.20 (or a higher patch version) before upgrading to etcd v3.6.0. Failure to do so may result in an unsuccessful upgrade. What’s the symptom? When upgrading a multi-member etcd cluster from a version between v3.5.1 and v3.5.19 to v3.6.0, the upgrade may fail due...| etcd
KubeCon NA 2023 in Chicago is just around the corner! This year, the etcd project has a diverse range of talks, tutorials, and even interactive contribfest sessions for you to get involved in . As a critical foundational pillar of the Kubernetes ecosystem, etcd’s presence at Kubecon underscores its importance in ensuring all our Kubernetes clusters continue to have robust and reliable distributed persistent state. Here’s a detailed overview of what you can expect from the Etcd Project’s...| etcd
Special Interest Groups (SIGs) are a fundamental part of the Kubernetes project, with a substantial share of the community activity happening within them. When the need arises, new SIGs can be created, and that was precisely what happened recently. SIG etcd is the most recent addition to the list of Kubernetes SIGs. In this article we will get to know it a bit better, understand its origins, scope, and plans.| etcd
Background Users can configure the quota of the backend db size using flag --quota-backend-bytes. It’s the max number of bytes the etcd db file may consume, namely the ${etcd-data-dir}/member/snap/db file. Its default value is 2GB, and the suggested max value is 8GB. 2GB is usually sufficient for most use cases. If you run out of the db quota, you will see error message etcdserver: mvcc: database space exceeded when trying to write more data, and see alarm “NOSPACE” (see example below) ...| etcd
In the last few months, the team at Ada Logics has worked on integrating continuous fuzzing into the etcd project. This was an effort focused on improving the security posture of etcd and ensuring a continued good experience for etcds users. The fuzzing integration involved enrolling etcd in the OSS-Fuzz project and writing a set of fuzzers that would bring the test coverage of etcd up to a mature level. In total, 18 fuzzers were written, and eight bugs were found, demonstrating the work’s ...| etcd
When we launched etcd 3.4 back in August 2019, our focus was on storage backend improvements, non-voting member and pre-vote features. Since then, etcd has become more widely used for various mission critical clustering and database applications and as a result, its feature set has grown more broad and complex. Thus, improving its stability and reliability has been top priority in recent development. Today, we are releasing etcd 3.5. The past two years allowed for extensive iterations in fixi...| etcd
Jepsen tested and analyzed etcd 3.4.3, and had both good results and useful feedback to share with us. A key part of etcd’s design is strong consistency guarantees across the distributed key-value store. Kubernetes, Rook, OpenStack, and countless other critical software projects rely on etcd, in part, because of the etcd project’s focus on reliability and correctness. Over the years, the etcd team has put tremendous effort on building testing and chaos engineering frameworks.| etcd