AIStore + HuggingFace: Distributed Downloads for Large-Scale Machine Learning| AIStore
I didn’t want to write this blog.| AIStore
Single-Object Copy/Transform Capability| AIStore
The current state of the art involves executing data pre-processing, augmentation, and a wide variety of custom ETL workflows on individual client machines. This approach lacks scalability and often results in significant performance degradation due to unnecessary data movement. Unlike most open-source and cloud ETL solutions, AIStore performs transformations on the same machines that store your data, minimizing redundant transfers by exploiting data locality.| AIStore
In distributed systems, maintaining seamless connectivity during lifecycle events is a key challenge. If the cluster’s state changes while read operations are in progress, transient errors might occur. To overcome these brief disruptions, we need an effective, intelligent retry mechanism.| AIStore
AIStore v3.28 introduces a unified rate-limiting capability that works at both the frontend (client-facing) and backend (cloud-facing) layers. It enables proactive control to prevent hitting limits and reactive handling when limits are encountered — all configurable at both the cluster and bucket levels, with zero performance overhead when disabled.| AIStore
The newly available support for Oracle Cloud Infrastructure (“OCI”) Object Storage was made possible by adopting OCI’s API via their OCI Golang SDK. OCI also supports the S3 API as well allowing users the choice of either protocol when configuring OCI Object Storage as a backend for AIStore. To assist in making the choice of backend protocols used to reach OCI Object Storage, this post provides some performance insights.| AIStore
Split-brain is inevitable. The way it approaches varies greatly but there are telltale signs that, in hindsight, you wish you’d taken more seriously.| AIStore
Oracle Cloud Infrastructure (“OCI”) has been supported via OCI’s Amazon S3 Compatibility API for quite some time. As such, AIStore’s support for the S3 protocol has enabled OCI’s Object Storage to be utilized as a backend. But OCI also provides a more optimized API. AIStore has now added support for using OCI Object Storage as a backend via this OCI-native protocol.| AIStore
AI training workloads primarily read data, and lots of it. So naturally, for most of our previous blogs and benchmarks we’ve focused heavily on GET performance. However, for any storage solution, the ability to add and modify data - and do it reliably and efficiently - is also critically important. In this article we’ll look at some performance benchmarks and options for getting data into AIStore.| AIStore
Enhancing ObjectFile Performance with Zero-Copy Techniques| AIStore
AIStore is a lightweight object storage system with the capability to linearly scale-out with each added storage node and a special focus on petascale deep learning. See more at: github.com/NVIDIA/aistore| AIStore
AIStore is a lightweight object storage system with the capability to linearly scale-out with each added storage node and a special focus on petascale deep learning. See more at: github.com/NVIDIA/aistore| AIStore