In this post, we demonstrate how to implement Fully Sharded Data Parallel (FSDP) fine-tuning of the dolly-v2-7b model using Amazon ECS. The solution uses a Ray cluster running on ECS with two services (head and worker) connected to Amazon S3, enabling efficient distributed training across multiple GPUs while abstracting away container orchestration complexities.
