Friday, May 15 3:00 PM  EST

Virtual chunks at Earthmover, from theory to production

(An ESIP Series Talk) 

Earthmover Forward Deployed Engineer Tom Nicholas PhD

Register (for the talk / ESIP series) here: https://www.esipfed.org/event/cloud-computing-2026-05-15/

Abstract: Virtual chunks are a core primitive for cloud-native data systems. Object storage's accessibility from anywhere, plus the high cost of moving, renaming, or reformatting objects, encourages referring to data where it already resides. In this talk we will introduce key concepts for virtual chunks one-by-one, and then describe how these concepts map to implementations in systems like Icechunk, VirtualiZarr, and Kerchunk. Although virtual chunks are conventionally used for creating cloud-optimized references to archival data formats, we also describe other use cases for them at Earthmover, as well as important security considerations.