Question

When scaling a distributed storage system, which technique minimizes the amount of data that must be remapped when the total number of nodes in the cluster changes?

Accepted Answer

The technique that minimizes data remapping when scaling a distributed storage system is consistent hashing. In a traditional hashing system, the location of data is determined by a formula such as hash(key) modulo n, where n is the number of nodes. If n changes, the result of the modulo operation changes for almost every key, requiring nearly all data to be moved to different nodes. Consistent hashing solves this by mapping both the data keys and the storage nodes onto a logical circular space called a hash ring. Each key is stored on the first node encountered when moving clockwise around the ring from the key&#x27;s position. When a new node is added or an existing node is removed, only the data keys located in the immediate vicinity of that node are remapped. Because the vast majority of keys remain mapped to their original nodes, the amount of data movement is reduced to a small fraction of the total dataset, specifically proportional to the number of keys divided by the number of nodes. To ensure that data is distributed evenly and to prevent hotspots where one node handles significantly more traffic than others, consistent hashing often uses virtual nodes. A virtual node is a single physical node represented by multiple points on the hash ring, which creates a more uniform distribution of data across all available hardware.

Home → All Courses → Engineering and Technology Courses → Distributed Systems Engineering → Flashcard

When scaling a distributed storage system, which technique minimizes the amount of data that must be remapped when the total number of nodes in the cluster changes?