Using Hashes for Data Integrity

Data integrity verification uses hash functions to detect whether data has been modified, corrupted, or tampered with. By computing a hash before and after an operation (transfer, storage, processing), you can confirm the data is unchanged by comparing the two hash values.

The fundamental principle:

A cryptographic hash function produces a fixed-size fingerprint of arbitrary data. If even one bit of the input changes, the output changes unpredictably. By storing or transmitting the hash separately from the data, a recipient can independently verify integrity: compute the hash of the received data and compare it to the expected hash. If they match, the data is intact with overwhelming probability.

File transfer verification:

When transferring files over a network, bits can be flipped by hardware errors, packets can be corrupted, and transfers can be truncated. Computing SHA-256 of the file before sending and verifying after receiving catches all of these issues. Protocols like SCP and rsync can perform hash verification automatically. For manual verification, publishing a SHA256SUMS file alongside downloads is standard practice.

Database integrity:

Databases can use row-level hashing to detect silent corruption. Compute a hash over critical columns when data is written, and periodically verify that the hash still matches. This catches storage-layer corruption, unauthorized modifications, and replication errors. Some databases (like PostgreSQL) offer page-level checksums as a built-in feature.

Backup verification:

Backup systems should hash files before backup and verify hashes on restore. This ensures that backup media degradation (bit rot) is detected before it causes data loss. ZFS and Btrfs file systems provide block-level checksumming using SHA-256 or similar algorithms. The hash verification catches errors that would be invisible to traditional backup tools.

Distributed systems:

In distributed systems, hash-based integrity verification is essential. Merkle trees (used in Git, blockchains, IPFS, and ZFS) organize data into a tree of hashes where each non-leaf node is the hash of its children. This structure allows efficient verification of any subset of data and enables identifying exactly which pieces have changed. Content-addressable storage systems use the hash of data as its address, making integrity verification automatic: if the data does not match its address (hash), it is corrupt.