RSA-OAEP Encryption Explained

RSA-OAEP: Secure Asymmetric Encryption

RSA-OAEP (RSA with Optimal Asymmetric Encryption Padding) is the recommended padding scheme for RSA encryption. Raw "textbook RSA" is deterministic and vulnerable to several attacks — OAEP adds randomized padding that makes RSA encryption semantically secure.

Why Raw RSA Is Insecure

Textbook RSA computes ciphertext = message^e mod n. This has critical weaknesses:

1. Deterministic — The same plaintext always produces the same ciphertext, allowing an attacker to build a dictionary
2. Malleable — An attacker can manipulate ciphertexts to produce related plaintexts
3. Small message attacks — If the message is small, the modular exponentiation may not wrap around, making the encryption trivially reversible

How OAEP Works

OAEP applies a two-round Feistel network to the message before RSA encryption:

1. Pad message M to block size with zero bytes
2. Generate random seed r
3. maskedDB = DB XOR MGF(seed)     // DB = padded message
4. maskedSeed = seed XOR MGF(maskedDB)
5. EM = 0x00 || maskedSeed || maskedDB
6. Ciphertext = RSA_Encrypt(EM)

MGF (Mask Generation Function) is typically MGF1 based on SHA-256. The random seed ensures that encrypting the same plaintext twice produces different ciphertexts.

Key Size and Message Length

RSA key sizes determine the maximum message length for OAEP:

RSA Key Size	Max Plaintext (SHA-256)
2048-bit	190 bytes
3072-bit	318 bytes
4096-bit	446 bytes

The formula is: keySize/8 - 2*hashSize - 2. This severe length limitation means RSA-OAEP is used for encrypting symmetric keys, not bulk data.

Hybrid Encryption Pattern

In practice, RSA-OAEP is combined with a symmetric cipher:

1. Generate random AES key (32 bytes)
2. Encrypt data with AES-GCM using the AES key
3. Encrypt the AES key with RSA-OAEP using the recipient's public key
4. Send: encrypted AES key + IV + AES-GCM ciphertext + auth tag

Web Crypto API Example

const keyPair = await crypto.subtle.generateKey(
  { name: "RSA-OAEP", modulusLength: 2048,
    publicExponent: new Uint8Array([1, 0, 1]),
    hash: "SHA-256" },
  true, ["encrypt", "decrypt"]
);

const ciphertext = await crypto.subtle.encrypt(
  { name: "RSA-OAEP" },
  keyPair.publicKey,
  plaintext
);

OAEP vs PKCS#1 v1.5

Older RSA implementations use PKCS#1 v1.5 padding, which is vulnerable to Bleichenbacher's attack — a chosen-ciphertext attack that can decrypt RSA ciphertexts through millions of oracle queries. OAEP was designed specifically to resist this class of attack and should always be preferred.