"hex" vs. "base64" encoding when enciphering

Question

Examining different implementations of encryption / decryption with cipher on Node.js, I paid attention that some implementations use hex encoding:

let encrypted = cipher.update(value, "utf8", "hex");

while other prefer base64:

let encrypted = cipher.update(value, "utf8", "base64");

Questions:

1. Does encoding choice play a significant role in encryption for a cipher?
2. From security point of view, can the wrong encoding choice affect the outcome?

Answer 1

Does encoding choice play a significant role in encryption for a cipher?

No. Ciphers operate on bit strings. It is far more important to focus on what kind of ‘cipher’ it is and what security goals it provides: you generally want an authenticated cipher like NaCl crypto_secretbox_xsalsa20poly1305 or AES-GCM, and you definitely don't want to reach for the letters A-E-S directly yourself. Focus on security contracts.

From security point of view, can the wrong encoding choice affect the outcome?

Not as far as the cryptography is concerned, but complicated encoding methods can pose security problems of their own:

• If you apply a base64 encoder/decoder to the plaintext rather than ciphertext you might leak information about the plaintext via timing side channels in a naive array lookup for hexadecimal digits.
• If you apply compression to the plaintext you might leak information about it via the length of the ciphertext as a side channel.
• If your encoding method is buggy like an ASN.1 parser your application might barf its memory contents out like a bleeding-heart OpenSSL.