Cryptopals challenge - not understanding the padding details

Question

I am working on some tasks in Crypto, specifically on this small task: https://cryptopals.com/sets/2/challenges/9, which I aim to implement in Python.

The point is to add padding for a certain number of bytes, and the value of each byte is the number of bytes added. The example given is "YELLOW SUBMARINE\x04\x04\x04\x04".

I have solved this task I think, but the output is non-displayable. Here is one clean solution I found online

def pad_pkcs7(buffer, block_size):
    if len(buffer) % block_size:
        padding = (len(buffer) / block_size + 1) * block_size - len(buffer)
    else:
        padding = 0
    # Padding size must be less than a byte
    assert 0 <= padding <= 255
    new_buffer = bytearray()
    new_buffer[:] = buffer
    new_buffer += bytearray([chr(padding)] * padding)
    return new_buffer

buffer = bytearray("YELLOW SUBMARINEX")
print str(pad_pkcs7(buffer, 20))

When I run this I get the same result as with my own solution: YELLOW SUBMARINE[?][?][?][?] (the []s are actually boxes with question marks in them)

My questions:

Why is the padding hex encoded in the first place? Why not YELLOW SUBMARINE4444? Odd that it is part ASCII and part hex
Did this solution actually solve the problem? Should not the hex-encoded version of 4 be quite possible to display as 4, 04, \x04?

Most padding should, and PCKS5/7 padding in particular is specified to, always be added, even when data is already a multiple of the block size. This is so you can remove the padding without sometimes removing data, for absolutely any data. — dave_thompson_085, Aug 02 '18 at 02:34

score 5 · Accepted Answer · answered Aug 01 '18 at 19:19

This is expected behavior when you try to display the byte 0x04, which is not a printable ASCII character.

Padding schemes like pkcs7 operate on byte sequences, not ASCII strings. In general, ciphers and padding routines only care about byte sequences; whether a specific byte corresponds to a printable ASCII character is not important.

The routine uses the byte 0x04 as opposed to the ASCII character 4 (0x34) because it doesn't know or care about ASCII. Also, using decimal digits 0-9 would mean padding can be at most 10 bytes long.

When viewing byte strings in python, you should avoid casting to a string, just keep it as a byte array instead. For debugging, you can do something like

 print pad_pkcs7(buffer, 20).__repr__()

For examples if you encrypt video, audio, images, etc, it's not ASCII characters, or any other character-code either. Ditto almost any compressed data, and most archived/dumped data. Ditto object or executable code. Ditto most data files. Ditto .... — dave_thompson_085, Aug 02 '18 at 02:38

Cryptopals challenge - not understanding the padding details

1 Answers1