We were provided with a .tap file, which was a casette tape data, in this case from ZX Spectrum. After loading this tape into ZX Spectrum 128k emulator (with a classic ZX Spectrum 48k we would get a nifty QR code saying that it needs 128k to run), we were greeted with a cool demo and an encoded image with password prompt.
After obtaining physical memory dump and reordering it to actually match the memory banks layout as seen in emulator, we started the analysis with an interrupt routine at 0x8000, which handles the input and redrawing. The point of interest was of course the input handling function, which did some simple checks and if the characters were alphanumeric, it stored them to prepared buffer.
RAM:8054 handleInput: ; CODE XREF: RAM:802C ... RAM:8065 cp 30h ; '0' RAM:8067 ret c RAM:8068 cp 3Ah ; ':' RAM:806A jr c, loc_807B RAM:806C cp 41h ; 'A' RAM:806E ret c RAM:806F cp 5Bh ; '[' RAM:8071 jr c, loc_807B ... RAM:807B loc_807B: ; CODE XREF: handleInput+16 RAM:807B ; handleInput+1D RAM:807B ld e, a RAM:807C ld a, (inputSize) RAM:807F cp 0Ah RAM:8081 ret z RAM:8082 inc a RAM:8083 ld (inputSize), a RAM:8086 ld hl, (inputEnd) RAM:8089 ld (hl), e RAM:808A inc hl RAM:808B ld (inputEnd), hl
After following references to inputSize variable, we arrived to a routine which was responsible for decoding of the image. This routine starts with a computation of 16-bit hash from the input copying the result few times over to make it a 128-bit long key.
RAM:6077 demoLoop: ; CODE XREF: RAM:6044p RAM:6077 ld a, (inputSize) RAM:607A ld c, a RAM:607B ld de, 810Bh ; inputBuffer RAM:607E call countHash RAM:6081 ld (word_62F4), hl RAM:6084 ld hl, 62F4h RAM:6087 ld de, 62F6h RAM:608A ld bc, 0Eh RAM:608D ldir
Immediately after key initialization is a loop which iterates through each image line, and decodes it with the key above.
RAM:608F ld hl, 60A9h ; encryptedImage RAM:6092 ld de, 878Dh ; decryptedImage RAM:6095 ld b, 20h ; ' ' ; numberOfLines RAM:6097 RAM:6097 loc_6097: ; CODE XREF: demoLoop+2F ... RAM:609A call processLine ... RAM:60A6 djnz loc_6097
The processLine function clears the destination buffer (one image line), initializes variables with predefined values and calls a stack based decoder, which sample is below.
RAM:61F8 decodeData: ; CODE XREF: processLine+22 RAM:61F8 ld hl, 62E4h RAM:61FB call stack_push_dword_indirect RAM:61FE ld hl, 62E0h RAM:6201 call stack_push_dword_indirect RAM:6204 ld hl, 0 RAM:6207 ld de, 4 RAM:620A call stack_push_dword RAM:620D call stack_shift_dword_left RAM:6210 ld hl, 62E0h RAM:6213 call stack_push_dword_indirect RAM:6216 ld hl, 0 RAM:6219 ld de, 5 RAM:621C call stack_push_dword RAM:621F call stack_shift_dword_right RAM:6222 call stack_xor_dwords
We unfortunately searched for used constants (namely 0x9E3779B9) too late and actually rewrote whole XTEA algorithm to C by hand ( decode.cpp ). Now we just needed to bruteforce through 2^16 possibilities and try to find the correct image. We thankfully noticed that some data in encoded image are repeating so we just tried to compare whether first two dwords match, as they might be some kind of border. This gave us the correct key 0x7DFB.
To display the final image we took the easy way and just set a breakpoint to address 0x6081, which is directly after the call to computeHash function and modified its return value (hl register) to the correct value. After letting emulator decode the image, we've got the solution.