How to encode file

Question

I am trying to decrypt a lua file which has been encrypting using a key. Some background to the lua file. Its from an android I decompiled the APK and found that all the lua files are encrypted.

Answer 1

You seem to have correctly identified the point where the decryption is invoked. It seems like a1 (the first argument of that function) is a this pointer, pointing to some kind of game engine wrapper object. At (a1+40), there is a pointer to the key, and (a1+48) contains the length of the key. The initialization of those members happens somewhere else, likely in the constructor or some initialization function of the engine wrapper object.

Answer 2

xor_key (16 byte): EB 16 0D DE E8 7E 0F 31 5E 98 B8 2D 04 E1 A3 02

// structure of encrypted file ..................
// 2     bytes 74 6A
// 1     bytes 21 | 65 | 7A
// 16    bytes of xor key xxtea
// 4     bytes size of original file
//       bytes total: 23
// other bytes data encrypted ...................

Algorithm decryption file

1. Check 3 bytes header, if match do step 2
2. Read next 16 bytes file_key and xor this with xor_key to get xxtea_key
3. Decrypt data with xxtea xxtea_key to get original data
4. If data[2] == 0x21 || data[2] == 0x7A then uncompress lz4

Example

xor_key   = EB 16 0D DE E8 7E 0F 31 5E 98 B8 2D 04 E1 A3 02
file_key  = 96 55 47 66 46 FF 23 18 0E EA 07 3D D8 55 CC 15 (in image)
xxtea_key = 7D 43 4A B8 AE 81 2C 29 50 72 BF 10 DC B4 6F 17 (xor_key ^ file_key)
data = raw[23:]
xxxtea_decrypt(data, xxtea_key)

source code (Golang)

func decryptData(data []byte, key []byte) ([]byte, bool) {
    // check header 1
    if data[0] != 0x74 && data[1] != 0x6A {
        return nil, false
    }
// check header 2
if data[2] != 0x21 && data[2] != 0x65 && data[2] != 0x7A {
    return nil, true
}

for i := 0; i < len(key); i++ {
    data[i+3] ^= key[i]
}
//fmt.Printf("xxtea_key: % 2X\n", data[3:0x13])
dec := Decrypt(data[0x17:], data[3:0x13])
if data[2] == 0x21 || data[2] == 0x7A {
    unz := make([]byte, binary.LittleEndian.Uint32(dec[:4]))
    _, err := lz4.UncompressBlock(dec[4:], unz)
    return unz, err == nil
}
return dec, true

}

I'll make video RE with IDA remote android debugger to get xor_key when I have time