Introduction to the Challenge

The challenge was a zip file containing a testdll.exe binary and loader.dll.

When executed, it ask for the key. And showed this message after incorrect key input.

Can you unpack this?.
Format: re{flag}
Password for unzip: secplayground

Anti-Debugging Techniques

Runtime software protection

Application close itself when open x64dbg

Open testdll.exe with x64dbg.

Click "Execute till return." The debugger will try to continue with the application flow.

After clicking for a while, I saw this error message, and testdll.exe closed itself.

It looks like the application was protected by WinLicense.

Flag Retrieval

Unpack DLL with unlicense

Use this tool to unpack the DLL file.

https://github.com/ergrelet/unlicense

Copy the loader.dll to the same directory as the downloaded unlicense.exe. Then, execute the command below.

.\unlicense.exe loader.dll

The unlicense tool will read and output an unprotected DLL. Replace loader.dll with unpacked_loader.dll.

x64dbg: Dynamic analysis

Now repeat the steps from earlier. This time, the application will not close itself.

Open Symbols tab. Click loader.dll on the left side, checkFlag Symbol will shown on the right side.

Right-click on checkFlag, then click Follow in Disassembler.

x64dbg will navigate back to the location of the checkFlag function.