A novel program code obfuscation approach involving the x86 mode switching is proposed in the paper. The details and existing applications of x86 mode switching are reviewed, as well as the possible consequences of using this switching to the reverse engineering tools. Based on this approach, a few specific methods are proposed and evaluated against the most popular reverse engineering tools of various purposes, including disassemblers, decompilers, binary instrumentation and symbolic execution tools. A method of seamless integration of these machine code level obfuscations to the C, C++ and possibly other compilers is also proposed.
Download file
Counter downloads: 8
- Title Using x86 mode switching for program code protection
- Headline Using x86 mode switching for program code protection
- Publesher
Tomsk State University - Issue Prikladnaya Diskretnaya Matematika - Applied Discrete Mathematics 61
- Date:
- DOI 10.17223/20710410/61/6
Keywords
code protection, reverse engineering, obfuscation, x86 mode switching, disassembly, decompilation, symbolic executionAuthors
References
Barak B., Goldreich O., Impagliazzo R., et al. On the (im)possibility of obfuscating programs. LNCS, 2001, vol. 2139, pp. 1-18.
https://github.com/SamboyCoding/Cpp2IL - Cpp2IL: Work-in-progress tool to reverse unity's IL2CPP toolchain, 2023.
Wang C., Davidson J., Hill J., and Knight J. Protection of software-based survivability mechanisms. Proc.Intern. Conf. Dependable Svst.Networks, Goteborg, 2001, pp. 193-202.
Collberg C., Thomborson C., and Low D. Manufacturing cheap, resilient, and stealthy opaque constructs. Proc. 25th ACM SIGPLAN-SIGACT Svmp. POPL'98, San Diego, California, USA, 1998, pp. 184-196.
Collberg C., Thomborson C., and Low D. Breaking abstractions and unstructured data structures. Proc.Intern. Conf.Computer Languages, Chicago, IL, USA, 1998, pp. 28-38.
Warren H. S. Hacker's Delight, Second Ed. Addison-Weslev, 2012. 512 p.
Junod P., Rinaldini J., Wehrli J., and Michielin J. Obfuscator-LLVM - software protection for the masses. IEEE/ACM 1st Intern. Workshop Software Protection, Florence, Italy, 2015, pp. 3-9.
https://tigress.wtf - the tigress c obfuscator, 2023.
Ugarte-Pedrero X., Balzarotti D., Santos I., and Bringas P. G. SoK: deep packer inspection: A longitudinal study of the complexity of run-time packers. EEE Svmp. Security and Privacy, San Jose, CA, USA, 2015, pp. 659-673.
Jamthagen C., LantzP., and Hell M. A new instruction overlapping technique for antidisassembly and obfuscation of x86 binaries. Workshop Anti-malware Testing Research, Montreal, QC, Canada, 2013, pp. 1-9.
Cohen F.B. Operating system protection through program evolution.Computers and Security, 1993, vol. 12, no. 6, pp. 565-584.
Lebedev R. К. and Koryakin I. A. Primenenie rasshireniv arkhitekturv x86 v zashchite programmnogo koda [Application of x86 extensions for code protection]. Prikladnava diskretnava matematika. Prilozhenie, 2021, no. 14, pp. 138-140. (in Russian).
https://wbenny.github.io/2018/11/04/wow64-internals.html - WoW64 internals,2018.
https://community.osr.com/discussion/246643/understanding-win-7-x64-gdt-ldt, 2013.
https://github.com/torvalds/linux/blob/master/arch/x86/kernel/cpu/common.c - Linux Kernel (GitHub), 2023.
Intel 64 and IA-32 Architectures Software Developer's Manual Combined Volumes: 1, 2A, 2B, 2C, 2D, ЗА, ЗВ, ЗС, 3D, and 4. December 2022. 5060 р.
https://github.com/NationalSecurityAgency/ghidra/issues/510 - Allow Different Instruction Sets for Different Memory Sections (Ghidra, GitHub), 2023.
Assembly language is too high level. DEF CON 25, 2017. https://media.defcon.org/DEFC0N25/DEFC0N25presentations/DEFC0N25-XIogicX-Assembly-Language-Is-Too-High-Level.pdf.
Collberg C., Thomborson C., and Low D. A Taxonomy of Obfuscating Transformations. Technical Report. Department of Computer Science, The University of Auckland, 1997, no.148. https://researchspace.auckland.ac.nz/bitstream/handle/2292/3491/TR148.pdf.
https://hex-rays.com/ida-pro - Hex Rays - State-of-the-art binary code analysis solutions, 2023.
https://github.com/NationalSecurityAgency/ghidra - Ghidra Software Reverse Engineering Framework (GitHub), 2023.
Nethercote N., and Seward J. Valgrind: a framework for heavyweight dynamic binary instrumentation. SIGPLAN Not., 2007, vol. 42, no. 6, pp. 89-100.
Shoshitaishvili Y., Wang R., Sails C., et al. SOK: (State of) The art of war: Offensive techniques in binary analysis. IEEE Svmp. Security Privacy (SP), San Jose, CA, USA, 2016, pp.138-157.
https://github.com/angr/pyvex/commit/46049al4985a8d78c6679d75f103540b94c22bc5 - Add generalized aam and aad instructions for x86, angr/pyvex (GitHub), 2022.
Using x86 mode switching for program code protection | Prikladnaya Diskretnaya Matematika - Applied Discrete Mathematics. 2023. № 61. DOI: 10.17223/20710410/61/6
Download full-text version
Counter downloads: 217