Shellcode Analysis from Raw Bytes

Shellcode is a small, position-independent payload that runs without a loader, ELF header, or sections — which means a hex/base64 blob of it defeats file, looks like “gibberish” when ASCII-decoded, and must be disassembled to be understood. Recognizing and triaging these blobs is a recurring CTF and malware-analysis task (challenge coins, exploit attachments, packet captures). The reliable workflow is extract → decode → disassemble → identify syscalls → (optionally) emulate.

Recognizing a blob as shellcode

file guesses by magic bytes at fixed offsets; raw shellcode has none, so it is routinely mislabeled — a common false positive is DOS executable (COM), because .COM files are headerless raw x86 and anything matches. Tell-tale shellcode signs instead:

• A base64/hex blob whose decoded form is mostly non-printable but ends in a readable ASCII string.
• A 0f 05 byte pair (x86-64 syscall) or cd 80 (x86 int 0x80) somewhere in the middle.
• A JMP-CALL-POP prologue: a short forward jmp (eb xx), a pop of a string register (e.g. 5e = pop rsi) near the start, and a backward call (e8 <neg rel32>) just before the embedded string.
• No valid file signature at offset 0 despite file claiming a format.

The JMP-CALL-POP pattern

Shellcode can’t hard-code the address of its own strings (position independence). The standard trick:

1. jmp short end — skip forward past the code to a call instruction sitting just before the string.
2. call start — pushes the return address (the string’s address) onto the stack and jumps back.
3. pop rsi — grabs that string address into an argument register.

A minimal x86-64 write(1, msg, len); exit(0) payload decodes to:

eb 1e                jmp  short get_string      ; skip to the call
5e                   pop  rsi                    ; rsi = &msg
48 31 c0             xor  rax, rax
b0 01                mov  al, 1                  ; __NR_write (x86-64 = 1)
48 89 c7             mov  rdi, rax               ; fd = 1 (stdout)
48 89 fa             mov  rdx, rdi
48 83 c2 15          add  rdx, 0x15              ; count = 21
0f 05                syscall                     ; write(1, msg, 21)
48 31 c0             xor  rax, rax
48 83 c0 3c          add  rax, 0x3c              ; 60 = __NR_exit
48 31 ff             xor  rdi, rdi               ; status = 0
0f 05                syscall                     ; exit(0)
e8 dd ff ff ff       call -0x23                  ; back to pop rsi, pushes &msg
; ...then the ASCII string bytes, e.g. "We have trust issues!\n"

Key bytes: eb = short jump, 5e = pop rsi, 48 31 c0 = xor rax,rax (null-free zeroing), b0 01 = mov al,1 (sys_write), b0 3c = mov al,60 (sys_exit), 0f 05 = syscall, e8 with a negative rel32 = the call-back. Null bytes are avoided throughout because a \x00 would truncate the payload in memory-copy contexts.

Disassembly and syscall identification

Because raw shellcode has no headers, you must tell the disassembler the architecture and bit-width — disassembling 64-bit code as 32-bit yields convincing but wrong instructions:

objdump -D -b binary -m i386:x86-64 -M intel sc.bin   # 64-bit
ndisasm -b 64 sc.bin                                  # NASM disassembler
r2 -a x86 -b 64 sc.bin                                # radare2, then `pd`

Identify the payload’s intent by its syscalls, which are the only way user-space code touches the kernel. On x86-64: syscall number in rax, args in rdi, rsi, rdx, r10, r8, r9. High-signal numbers (x86-64, x86 in parens): write 1 (4), execve 59 (11), exit 60 (1), dup2 33 (63), socket/connect/bind/listen/accept 41–43. A socket → connect → dup2 → execve chain is a reverse shell; bind → listen → accept is a bind shell; write + exit with an embedded string is a “print a message” proof/CTF payload. Confirm numbers against /usr/include/asm/unistd_64.h or ausyscall --dump.

For safe dynamic confirmation, emulate rather than execute: scdbg/libemu (x86), Mandiant Speakeasy (Windows user+kernel), or step through in gdb inside a throwaway VM — never run unknown shellcode on a host you care about.

Where this shows up

• CTF / challenge artifacts — puzzle coins, badges, and props frequently embed a tiny write-and-exit payload whose printed string is the flag/answer (the “gibberish” prefix around a readable base64 tail is the giveaway).
• Exploit attachments & memory dumps — the payload after a NOP sled or decoder stub.
• msfvenom output — e.g. linux/x64/exec CMD=/bin/sh is a ~47-byte JMP-CALL-POP execve("/bin/sh -c ...") payload of exactly this shape (see msfvenom).

Sources

• Shellcode Analysis for Security Researchers
• Linux Syscalls for Exploit Development
• 2018 — x64 Linux Metasploit execve /bin/sh Shellcode Analysis
• Wikipedia: List of File Signatures

Related: msfvenom, gdb, shell-stabilization