System calls

When we want to understand what a program is doing or how it's running under the hood.

Programs make system calls to request services from the kernel—opening files, allocating memory, creating processes, and more.

We can use tracking , by using strace which let us observe every system call a program makes:

strace (Linux) and dtruss (macOS)

# Trace all system calls
strace ./my_program

# Trace only file-related calls
strace -e trace=file ./my_program

# Follow child processes (important for programs that start other programs)
strace -f ./my_program

# Trace a running process
strace -p <PID>

# Show timing information
strace -T ./my_program

Awesome strace guide

Network Debugging

We can also inspect the network through tcpdump

# Capture packets on port 80
sudo tcpdump -i any port 80

# Capture and save to file for Wireshark analysis
sudo tcpdump -i any -w capture.pcap

For HTTPS traffic, the encryption makes tcpdump less useful. Tools like mitmproxy can act as an intercepting proxy to inspect encrypted traffic. Browser developer tools (Network tab) are often the easiest way to debug HTTPS requests from web applications—they show decrypted request/response data, headers, and timing.

System Sanitizers:

One approach to finding memory bugs is to use sanitizers, which are compiler features that instrument your code to detect errors at runtime. For example, the widely used AddressSanitizer (ASan) detects:

• Buffer overflows (stack, heap, and global)
• Use-after-free
• Use-after-return
• Memory leaks

There is a tool called valgrind , which is an interpreter. It pretends to be the CPU, runs the program, and executes its instructions.

valgrind --leak-check=full ./my_program

Record-Replay (rr), a Linux based tool.

It's a powerful tool for Linux that records program execution and allows deterministic replay with full debugging capabilities. It works with Python, especially Cpython

# Record a program execution
rr record ./my_program

Find more info here

Timing

There is a tool every shell has access to, called time. The simplest way to measure performance is to time things. In many scenarios it can be enough to just print the time it took your code between two points.

• Real - Wall clock time from start to finish, including time spent waiting
• user time (How much the CPU spent running user code)
• How much the CPU spent into the kernel

Resource monitoring

Sometimes the first step towards analyzing the performance of your program is to understand what its actual resource consumption is. Programs often run slowly when they are resource constrained.

• General Monitoring: htop is an improved version of top that presents various statistics for currently running processes.
• I/O Operations: iotop displays live I/O usage information.
• Open Files: lsof lists file information about files opened by processes. Useful for checking which process has opened a specific file

CPU Profiler

Most of the time when people refer to profilers they mean CPU profilers There are two main types:

• Tracing profilers keep a record of every function call your program makes
• Sampling profilers probe your program periodically (commonly every millisecond) and record the program’s stack

Sampling profiler

perf is the standard Linux profiler. It can profile any program without recompilation:
perf stat gives you a quick overview of where time is spent:

$ perf stat ./slow_program

 Performance counter stats for './slow_program':

         3,210.45 msec task-clock                #    0.998 CPUs utilized
               12      context-switches          #    3.738 /sec
                0      cpu-migrations            #    0.000 /sec
              156      page-faults               #   48.587 /sec
   12,345,678,901      cycles                    #    3.845 GHz
    9,876,543,210      instructions              #    0.80  insn per cycle
    1,234,567,890      branches                  #  384.532 M/sec
       12,345,678      branch-misses             #    1.00% of all branches