How does antivirus detect malware?

1. Signatures: matching known malware

A signature is a fingerprint of a known piece of malware, and signature scanning checks every file against a database of them.

The simplest signature is a hash: a fixed-length fingerprint calculated from every byte of a file. If a file's SHA-256 hash matches one on the list of known malware, it is that malware. The weakness shows when you change a single character. Try it:

The same thing in Python, for anyone who wants to run it (this is the output it printed for us):

import hashlib

for text in ["invoice.pdf.exe version 1", "invoice.pdf.exe version 2"]:
    digest = hashlib.sha256(text.encode()).hexdigest()
    print(f"{text!r:30} {digest}")

'invoice.pdf.exe version 1'    803afddef09393dce7fddc36c279c6125d2ada6448ffb51276a37d9c8c3aca88
'invoice.pdf.exe version 2'    aef39039781ace75e16e6ce2df87bb695802391de335ce86475dc2abd56c8f18

So attackers repack their malware for every campaign, and a hash-only signature misses each new copy. Modern signatures match patterns instead: sequences of bytes or strings that survive repacking. Security teams write these as YARA rules, like this simplified one:

rule Suspicious_Encoded_PowerShell
{
    strings:
        $a = "-EncodedCommand" nocase
        $b = "FromBase64String" nocase
    condition:
        all of them
}

It flags any file that contains both strings, a combination common in malicious scripts and rare in normal ones. Catches: known malware, fast and with few false alarms. Misses: anything new.

2. Heuristics: suspicious features

Heuristic scanning scores unknown files on features that malware tends to have, without needing to have seen that exact file before.

Examples: a program packed so its code cannot be read, an Office document with macros that start automatically, a file named invoice.pdf.exe, or code that imports functions for injecting into other processes. No single feature proves anything; enough of them together does. Catches: new variants of known families. Misses: malware written to look ordinary, and it raises more false alarms.

3. Behaviour monitoring: what a program does

Behaviour monitoring watches programs as they run and stops them when their actions look like an attack, whatever the file looked like.

A program that rewrites hundreds of documents a second is acting like ransomware. Word starting PowerShell, which then downloads a file, is acting like a malicious macro. Behaviour blocking stops these in progress, and is how products catch malware that has never been seen before. On Windows, the Antimalware Scan Interface (AMSI) also lets security software inspect scripts after they have been decoded, just before they run.

4. Sandboxing: running it somewhere safe first

A sandbox runs a suspicious file in an isolated virtual machine and records what it does before it is allowed anywhere near a real system.

Email gateways and cloud scanners sandbox attachments routinely. The catch is that malware knows: some checks whether it is in a virtual machine, or waits an hour before doing anything, and behaves until the sandbox gives up.

5. Machine learning and cloud reputation

Machine-learning models, trained on millions of known good and bad files, give each new file a probability of being malicious in milliseconds.

Alongside them, cloud reputation asks a simpler question: how many other computers have seen this file, and for how long? A program that appeared an hour ago on three machines worldwide gets far more scrutiny than one installed on millions.

The five methods side by side

When all five miss, organisations rely on EDR, which records what happened so a person can investigate. And when a file needs a closer look, that is malware analysis. To start from the beginning, see what malware is.