Ecosystem · Services

Hash databases & services

The natural extension of any hash function is asking has anyone seen this digest before? The big public services index file hashes for malware, software, and breached credentials. Each uses one or more of the algorithms in the catalog as its lookup key.

VirusTotal

• What: file-reputation service. Submit a file (or its hash) and get back AV scan results from 70+ engines, behavioral analysis, related samples.
• Hash keys used: MD5, SHA-1, SHA-256, imphash, ssdeep, TLSH, Authentihash. All are looked up against the same internal database.
• API: v3 public API requires an API key; community key is free with a 4-request-per-minute rate.
• Caveats: uploading sensitive files makes them visible to the threat-intel community. Hash-only lookups are safer.
• virustotal.com

NSRL (National Software Reference Library)

• What: NIST’s reference database of hashes for known software. Used in digital forensics to exclude known-good files from analysis.
• Hash keys: MD5, SHA-1 (and SHA-256 in the modernized RDS releases).
• Size: tens of millions of entries covering shipped commercial and open-source software.
• Use: standard step in computer forensics , hash every file on a seized disk, drop the ones whose hashes are in NSRL, focus on the rest.
• NSRL at NIST

Have I Been Pwned (HIBP)

• What: Troy Hunt’s service indexing credentials from public data breaches.
• Hash keys: SHA-1 of password (legacy), NTLM (newer endpoint).
• Privacy trick , the k-anonymity API: send the first 5 hex characters of your password’s SHA-1; HIBP returns every hash that starts with those 5 chars and the count of times each was seen in breaches. You match locally. The server never sees your password (or its full hash).
• Use: wire HIBP into your sign-up flow to refuse known-breached passwords.
• HIBP Pwned Passwords API

Malware Bazaar (abuse.ch)

• What: open repository of malware samples for researchers.
• Hash keys: SHA-256, SHA-1, MD5, imphash, TLSH.
• Distinctive: downloadable samples for authorized researchers; tagged by family / campaign.
• bazaar.abuse.ch

Other useful indexes

• Microsoft / Apple file reputation services , SmartScreen, Notarization. Not public APIs but routinely consulted by their OSes.
• Common Crawl , not a malware DB, but indexes hashes of crawled pages for dedup.
• ImpHash search in VT, a free pivot if you know one Windows malware sample’s imphash.
• Sigma rule hashes , YAML-encoded detection rules sometimes ship known-bad SHA-256 indicators.

Privacy posture, by service

References

• VirusTotal API reference
• NSRL overview
• Have I Been Pwned
• Malware Bazaar
• Fuzzy hashes