AI song recognition relies on two kinds of fingerprint that do different jobs. Audio fingerprinting identifies an exact recording by matching a compact signature against a catalogue. Melody matching works on a hum, a whistle or a sung line with no lyrics, and identifies the tune rather than the recording. Landmark fingerprints are tied to one specific recording and are not built for live versions or pitch-shifted or time-stretched audio. In a patented rights-management design, melody fingerprints identify the composition in covers and live performances. Song identification is also a separate problem from the other things sold as “AI in music”. Our AI in music, audio and sound production hub covers the other problems, and the practical map of music AI problems explains why composing, voice synthesis and stem separation need different engineering.
From spectrogram peaks to a matched recording
The method Avery Wang described for Shazam is the canonical design. It starts with a spectrogram and keeps only local energy peaks: points louder than everything around them, chosen so they cover the recording fairly evenly. The reasoning is that the highest-amplitude peaks are the most likely to survive distortions such as noise and reverb. Wang calls the result a “constellation map”.
To raise the entropy of what it searches on, the system pairs points instead. Each anchor point is paired with the points in its target zone, and each pair becomes a hash built from two frequencies and the time gap between them, small enough to pack into a 32-bit integer. Wang’s slides say the fan-out multiplies the number of tokens but is compensated by a nearly million-fold gain in speed and specificity. There is a cost. If a single peak survives degradation with probability p, a pair survives with roughly p².
Matching depends on time alignment, not just hash counts. Each hash found in the query is looked up in the database, and the pair of time offsets (in the query and in the stored track) is recorded per candidate track. A genuine match shows up as a cluster of offset pairs along a diagonal line, and the match score is the size of the histogram peak that cluster produces. Wang reports that a heavily corrupted 15-second sample can still give a statistically significant match when only about 1-2% of its hash tokens survive. In the same paper, search over a database of about 20,000 tracks on a PC took roughly 5 to 500 milliseconds depending on settings, and under 10 milliseconds for radio-quality audio.
Peak-pair hashing is not the only design. The Philips algorithm splits audio into frames, computes energy across frequency sub-bands and hashes the energy differences into sub-fingerprints. The 2024 multimedia fingerprinting survey counts it among the most reliable extractors, but notes that it struggles in high noise and with large linear speed changes. The SAMAF paper describes a two-stage search: a coarse lookup through an index of sub-fingerprints, then a heavier fine comparison (Hamming distance on the full fingerprint) for songs with many matches.
The survey gives about 3 seconds as the minimum clip for Philips-style retrieval. The open-source Dejavu system is reported at 60% accuracy from one second of input and 100% at five seconds or more.
How can AI identify a song from humming when there are no lyrics?
A hum lacks the lyrics, background vocals and instruments of the studio recording, and its pitch, key or tempo often varies. Humming recognition therefore keeps the melody and discards accompaniment, timbre and tone. The early query-by-humming system from Ghias and colleagues at Cornell pitch-tracked the hum and turned each note into one of three letters: up, down or the same as the previous note. Songs in the database were converted to the same three-letter strings and compared with fuzzy matching, which allowed for the mistakes people make when they hum. The authors found that sequences of 10 to 12 pitch transitions were enough to tell apart 90% of the songs in their collection. Because matching was approximate, a query could match any part of the melody, not just the opening.
Modern systems learn the melody representation with machine learning. Google’s Hum to Search produces an embedding of the melody directly from a spectrogram, so a hum can be matched to the original polyphonic recording without a hand-made MIDI or hummed version of each track. The network is trained on pairs of audio: two clips with the same melody should land close together even when their accompaniment and voices differ, and clips with different melodies should land far apart. Google’s product blog describes the result as a number-based sequence with the instruments, timbre and tone removed. To get enough hummed training data, the team generated synthetic hums with its SPICE pitch-extraction model and later replaced the simple tone generator with a neural network that produces audio resembling a real hummed or whistled tune.
One challenge is variation from the person humming. Google’s researchers note that pitch, key, tempo and rhythm can all drift a little or a lot from the original, by mistake or on purpose. We would treat tolerance to that drift, rather than raw catalogue size, as the main design constraint for a hum-to-search feature.
Can Google AI identify a song, and is hum to search just another Shazam?
Yes, and Google uses both kinds of recognition in separate products. Now Playing, launched on the Pixel 2 in 2017, uses deep neural networks to recognise music at low power on the device itself. It is a fingerprint system, but a neural one. A convolutional network turns an eight-second window into a sequence of embeddings (seven two-second clips at one-second intervals) and searches an on-device database of tens of thousands of songs. Using a sequence matters: any single embedding produces too many false positives, but the correct song matches every embedding while false matches only come close to one or two. A later version widened each embedding from 96 to 128 dimensions and computed one every 0.5 seconds instead of every second.
Hum to Search, launched in October 2020, answers a different question. Google says it can now recognise songs “without the lyrics or original song”, and that the melody itself works like the song’s fingerprint. A violin trade outlet summarised it as turning about 10 seconds of humming into a sequence of numbers that is compared against recorded music. Google reports high accuracy on a database of over half a million songs.
Google itself calls the melody sequence a “fingerprint”, but it is a melody representation trained to ignore timbre and accompaniment, not a recording-specific fingerprint.
Four ways an AI song finder can work
| Approach | What it identifies | Input it needs | What it survives | What breaks it |
|---|---|---|---|---|
| Landmark fingerprint (Shazam-style) | One exact recording | A few seconds of the recording itself | Noise, voices, reverb, codec compression | Live versions, covers, pitch shifts |
| Neural fingerprint (Now Playing) | One recording, using a sequence of embeddings | An eight-second window | Not stated in Google’s account | Very quiet music, very noisy rooms |
| Melody embedding (Hum to Search) | The tune, across hummed, sung, whistled and studio versions | Humming, whistling or singing | Changes in timbre and accompaniment, plus noise and reverb | Pitch, key or tempo drift in the hum (a challenge Google names) |
| Pitch- and tempo-invariant fingerprint (Panako) | A recording that has been sped up or pitched | A short fragment | Pitch shift, time stretch, GSM compression | Time changes above 8%, chorus effects |
Why song recognition fails on covers, live versions and noisy recordings
Fingerprinting fails on covers and live performances because it is version-specific. Wang states that his algorithm is not expected to generalise to live recordings, and that it is very sensitive to which version of a track was sampled, to the point of picking the right studio take from several the human ear cannot distinguish. That precision serves exact identification but works against cover matching. A cover can differ from the original in timbre, tempo, structure, key, arrangement and even the language of the vocals.
If a band plays twice as fast, consecutive frames of the cover map onto repeated frames of the original, so frame sequences cannot be compared directly. Shazam-style fingerprints hold up against most distortions but are sensitive to pitch shifting. Rafii’s live-version work at Northwestern lists the same causes: tempo, key and instrumentation. That system was tested against 389 studio tracks with live-album queries of 6 and 9 seconds.
There are several mitigations, and they address different failures:
- Change the time-frequency representation. A Constant-Q transform has log-frequency bins aligned with musical notes, so a pitch shift becomes a vertical shift of the image. Rafii adds a Hough transform that tries different line angles to absorb tempo deviations.
- Hash invariant quantities. Panako groups peaks in threes, stores time ratios rather than absolute gaps, and keeps only frequency differences, so its fingerprints survive time stretching and pitch shifting. Its performance drops once time modification goes beyond 8%, and it struggles with chorus effects, very low-energy or evenly spread spectra, and highly repetitive music.
- Build for covers, not recordings. Cover identification systems estimate the beat and aggregate features per beat for tempo invariance, align sequences with dynamic programming, and summarise the most repeated sections to cope with changed structure. The Serrà review concludes that a well-built system needs tonal, temporal and structural invariance together.
Low-end microphones, heavy noise, lossy compression and variable latency can remove much of the time-local structure fingerprinting depends on, and the 2025 LFCS paper notes that systems like Shazam were not designed for extremely fragmented input. Even Google says Now Playing does not always match very quiet music or very noisy environments. One response, proposed in the LFCS paper, is to leave the first-stage fingerprint search alone and re-rank its shortlist. The LFCS authors report that their method improved retrieval rank on heavily degraded queries when re-ranking candidates returned by Audfprint, especially where the base system’s own confidence was low. They place it in the secondary phase, where tens or hundreds of very similar candidates remain. We find this two-stage split a sound architecture: a cheap, robust index narrows the field, and a more expensive model is spent only on the hard cases.
Catalogue search, licensing and rights tracking
For rights work, decide first whether you need to identify the recording or the composition, because the two use different fingerprints. A patent on media rights management separates the composition from its recordings. In the patent, audio fingerprints distinguish between different recordings of the same composition. A melody fingerprint represents the sequence of pitch intervals between dominant tones and is built to stay the same across key, instrumentation, interpretation and recording artefacts.
In the patented flow, the system first tries an audio fingerprint match. A hit supports a claim for both the recording owner (typically the label) and the composition owner (typically the publisher). If no audio match is found, it computes a melody fingerprint. A melody match supports a claim for the composition only, since no particular recording has been identified. The patent also describes combining the two, where a strong melody match raises the confidence of a weak audio match to the same reference. This is how a known composition can be detected in a cover or live recording nobody has catalogued.
Broadcast and royalty monitoring is another use. BMAT, a vendor in this market, presents its fingerprinting as detecting music usage from radio jingles to TV commercials so that royalties reach the right rights holders. Our page on AI engineering for media, broadcast and telecom covers building recognition and monitoring systems for these settings. Wang’s original slides also list exact-recording identification for rights management as an application.
How do music producers, content creators and marketers use melody identification?
Radio stations sometimes play songs slightly faster to fit a time slot, and DJ sets involve almost continuous pitch shifting and time stretching. Content recognition systems at media hosting services identify recordings and compositions inside user uploads, including covers and live footage. In practice this means a creator who posts a cover can still be matched to a composition owner, even when no recording-level fingerprint matches.
