AVX-512 Is a Gift to Vector Search, If You Use It Wisely

Vector search spends a surprising amount of time asking one basic question: how far is this candidate from the query?

The math is usually the easy part. The interesting part is persuading a modern CPU to do that math quickly without wasting time hauling unnecessary data around.

This is where SIMD enters the story. SIMD stands for single instruction, multiple data. In plain English, one instruction performs the same operation across several values at once. AVX-512 gives us 512-bit vector registers, which means one register can hold sixteen 32-bit floating-point numbers. Sixteen subtraction problems at once is a pretty nice gift.

Like many gifts, it works best if you read the instructions.

We will look at two small vector-search kernels:

1. Squared L2 distance over dense float vectors, using packed subtraction and fused multiply-add.
2. Asymmetric distance over 8-bit product-quantization codes, using byte widening and indexed gathers.

The first kernel is SIMD on easy mode. The data is contiguous and every lane does exactly the same work. The second is where things get interesting. It scores compressed vectors without reconstructing them, but it has to fetch values from different locations in a lookup table.

The accompanying VectorAmp Labs reference implementation includes scalar, AVX2, and AVX-512 implementations, correctness tests, generated assembly, benchmark tooling, raw CSV output, and environment metadata. It is a clean educational implementation, not production SABLE source code.

Kernel 1: dense squared L2 distance

For a query vector q and a candidate vector x, squared L2 distance is:

d²(q, x) = Σᵢ (qᵢ - xᵢ)²

That means we subtract each pair of coordinates, square each difference, and add everything together.

We use squared distance because taking the square root would not change the ranking. If candidate A has a smaller squared distance than candidate B, it also has a smaller ordinary Euclidean distance. The square root would just make the CPU do extra work for the same ordering. We care about CPU cycles, so we let it skip that part.

A scalar implementation follows the formula directly:

float l2_squared_scalar(
    const float* query,
    const float* candidate,
    std::size_t dimensions) {
  float sum = 0.0f;
  for (std::size_t i = 0; i < dimensions; ++i) {
    const float delta = query[i] - candidate[i];
    sum += delta * delta;
  }
  return sum;
}

There is nothing wrong with this code. It is clear, correct, and easy to test. It also describes sixteen nearly identical pieces of work that AVX-512 can perform together.

The central vectorized loop looks like this:

A quick note on intrinsics: We use C++ intrinsics here because they make the relationship between the algorithm and the AVX-512 instructions easier to follow. Intrinsics compile to native machine instructions, so there is no abstraction layer running inside the hot loop. Handwritten assembly can sometimes improve scheduling or register allocation, but it is not automatically faster. As always, inspect the generated code and measure.

__m512 sum0 = _mm512_setzero_ps();
__m512 sum1 = _mm512_setzero_ps();
__m512 sum2 = _mm512_setzero_ps();
__m512 sum3 = _mm512_setzero_ps();

for (std::size_t i = 0; i + 64 <= dimensions; i += 64) {
  const __m512 d0 = _mm512_sub_ps(
      _mm512_loadu_ps(query + i),
      _mm512_loadu_ps(candidate + i));
  const __m512 d1 = _mm512_sub_ps(
      _mm512_loadu_ps(query + i + 16),
      _mm512_loadu_ps(candidate + i + 16));
  const __m512 d2 = _mm512_sub_ps(
      _mm512_loadu_ps(query + i + 32),
      _mm512_loadu_ps(candidate + i + 32));
  const __m512 d3 = _mm512_sub_ps(
      _mm512_loadu_ps(query + i + 48),
      _mm512_loadu_ps(candidate + i + 48));

  sum0 = _mm512_fmadd_ps(d0, d0, sum0);
  sum1 = _mm512_fmadd_ps(d1, d1, sum1);
  sum2 = _mm512_fmadd_ps(d2, d2, sum2);
  sum3 = _mm512_fmadd_ps(d3, d3, sum3);
}

const __m512 lanes = _mm512_add_ps(
    _mm512_add_ps(sum0, sum1),
    _mm512_add_ps(sum2, sum3));
float sum = _mm512_reduce_add_ps(lanes);

The compiler turns the hot part into instructions similar to these:

vmovups      zmm5, [query + offset]
vsubps       zmm0, zmm5, [candidate + offset]
vfmadd231ps  zmm1, zmm0, zmm0

Here is what is actually happening:

• vmovups loads sixteen 32-bit floating-point values into a 512-bit register.
• vsubps subtracts sixteen candidate coordinates from sixteen query coordinates.
• vfmadd231ps multiplies each difference by itself and adds the results to sixteen running totals.

That last instruction is called fused multiply-add. You will often see it shortened to FMA. Instead of issuing one instruction to multiply and another to add, the processor performs both operations as one instruction. It also rounds once instead of rounding after the multiplication and again after the addition.

So in this kernel, FMA is not mysterious wizardry. It is simply the instruction that turns:

sum += delta * delta;

into sixteen simultaneous versions of the same operation.

The vector loop keeps sixteen lane-local sums and combines them only at the end. That final combination is called a horizontal reduction. Those lanes form sixteen independent chains rather than one scalar chain, but each lane still depends on its own total from the previous iteration. The public benchmark therefore uses four vector accumulators and combines them at the end. That small unroll gives the processor more independent work while an FMA is still making its way through the machinery. The code is a little busier, but the CPU gets fewer opportunities to stare into space.

⤢

Dense L2 is an unusually friendly SIMD workload. The memory is contiguous. Hardware prefetchers can see where we are going. Every lane follows the same path. Nobody is wandering off to find a value in a different corner of memory.

The reference code handles leftover dimensions with a short scalar loop. A production kernel might use AVX-512 mask registers instead. The boring tail is still part of the job, even when the first sixteen lanes look glorious.

Kernel 2: product-quantized distance without reconstruction

Now we make the problem more interesting.

Product quantization, usually shortened to PQ, splits a vector into smaller subvectors. Each subvector is represented by a small integer code that identifies a learned centroid. With an 8-bit code, each subquantizer can select one of 256 centroids.

For each query, we build a distance lookup table:

LUTₘ[k] = ‖qₘ - cₘ,ₖ‖²

qₘ is the query subvector for subquantizer m. cₘ,ₖ is centroid k in that subspace.

Once those tables exist, a compressed candidate can be scored with:

d_ADC(q, x̂) = Σₘ LUTₘ[codeₘ(x)]

This is called asymmetric distance computation, or ADC. The query stays at full precision. The candidate stays compressed. Each candidate code chooses one precomputed distance from the corresponding table row.

The useful trick is what we do not do. We do not rebuild the candidate's full-precision vector before scoring it.

A scalar implementation is straightforward:

float pq_adc_scalar(
    const std::uint8_t* codes,
    const float* distance_tables,
    std::size_t subquantizers) {
  float sum = 0.0f;
  for (std::size_t m = 0; m < subquantizers; ++m) {
    sum += distance_tables[m * 256 + codes[m]];
  }
  return sum;
}

This time, however, our inputs are not sixteen neighboring floats. Every byte-sized code selects a different value from a different table row. Ordinary packed loads cannot help much because the values we want are scattered.

AVX2 introduced indexed gathers for exactly this sort of awkward family reunion. AVX-512 widens the idea to sixteen single-precision lanes and adds mask-register control:

const __m128i bytes = _mm_loadu_si128(
    reinterpret_cast<const __m128i*>(codes + m));
const __m512i code_indices = _mm512_cvtepu8_epi32(bytes);
const __m512i row_base = _mm512_add_epi32(
    row_offsets, _mm512_set1_epi32(m * 256));
const __m512i indices = _mm512_add_epi32(row_base, code_indices);
const __m512 values = _mm512_i32gather_ps(
    indices, distance_tables, 4);
accumulator = _mm512_add_ps(accumulator, values);

The relevant assembly looks like this:

vpmovzxbd    zmm0, [codes]
vpbroadcastd zmm2, block_offset
kmovw        k2, k1
vpaddd       zmm0, zmm0, zmm_row_offsets
vpaddd       zmm0, zmm0, zmm2
vgatherdps   zmm2{k2}, [distance_tables + zmm0*4]
vaddps       zmm1, zmm1, zmm2

Step by step:

• vpmovzxbd loads sixteen 8-bit codes and widens them into sixteen 32-bit indices.
• vpaddd adds the offset for each subquantizer's table row.
• vgatherdps fetches sixteen floating-point values using sixteen calculated indices.
• vaddps adds those values to the running lane totals.

The {k2} attached to vgatherdps is an AVX-512 opmask. It enables the gather lanes, and the instruction clears mask bits as individual loads complete. The compiler therefore refreshes that mask before the next full-lane gather.

This is the clever part. SIMD is constructing sixteen addresses and gathering sixteen selected values while the candidate remains compressed. We save candidate memory bandwidth and avoid reconstruction.

That sounds fantastic. It is fantastic. It is also not free.

⤢

One gather instruction is still a lot of memory requests

It is tempting to see one vgatherdps instruction and imagine the CPU performing one magical memory operation. The instruction is compact. The underlying work is not.

The processor still has to calculate multiple addresses and fetch multiple values. If the lookup tables are not in cache, AVX-512 cannot bully memory latency into disappearing. Even when the tables are cache-resident, gather performance varies across processor generations and microarchitectures.

Our pinned-core Intel benchmark makes this point nicely.

On an AWS r6id.2xlarge backed by an Intel Xeon Platinum 8375C, we used four independent accumulators in the scalar, AVX2, and AVX-512 dense L2 kernels. At 1,536 dimensions, scalar measured 669.547 nanoseconds per distance, AVX2 measured 176.892 nanoseconds, and AVX-512 measured 217.820 nanoseconds. AVX-512 was 3.07 times faster than scalar, but AVX2 was faster than AVX-512 on this workload. Wider did not win this round.

For PQ ADC with 64 subquantizers, AVX-512 produced a clear win over scalar:

• Scalar: 49.759 nanoseconds.
• AVX2 gather: 19.803 nanoseconds.
• AVX-512 gather: 19.073 nanoseconds.

AVX-512 was 1.94 times faster than scalar at 16 subquantizers, 2.06 times faster at 32, and 2.61 times faster at 64. At 64 subquantizers it beat AVX2 by only about 3.8 percent, so CPU generation and workload shape still matter.

This is not bad news. This is the useful news.

The result reminds us that SIMD is a tool, not a religious commitment. Dense arithmetic usually likes wide vectors, but this tuned run preferred AVX2. Indexed lookup has its own complicated relationship with width. The CPU, as usual, declined to read the marketing copy.

⤢

These are microbenchmarks, not an end-to-end VectorAmp or SABLE performance claim. The published values are medians from ten CPU-pinned runs with three million operations per case. The raw runs, aggregate statistics, environment metadata, binary hashes, and disassembly proof are included in VectorAmp Labs.

What decides whether gather wins?

Several things move the crossover point:

• Lookup-table locality. Cache-resident tables are a completely different workload from tables fetched from slower memory.
• Amount of work. A short code sequence may not amortize setup and reduction. Larger PQ widths or batches can change the result.
• Data layout. A layout optimized for one candidate across many subquantizers may differ from a layout optimized for many candidates across one subquantizer.
• CPU generation. AVX-512 defines what the instructions do. It does not guarantee how quickly every processor implements them.
• Frequency and power. Wider instructions can affect sustained clock behavior on some processors. The best kernel is the one that wins under the real workload, not the one with the most impressive register name.
• Workload shape. Sparse filtered candidates and dense sequential scans may favor different kernels.

This is why serious search systems keep multiple implementations and select among them using runtime CPU detection and measured workload thresholds. Scalar, AVX2, and AVX-512 can all be the correct answer. Context gets a vote.

Why these kernels make a good pair

The dense kernel speeds up vector search by performing more regular arithmetic per instruction. It is classic data parallelism, and it works beautifully.

The PQ kernel attacks a different problem. It uses SIMD address construction and gathers to score compressed candidates directly. Its promise comes from compact codes, cache-friendly query tables, and avoided reconstruction. Register width alone does not carry the whole performance story.

In a larger system, the two ideas can complement each other. Approximate compressed scoring can inspect a broad candidate set. Exact dense SIMD scoring can rerank a much smaller finalist set against full-precision vectors.

That is as far as this article needs to go. Candidate selection, graph traversal, index organization, batching, and search-budget policy are separate topics. They are also where search-engine engineers start becoming suspiciously protective of their whiteboards.

A quick look toward AVX10

Intel revised AVX10 so that implementations support 128-, 256-, and 512-bit vector lengths rather than offering a separate 256-bit-only configuration. That makes width a performance choice instead of a capability question. The useful question is, "Which width gives this workload the best latency, throughput, cache behavior, frequency, and power?"

Sometimes the answer may be 256 bits. Sometimes it may be 512. Hardware has a sense of humor like that.

The same VectorAmp Labs structure can test those choices as AVX10 becomes broadly available. The important measurements will remain correctness, latency, throughput, cache behavior, CPU details, compiler details, and workload shape. A shiny instruction name is not a benchmark result. The details are in Intel's AVX10.2 architecture specification.

Reproduce the experiment

The complete educational implementation lives in VectorAmp Labs under articles/avx512-vector-distance.

The lab provides:

• Scalar, AVX2, and AVX-512 implementations.
• Runtime CPU-feature checks.
• Correctness tests across vector dimensions and PQ widths.
• A CMake build.
• A benchmark runner that emits CSV.
• Captured operating-system, CPU, compiler, and build-tool metadata.

The takeaway is simple: AVX-512 really is a gift to vector search. Dense distance calculation opens the box and starts playing immediately. Compressed lookup scoring reads the manual, rearranges the furniture, checks the cache, and then decides whether the gift fits.

Use it wisely, measure everything, and keep the scalar version around. It has a habit of humbling people.