Haw-Shiuan Chang

When building ColBERT, I assumed it will pave the way for hypernetwork-based, pruning-capable retrieval indexes. Let me explain. The big insight in ColBERT is that we can encode each document upfront *not* into a vector, but into a rich scoring function, f: query -> float, which simultaneously supports pruning, so you can skip most computation. In v1/v2, the choice of function was "a matrix + MaxSim". It showed that at inference time, we could do a lot better than dot products. But in the future, the function could also be a small DNN constructed out of each document! The encoder is then a hypernetwork producing functions f with the same query -> float signature, allowing each document to decide its strategy for deciding if a query is relevant to it. How to do this while allowing pruning (so that retrieval is sub-linear at scale) is a rich question you can steal if you're doing NLP systems or IR.