TensorChord

With the 0.4.11 release, Hindsight now supports VectorChord (vchord) from @TensorChord, a high-performance, open-source PostgreSQL extension for similarity search

II-Commons  Infrastructure for shared knowledge.  Transparent. Distributed. Open source.  The foundation for trustworthy AI.

Building Earth Index as a searchable planet has always been ambitious & we've made sure to use the right tools to make this a reality. We're happy to use @TensorChord's PostgreSQL extension to enable environmental action at this scale. Read more here➡️blog.vectorchord.ai/3-billion-vect…

🎉 10M Docker Pulls for pgvecto.rs! 🚀 We’re thrilled to celebrate this incredible milestone for pgvecto.rs! But the journey doesn’t stop here! Meet github.com/tensorchord/ve…, its successor, offering affordable disk-based PG vector search! #PostgreSQL

ARM + VectorChord: Affordable, Fast Vector Search blog.vectorchord.ai/arm-vectorchor…

HNSW is popular but has major drawbacks, like high memory use and complex updates. Disk-based solutions like IVF, especially with RabitQ, offer better scalability and efficiency. It's time to consider simpler alternatives! #VectorDatabases #HNSW #IVF blog.pgvecto.rs/why-hnsw-is-no…

Explore the counterintuitive insights behind the performant RaBitQ algorithm for binary and scalar quantization in vector databases! dev.to/gaoj0017/quant… #RAG #vectorsearch #vectordatabase #rabitq

Check out how our vector extension supports 400k vectors for just $1! #VectorSearch #LLM #Database blog.pgvecto.rs/vectorchord-st…

> First, we’re introducing explicit functions that treat external object storage locations as first-class data citizens. Second, we’re integrating the ability to run LLMs directly within the Postgres platform. Interview with my senior coworker, Torsten. odbms.org/2024/10/on-edb…

@spectate_or Hi, are you using pgvecto.rs 0.2? The memory issue should be resolved in 0.3 release.

> find new promising vector search extension (pgvecto.rs) for postgres > convert all 20m vectors into its format > learn that the extension has a memory leak > can't convert the vectors back into pgvector format due to OOM > need to drop the column and recompute all embeddings

We are thrilled to announce the release of our Terraform provider for PGVecto.rs Cloud. This new provider is designed to simplify the process of managing resources on our cloud service blog.pgvecto.rs/introducing-te…

We're excited to announce the release of pg_bestmatch.rs, a PostgreSQL extension that brings the power of Best Matching 25 Score (BM25) text queries to your database, enhancing your ability to perform efficient and accurate text retrieval. blog.pgvecto.rs/pgbestmatchrs-…

@mattshumer_ I ended up writing it out myself using @modal_labs for getting embeddings, and @qdrant_engine for vectors, then realised I need lots of filtering and switched to pgvecto.rs

The results are conveniently visualized in the terminal using @textualizeio. The reranking step has dramatically improved quality of the top-k candidates. You can try different queries to see which retrieval method suit your requirements better.

Check out github.com/tensorchord/qt…. We've been exploring how reranking enhances the quality of RAG. #RAG #LLM

There’s thousands of RAG techniques and tutorials, but which ones perform the best? ARAGOG by Matous Eibich is one of the most comprehensive evaluation surveys on advanced RAG techniques, testing everything from “classic vector database” to reranking (@cohere, LLM) to MMR to @llama_index native advanced techniques (sentence window retrieval, document summary index). The findings 💡: ✅ HyDE and LLM reranking enhance retrieval precision ⚠️ MMR and multi-query techniques didn’t seem to be as effective ✅ Sentence window retrieval, Auto-merging retrieval, and the document summary index (all native @llama_index techniques) offer promising benefits in either retrieval precision and answer similarity! (And also interesting tradeoffs). It’s definitely worth giving the full paper a skim. Check it out: arxiv.org/pdf/2404.01037…

tensorchord / pgvecto.rs: Scalable, Low-latency and Hybrid-enabled Vector Search in Postgres. Revolutionize Vector Search, not Database. ★1271 github.com/tensorchord/pg…

TIL about binary vector search... apparently there's a trick where you can take an embedding vector like [0.0051, 0.017, -0.0186, -0.0185...] and turn that into a binary vector just reflecting if each value is > 0 - so [1, 1, -1, -1, ...] and still get useful cosine similarities!

It is important to note that this is closely tied to the embedding model being used. The benchmark we conducted specifically focused on OpenAI's text-embedding-3-large model. Maybe some binary embedding models have better performance. e.g. @cohere @JinaAI_

After conducting a thorough benchmarking process using @qdrant_engine's dataset, we have discovered fascinating insights regarding the superiority of binary vectors and shortened vectors. blog.pgvecto.rs/my-binary-vect…