Gaurish Chandelkar

𝗔 𝗩𝗶𝘀𝘂𝗮𝗹 𝗚𝘂𝗶𝗱𝗲 𝗢𝗻 𝗛𝗼𝘄 𝗧𝗼 𝗖𝗵𝗼𝗼𝘀𝗲 𝗧𝗵𝗲 𝗥𝗶𝗴𝗵𝘁 𝗗𝗮𝘁𝗮𝗯𝗮𝘀𝗲 Choosing the right data store can be confusing with so many options around. This diagram shows a selection choice for a datastore based on a use case (𝗦𝗤𝗟 𝘃𝘀 𝗡𝗼𝗦𝗤𝗟). Data can be 𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱 (𝗦𝗤𝗟 𝘁𝗮𝗯𝗹𝗲 𝘀𝗰𝗵𝗲𝗺𝗮), 𝘀𝗲𝗺𝗶-𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱 (𝗝𝗦𝗢𝗡, 𝗫𝗠𝗟, 𝗲𝘁𝗰.), 𝗮𝗻𝗱 𝘂𝗻𝘀𝘁𝗿𝘂𝗰𝘁𝘂𝗿𝗲𝗱 (𝗕𝗹𝗼𝗯). In the case of structure, they can be relational or columnar, while in the case of semi-structured, there is a wide range of possibilities, from key-value to graph. Credits Satish Chandra Gupta. Back to you, which database have you used for which workload? Check the full source in the comments. _______ If you like my posts, please follow me, @milan_milanovic, and hit the 🔔 on my profile to get a notification for all my new posts. Learn something new every day 🚀! #SQL #NoSQL #Data #Database #AWS

How do you build a 𝗟𝗟𝗠 𝗯𝗮𝘀𝗲𝗱 𝗖𝗵𝗮𝘁𝗯𝗼𝘁 𝘁𝗼 𝗾𝘂𝗲𝗿𝘆 𝘆𝗼𝘂𝗿 𝗣𝗿𝗶𝘃𝗮𝘁𝗲 𝗞𝗻𝗼𝘄𝗹𝗲𝗱𝗴𝗲 𝗕𝗮𝘀𝗲? Let’s find out. First step is to store the knowledge of your internal documents in a format that is suitable for querying. We do so by embedding it using an embedding model: 𝟭: Split text corpus of the entire knowledge base into chunks - a chunk will represent a single piece of context available to be queried. Data of interest can be from multiple sources, e.g. Documentation in Confluence supplemented by PDF reports. 𝟮: Use the Embedding Model to transform each of the chunks into a vector embedding. 𝟯: Store all vector embeddings in a Vector Database. 𝟰: Save text that represents each of the embeddings separately together with the pointer to the embedding (we will need this later). Next we can start constructing the answer to a question/query of interest: 𝟱: Embed a question/query you want to ask using the same Embedding Model that was used to embed the knowledge base itself. 𝟲: Use the resulting Vector Embedding to run a query against the index in the Vector Database. Choose how many vectors you want to retrieve from the Vector Database - it will equal the amount of context you will be retrieving and eventually using for answering the query question. 𝟳: Vector DB performs an Approximate Nearest Neighbour (ANN) search for the provided vector embedding against the index and returns previously chosen amount of context vectors. The procedure returns vectors that are most similar in a given Embedding/Latent space.  𝟴: Map the returned Vector Embeddings to the text chunks that represent them. 𝟵: Pass a question together with the retrieved context text chunks to the LLM via prompt. Instruct the LLM to only use the provided context to answer the given question. This does not mean that no Prompt Engineering will be needed - you will want to ensure that the answers returned by LLM fall into expected boundaries, e.g. if there is no data in the retrieved context that could be used make sure that no made up answer is provided. To make it a real Chatbot - face the entire application with a Web UI that exposes a text input box to act as a chat interface. After running the provided question through steps 1. to 9. - return and display the generated answer. This is how most of the chatbots that are based on a single or multiple internal knowledge base sources are actually built nowadays. We will build such a chatbot as an upcoming hands on SwirlAI Newsletter series so stay tuned in! -------- Follow me to upskill in #MLOps, #MachineLearning, #DataEngineering, #DataScience and overall #Data space. Also hit 🔔to stay notified about new content. 𝗗𝗼𝗻’𝘁 𝗳𝗼𝗿𝗴𝗲𝘁 𝘁𝗼 𝗹𝗶𝗸𝗲 💙, 𝘀𝗵𝗮𝗿𝗲 𝗮𝗻𝗱 𝗰𝗼𝗺𝗺𝗲𝗻𝘁! Join a growing community of Data Professionals by subscribing to my 𝗡𝗲𝘄𝘀𝗹𝗲𝘁𝘁𝗲𝗿: newsletter.swirlai.com

Glad to have completed an intensive training of #Architecture #Competency that helped see the horizon more clearly. Thanks, @Persistentsys for organizing this. A shout out to the #PersistentUniversity and @Stack_Route team for crafting this #SeeBeyondRiseAbove journey.

AWS Config can: 💥 Record all of the configuration data that runs through the system. 💥 Build rules to help us ensure compliance. As a bare minimum, here are 12 recommended Config rules courtesy of cloud architect and security engineer @DonMagee. okt.to/pCDOLi

Excellent video to get understanding on #OAuth2 #OpenIDConnet youtu.be/996OiexHze0

Design to NOT hit AWS Limits — soft and hard link.medium.com/A2nG0tVb2gb #aws #limits #lambda #resilient #cloud #design

Design to avoid duplicates in AWS Streaming apps link.medium.com/oZ7X8nuYPgb #aws #duplicates #streaming #kinesisfirehose #s3