Gaurab Bhattacharjee

AI is, if not more paradigm-shifting than the internet. Think of it as horse & carriage vs car. You'll still get there, just 84 days later. You're already late if Time to Market (TTM) is your metric. This is ​AI First​. via @StartUpScaleUp_ startuptoscaleup.com/newsletter-sig…

International Data Corporation projects that 750 million cloud-native applications will be created globally by 2025 as businesses work toward building these sustainable digital value engines. #DigitalTransformationChallenges idc.com/getdoc.jsp?con…

Well written, Abhay! Prompts, if not managed well, definitely have security implications. At the same time, do we need to do something very different to protect against any abuse cases? Prompts are inputs coming from untrusted sources and should be treated as such. Prompts with placeholders for customization based on user input are particularly important to keep an eye on.

"Prompt Injection is not a security issue" Is essentially what I am hearing from some security/appsec folks out there. At first, I was inclined to agree. LLMs are probablistic in nature and prompt injection doesn't *seem* like a security issue. However, this has deeper implications. Let me try and dive in, based on what I have been experimenting with, in this space. FYI - I am not an AI expert. I am just someone trying to figure things out as I go. I am sure I will make some mistakes, perhaps even in this post. So please excuse my ignorance and point out where I might be wrong or right. First - I see Prompt Injection as a distinct family of attacks. An Attack Vector that has a variety of attack impacts and effects. A lot of us see prompt injection examples online as being able to "trick" a chatbot into giving us answers in humorous ways, or give us completely irrelevant information, not related to the area that the chatbot is "trained" on. Yes, this is an issue. Its not the *ONLY* issue. This in my view is one type of Prompt Injection, where Topical relevance of a GenAI application can be subverted to get the application to access other topics. Depending on the seriouness of the app, this can be either trivial or very bad. Then we have Sensitive Information Disclosure. Let's suppose that the GenAI app is trained on a bunch of the org's data. Some of it, sensitive. If an attacker is able to craft a prompt that gets the Application to spit out sensitive info, that is also (Prompt) Injection in my book. Yes, its an outcome of the same attack vector. But done through subversion of the same prompt Then we have "Excessive Agency". This is a scenario where a GenAI application uses Agents (that execute actions) based on one/many LLMs. For example - An Agent/Agents app that writes the entire codebase based on a user instruction or uses agents to manage a user's email inbox based on the user's instruction/prompt. This is actually much deadlier IMO than the other ones, because now its not only Text/Code Gen, but its interacting with Agents that are interacting with external APIs. This is also done by subverting the prompt. Finally we have Second Order Injections. These are Injections that are triggered by the use of external data in prompts. This includes being trained on datasets that are poisoned by a malicious actor. For example - If a RAG application is trained on a dataset, which selects a leader from a given dataset. Let's assume that somehow, the dataset has been tainted to get the leader's name to appear in multiple datasets, then the App might select this person as a leader. This *may* also have something to do with how the prompt is structured. This is why I see Prompt Injection as a very deep-seated security problem. And while it is made worse with other misconfigurations, it seems front-and-center in the discussion around vulnerabilities that any of us have around LLM Security

Security in Software Development's Feedback Loop: Bridging the DevOps-Architect/Product Manager Divide in Security #DevSecOps #SecureSoftwareDevelopment #SecureProductManagement appsec360.com/post/closing-t…

🚀 Delving into the #BSIMM14 Report: A guide through the ever-evolving world of software security. Key takeaways: automation in security, supply chain risks, adapting to AI, and the rise of product security. appsec360.com/post/navigatin… 🛡️ #SoftwareSecurity #bsimm14

In the blog post, we map the CIA Triad to the STRIDE model in #Cybersecurity. Understand how these frameworks intersect for effective threat analysis and protection. appsec360.com/post/mapping-b… #InfoSec #CyberThreats #CIAvsSTRIDE

Part 2 of the series is out! Dive into the essentials of a Cybersecurity Assessments Delivery Framework tailored for the AI-first world. appsec360.com/post/cybersecu… #AI #Cybersecurity #Innovation"

As AI integration becomes mainstream, it's vital for product security teams to adapt and stay ahead. We're launching a series focussed on strengthening product security teams for an AI-driven future. appsec360.com/post/ramp-up-p… #Cybersecurity #AI #DigitalTransformation

Inside the White-Hot Center of A.I. Doomerism nytimes.com/2023/07/11/tec… #ai-safety #safe-ai

The introduction to the latest edition of tl;dr has a hilarious way of explaining what it means to be cybersec 😂🤣 Rest of the content is absolute gold as always 👏 tldrsec.com/p/tldr-sec-214 via @clintgibler

AI Hype Alert! Don't get blinded by the buzz. Remember, like any software, AI MUST be Secure by Design. #SecureAI #CISAnews cisa.gov/news-events/ne…

"We encourage every technology manufacturer to build their products based on reducing the burden of cybersecurity on customers..." Yes, to empowered users & less vulnerability! #CybersecurityResponsibility #SoftwareUsability

"By incorporating secure by design practices, we can break the vicious cycle of constantly creating and applying fixes." Stop the patching madness! Build secure foundations. #SecureSoftware #BreakTheCycle cisa.gov/resources-tool…

Keep AI models and security tools updated with the latest security patches and threat intelligence.

Train developers on the potential risks of AI-generated (& human-written) code and encourage critical evaluation of suggestions. (if possible) train AI models on secure code examples

There is a lot of buzz around whether AI-generated source code requires special security treatment (or not!). The research paper (arxiv.org/pdf/2310.02059…) presents a detailed analysis of this topic! #ai #security

I have been a @NotionHQ user, and 🩷 this piece on how Notion leveraged #ProductLedGrowth to grow - How Notion Grows, by open.substack.com/pub/aakashgupt…