A Gaussian mechanism with ε = 6 can be less private than one with ε = 8. This points to a problem with how we report privacy guarantees in machine learning. A thread 🧵
English
Bogdan Kulynych
1.7K posts
Bogdan Kulynych
@hiddenmarkov
privacy, security, and reliability of ML · Ex @EPFL, @hseas, @Google
Katılım Eylül 2012
1.4K Takip Edilen2K Takipçiler
Here's a paper with all the details:
arxiv.org/abs/2503.10945
@physics_Felipe presenting it at #SatML on Tuesday March 24.
w/ @physics_Felipe, Borja Balle, Jamie Hayes, @FlavioCalmon, @ahonkela
English
We've built a new Python package gdpnum to compute non-asymptotic GDP guarantees and estimate their precision for many practical algorithms:
github.com/interpretable-…
English
Here's an example for a specific instantiation of DP-SGD in terms of f-DP trade-off curves (an equivalent operational version of privacy profiles). As we see, a non-asymptotic GDP trade-off curve fits the DP-SGD trade-off curve almost exactly.
English
Many ML algorithms, especially those involving many compositions like DP-SGD, can be very precisely characterized with GDP. This is a *non-asymptotic* result, not just a central limit approximation!
English
GDP characterizes the entire privacy profile ε(δ) of a Gaussian mechanism exactly using a single number μ. Interpretation: if a mechanism satisfies μ-GDP, then running membership inference against it is as hard as distinguishing N(0,1) from N(μ,1) based on a single observation.
English
Can we do better without reporting an entire privacy profile? Yes! With Gaussian differential privacy (GDP).
English
As the convention sets δ in a data-dependent way, this matters whenever you compare models across datasets or papers.
English
Issue 2: You can't properly compare two mechanisms by ε if their δ values differ. A Gaussian mechanism with ε = 6 at δ = 10⁻⁵ is less private than one with ε = 8 at δ = 10⁻⁹. This is because you cannot properly compare ε if δ is different.
Română
No attacker in the universe can achieve that 98% rate: It's purely an artifact of compressing the entire privacy profile into one pair (ε, δ). My colleagues and I detailed on this problem in detail in this NeurIPS'24 paper:
arxiv.org/abs/2407.02191
English
Issue 1: A single (ε, δ) pair can massively overstate privacy risk. Example: DP-SGD with ε = 8 at δ = 10⁻⁵ suggests a worst-case membership inference accuracy of ~98% using standard conversions. But using the full privacy profile, the actual maximum is only ~68%.
English
The standard way is to report is to use a single (ε, δ) pair for a small δ. The community has developed informal conventions, e.g., ε < 10 is generally considered OK in privacy-preserving machine learning. But this convention has two big issues.
English
Presenting this on Thursday Dec 4 at #EurIPS in Copenhagen. Come by at the poster session if this sounds interesting!
#NeurIPS2025
Bogdan Kulynych@hiddenmarkov
New paper at #NeurIPS2025! "Unifying Re-Identification, Attribute Inference, and Data Reconstruction Risks in Differential Privacy" in which we derive unified, tighter bounds on operational attack risks for any DP mechanisms, using f-DP. Link: arxiv.org/abs/2507.06969 Thread👇
English
This is a unifying framework which can model various types of risk.
English
New paper at #NeurIPS2025!
"Unifying Re-Identification, Attribute Inference, and Data Reconstruction Risks in Differential Privacy" in which we derive unified, tighter bounds on operational attack risks for any DP mechanisms, using f-DP.
Link: arxiv.org/abs/2507.06969
Thread👇
English
Very excited, and I think this will be quite useful for practical deployments of DP.
This is a joint work with great Felipe Gomez ( felipe-gomez.com ), George Kaissis, Jamie Hayes, Borja Balle, @FlavioCalmon, JL Raisaro.
English
Another (final) finding. The unified f-DP bound extends to a form of a generalization bound. Given that we can compute f-DP curves precisely, this is likely the tightest generalization bound applicable to deep learning, but it is only for on-average generalization unfortunately.
English
Continuing the thread on "Unifying Re-Identification, Attribute Inference, and Data Reconstruction Risks in Differential Privacy", for some reason it got borked.
x.com/hiddenmarkov/s…
Bogdan Kulynych@hiddenmarkov
New paper at #NeurIPS2025! "Unifying Re-Identification, Attribute Inference, and Data Reconstruction Risks in Differential Privacy" in which we derive unified, tighter bounds on operational attack risks for any DP mechanisms, using f-DP. Link: arxiv.org/abs/2507.06969 Thread👇
English