BiasedMerc

141 posts

BiasedMerc banner
BiasedMerc

BiasedMerc

@BiasedMerc

Smart Contract Auditor - Competing in Competitive Audits

เข้าร่วม Ocak 2023
134 กำลังติดตาม90 ผู้ติดตาม
Destiny | Steven Bonnell II
Destiny | Steven Bonnell II@TheOmniLiberal·
Hey! I’m not a conservative but I’ve had a lot of conversations with conservatives (Ben Shapiro, Jordan Peterson) and a lot of conversations about Ukraine and the current war with Russia. Would be awesome to get the chance to talk, would be willing to travel to Kyiv to make this work!
English
112
369
6.6K
253.2K
Path of Exile
Path of Exile@pathofexile·
We're still working on deploying the database improvements and will keep you updated as soon as we know more. The moment we can provide a precise timeline we will. Thank you very much for your patience.
English
1.3K
454
10.9K
1.4M
Path of Exile
Path of Exile@pathofexile·
❗️ Path of Exile 2 Early Access Launch Day Thread ❗️ Today is the day! Follow this thread for updates on as we head into Early Access for Path of Exile 2. We'll also be adding information to the Live Updates thread on our website: pathofexile.com/forum/view-thr…
Path of Exile tweet media
English
1.3K
633
10.7K
3.7M
BiasedMerc รีทวีตแล้ว
Free Pertsev & Storm
Free Pertsev & Storm@FreeAlexeyRoman·
Thank you to anonymous donor 0x5cB...52A49 for contributing 25 ETH to Alexey's legal defense on Juicebox. With ~ 53 ETH ($140k) raised so far, your generosity brings us closer to our goal of 750k-1M needed to fund Alexey's appeal. Open source is not a crime. Free Alexey.
Free Pertsev & Storm tweet media
English
2
13
46
1.8K
BiasedMerc
BiasedMerc@BiasedMerc·
@Xc1008Cui We tanking our accuracy ratio with this one 🔥
English
1
0
4
258
0xladboy | Sparkware
0xladboy | Sparkware@Xc1008Cui·
If a contest has 1000 sloc, in theory, you simply submit 1000 issues claiming every line of code is wrong and every issue is then your duplicates and all bugs are found.
English
15
1
51
6.9K
BiasedMerc
BiasedMerc@BiasedMerc·
@chrisdior777 Great advice! Do you know a good source to find interesting articles related to Web3 security/ auditing? I love reading articles but find it hard to find new ones consistently
English
0
0
0
37
chrisdior.eth
chrisdior.eth@chrisdior777·
Tips to Improve as a Smart Contract Auditor: - Read Web3 security experts' tweets daily - Read 1-2 articles daily - Study findings and attack vectors daily - Read and analyze a lot of code - Practice on Code4rena and Sherlock - Chat with fellow auditors
English
3
16
108
4.4K
Jack Sanford 🛡️
Jack Sanford 🛡️@jack__sanford·
Where are you sitting on the way to EthCC?
Jack Sanford 🛡️ tweet media
English
41
4
90
15K
Ilchovski
Ilchovski@ilchovski98·
If the same 200k contest is running on multiple competitive audit platforms where would you compete?
English
7
0
15
2.9K
BiasedMerc รีทวีตแล้ว
Alex
Alex@__alexxander_·
SR: "Man, I wonder why escalations / pjqa take so long." Also SR:
Alex tweet media
English
5
1
33
4K
BiasedMerc
BiasedMerc@BiasedMerc·
@_sammytm @0xjuaan @jack__sanford Very true, that's why I am always worried to ask questions about specific part of the code where I think there might be a bug :/
English
0
0
4
73
sammy
sammy@sammyaudits·
@0xjuaan @jack__sanford Have experienced #2 before, when someone already disclosed a vuln in private threads. Sponsors do unknowingly drop hints.
English
2
0
7
277
Jack Sanford 🛡️
Jack Sanford 🛡️@jack__sanford·
Question for security researchers: Do you think it's fine for an audit contest platform to share your submissions with the client team while the contest is still running?
English
15
1
22
15.8K
BiasedMerc
BiasedMerc@BiasedMerc·
Great reply. One aspect I wanted to ask about is that for a newcomer into the space like me, the escalation system for Sherlock seems more inviting. It is open to anyone to be able to escalate their own issues, even if they do not have any special access like is needed in Code4rena to view findings before the audit is over. I would be curious to see if this has an effect between the two platforms on the percentage of reports that are correctly resolved to be marked as valid, where they have been initially incorrectly marked as invalid due to automated bot sorting or a judging mistake. In Sherlock, anyone can view all issues during the escalation period, so Wardens can check their own reports and ensure they have been correctly duplicated with other valid findings. In case their issue was missed, they can point it out. In Code4rena, the average auditor doesn’t have this privilege due to the required role access, meaning some valid findings that were incorrectly invalidated may never be validated, which can discourage people from participating if they feel they may not be rewarded for their valid findings in such cases.
English
0
0
1
362
Sock
Sock@sockdrawermoney·
Part of the problem is you can’t compare apples to apples. C4 has a wider and deeper talent base to draw from and while the c4 model is brutally meritocratic, it’s not inherently mercenary and cannot result in a scenario where someone can sociopathically “mine” the unarbitraged value of a position they’ve achieved via accrued reputation. C4 *adding* LSW on top of a working incentive system would be inherently different from Sherlock using LSW to bootstrap their platform by paying top C4 people to compete. I consider LSW a “sociopathic” incentive rather than a meritocratic one, as the highest possible short term EV is achieving LSW and double booking, putting in reduced effort, or even selling off those opportunities to others, which has happened. I believe by adding sociopathic incentives at the core of the model, you open yourself up to a lot more “I’m getting mine” behavior which is part of my theory as to why Sherlock escalations are intensely PvP to a massively greater extent than the equivalent of what we do, which we call post-judging QA. Even the framing makes a difference imo. The goal in C4 is a collaborative end result from a competitive environment. We have an “escalation” phase in order to do our best to make sure things aren’t missed, but we consider it QA on the judge’s work and not self-focused escalation. The very existence of the LSW mentality and its third-order effects has gradually had a net negative impact on the broader competitive audit landscape, even on C4. Wardens prior to Sherlock will tell you the environment was fiercely competitive but FAR more collegial. One of the funny ironies is that while we are definitely pushing competition, the majority of the very best SRs I know are gentle, humble, curious souls who aren’t really conflict-loving—which is different from feeling compelled to point out when “someone is wrong on the internet.” I think common auditor personality traits lean these folks toward a greater natural affinity for C4, and so we have a deeper roster of talent to draw from. Thus “you have plenty of skilled auditors who will definitely participate” just tends to be more true on C4.
English
2
0
11
7.1K
Whitehat Bandit
Whitehat Bandit@banditx0x·
My game theory model of why @code4rena has 300% to 500% more community audit coverage than @sherlockdefi🌶️ Let's assume there are 2 contests running with the exact same Contest Pool, nSLOC, duration: - Code4rena - Sherlock Now let's say all the auditors are given a ELO rating and instead of actually participating in the audit the community pool is split based on your ELO score. The auditors have a wide variety of scores (avg is around 10), and all the auditors scores sum to 1000. The best auditor in the world is CMichel. He has an ELO of 200 which is much higher than anybody else. The rational reaction for all possible other participants is to distribute themselves such that 300 ELO worth of auditors does the Sherlock contest and 500 ELO worth of auditors. However in Sherlock, 35% of the pool goes to the Senior Watson, leaving 65% to the the . Let's simpify the maths by assuming some auditors drop out of the Sherlock audit instead of switching to Code4rena. Now the total Sherlock ELO will have to be 65% of 500 = 325. This would be 125 ELO of community auditors and + 200 ELO from CMichel Compared to 500 ELO of community auditors go to Code4rena. In summary: With participants acting in their own self interest, Sherlock had overall a lower aggregate ELO even including CMichel's high contribution. Code4rena had 4x the community audit participation despite Sherlock's reward pool being reduced only 35%. The key point is that auditors respond not only to the reduced prize pool, but also to the loss in Expected Value from competing against CMichel.
Jack Sanford 🛡️@jack__sanford

I'm spilling some alpha here but I guess I'll tell the real story: Sherlock realized that smart contract coverage doesn't work if audits don't work. So Sherlock got really focused on auditing. Started doing traditional audits, then tried C4 for our own smart contracts and it seemed better than trad audits. When we did the C4 audit, we had been in contact with @cmichelio and wanted him to participate in our audit contest as well as 2 other top auditors back in the day. But we ran the C4 contest and none of them showed up. I think one of them had a wedding or something that week. We still got a good result from the contest, but we didn't get the top guys. Now that we're 2 years into running Sherlock audit contests where we reserve the top guys, we know that getting the top guys is SUPER important. We know the LSW finds a High Solo every 5 Highs on average. The rest of the field misses 1 in 5 Highs. If the pot was a bit bigger would that help? Maybe marginally. The point is: the top guys really are THAT good. Sherlock considered NOT running audit contests and telling coverage customers to go to C4. But due to the top auditor reservation problem and a handful of others, Sherlock decided we could only trust ourselves to create the type of audit contest that could reliably justify cheap coverage afterward. @sockdrawermoney 1) The LSW drives super important security outcomes (1 in 5 critical vulns missed otherwise) 2) There was a very specific security gap: the top guys are important and sometimes they don't show up 3) It wasn't an unfounded customer objection because we WERE the C4 customer who had this objection @0xMackenzieM I think the LSW model is actually the beginning of a very large trend in the opposite direction. I think we're going to see audit contest platforms follow Sherlock and reserve multiple auditors for each contest like we are doing now. The reason for this is simple: How else are you going to get the top guys when 22 other contests are running?

English
12
3
62
16.1K
BiasedMerc
BiasedMerc@BiasedMerc·
@czar102 FACTS! You need time to recover and consolidate your learnings once the day is over.
English
0
0
3
179
Czar102
Czar102@Czar102·
Every auditor *needs* to be aware that THEY ARE PAID TO SLEEP well. If you slept 4 hours and go to your auditor work, you may as well haven't come to the office at all. It's even worse if you come to work because you'll think you checked the code while it's not really the case.
English
5
1
49
3.1K
Juan
Juan@0xjuaan·
Excited to announce that I'll be learning the magic of ZK with the best educators in the space @RareSkills_io Follow for potential ZK content in the future 👀
Juan tweet media
English
6
2
81
6.9K
BiasedMerc
BiasedMerc@BiasedMerc·
I think saying that “he probably won’t” show up is not fair to be used in this scenario. In audits, there is a clear financial incentive to show up, find bugs, and get paid, even without the LSW pay (which would lead to a 1/3 bigger reward pool for findings). Yes, LSW pay guarantees that a LSW will be present, but without it, would those auditors not be competing in audits? For the Zivoe contest, the total rewards were 80,000 USDC; however, the LSW pay was 24,500 while the rewards for findings were 47,500. I strongly believe if the findings rewards were 72,000, it would have led to more top auditors competing, rather than only heavily incentivizing one LSW to compete. However, I do agree that it does guarantee a more even distribution of top auditors across Sherlock audits, especially for low total reward audits.
English
2
0
1
402
Jack Sanford 🛡️
Jack Sanford 🛡️@jack__sanford·
So you want to throw a party 🎉 You rent out a venue 💃 You hire a DJ 🪩 You offer free drinks 🍸 You even spend $50k to upgrade the sound system 🎶 Unless you get confirmation from Cristiano Ronaldo that he'll show up, he probably won't @sherlockdefi is the only platform that confirms the world's top auditors will show up to your audit contest And then incentivizes them to try their hardest
Sock@sockdrawermoney

This is the observation we made that resulted in us not adopting the same model. Platforms and their constituents aren't 1:1 fungible. Tweak incentives and you get different communities, different participation, different types of participation, and different results. LSW was absolutely necessary to bootstrap participation on Sherlock. It made perfect sense in that context. Again: smart! No shade. In the context of C4 it made more sense to encourage booking a solo auditor who a customer really wants to work with at their desired rate.

English
6
0
8
5.9K
BiasedMerc
BiasedMerc@BiasedMerc·
I do think the LSW model is interesting and something to aspire to (even though a small minority will ever reach it), but I do agree that it likely has no real impact on the outcome. I can imagine that currently, the LSWs are much less likely to compete where they are not the LSW for that specific contest. (I mean, they get 1/3 of the total prize pot for contests where they are LSWs, so why would they waste their time if they are not getting the fixed pay?) However, if there were no LSW pay, then the pot would be bigger, and the top Watsons (currently LSWs) would participate anyway and get higher pay for their found issues. I do think the C4 new hunter and gatherer bonuses are fairer, as in theory, anyone could get the bonus if they have a top-level performance in a contest. There is no need to reach the top leaderboard to qualify for this bonus. While LSWs get a large percentage of the pot, they need to be at the top of the leaderboard. This disincentivizes others from competing, as why would any top-level Watson compete in a contest where the prize pot has been reduced by 1/3 to give another Watson fixed pay? This can turn some ‘competitive’ audits into basically solo audits, where others can compete to get severely underpaid for their findings.
English
0
0
3
1.4K
Sock
Sock@sockdrawermoney·
I highly respect LSW as a savvy hedge that acknowledges competitive audit concept sounds scary to uninitiated customers, but LSW performance is largely irrelevant to security outcomes based on the statistics. In @jack__sanford's epic DSS talk analyzing the stunning effectiveness of the c4 model, it's not like he identified LSW as solving some specific *security* gap. For Sherlock to pivot to competitive audits, _some_ differentiation from c4 was needed. The differentiation they picked was addressing an (arguably unfounded) customer objection. It was a business savvy move—made me quickly realize Jack's probably a more shrewd business guy than I'll probably ever be.
English
2
0
15
16.5K
Nirlin - Security Auditor
Nirlin - Security Auditor@0xnirlin·
He is such a genius. But it raise few question too: 1. If Obront won by such a margin in a highly competitive contest, what does this say about the quality of audits with low competition? 2. Considering there are currently 23 contests listed on Daily Warden, how does this affect the overall quality of competitive audits, given that a high number of experienced auditors are likely participating in Euler right now? 3. What about the projects running contests that aren't from the Ethereum ecosystem or using Solidity? For example, some of the Rust contests happening right now. Wouldn't they be better off not running the contest at this time and instead allocating that budget to something like Immunefi Boost, which has more Rust-based talent?
obront | eth/acc@zachobront

op stack is my love language

English
5
0
46
12.9K
BiasedMerc
BiasedMerc@BiasedMerc·
@0xOwenThurm All these influencers selling out for a bag 😭🤣
English
0
0
1
136
Martin Marchev
Martin Marchev@MartinMarchev·
@ilchovski98 @audit That’s why I always write the report as soon as I find the issue. Less context switching and I can manage my time a bit better that way.
English
1
0
1
116
Ilchovski
Ilchovski@ilchovski98·
Always write down your attack ideas during an audit and go through them. Otherwise you risk to end up like me while sitting on a bench in the park 2 days after the contest’s end to remember an issue with the codebase
English
1
0
24
943
BiasedMerc
BiasedMerc@BiasedMerc·
@high_byte In any industry you can find terrible companies that will overwork you like crazy, whilst others will provide great benefits and have a good environment. I've experienced both and know the 1st option will spill into personal life and make you miserable...
English
0
0
0
59
high_byte
high_byte@high_byte·
firefighter calls software development mental and physical abuse
high_byte tweet media
English
4
0
7
794
BiasedMerc
BiasedMerc@BiasedMerc·
@0xSimao @sherlockdefi Nice work! Reading through your findings now, and I see you had some tough judging discussions💪
English
1
0
2
134
Martin Marchev
Martin Marchev@MartinMarchev·
@BiasedMerc @bytes032 You got me, man 😁 I've loved quotes ever since I was a kid. My grandpa had a book called "Wisdom Through the Ages" on his desk. It was filled with quotes from famous thinkers on various topics. I used to spend entire afternoons immersed in it.
English
1
0
1
23
@bytes032.xyz
@bytes032.xyz@bytes032·
For whoever needs to hear this: Learning technical stuff can be a real drag—sometimes, it's just boring, and other times, it's downright frustrating. You'll have days when you're pumped up and can power through the tough parts and other days when you're ready to throw your computer out the window. The superlearners are defined by how they handle those tough days when they feel like giving up. On those days, you can brain-hack yourself by saying: "I'm going to commit to doing [insert one small thing] today." It has to be a tiny commitment so it doesn't feel overwhelming. Some days, I do the bare minimum and call it a day. But most of the time, once I start, I study for 2-3 hours and feel great. On low-energy days, you might switch to watching or reading educational content instead of diving into hands-on work. It’s still productive but takes less effort. The key is to do something every day to keep moving forward in your career! Remember that it's a marathon and not a sprint.
English
11
24
201
11.8K

ค้นพบ

@TheOmniLiberal @dmtrltvn @pathofexile @Xc1008Cui @chrisdior777 @jack__sanford @xiaoming9090 @ilchovski98