Gene Firewind

@discord_support What if staff actually responded to support tickets rather than letting the system filter literally every request to canned: nope we were right you were wrong... next You're also ignoring the problem that using machine learning as a moderation tool has never and will never work

We know that's not a satisfying explanation if this was your account, and we should have caught this sooner. We're working on better safeguards so this can't quietly happen again, and more broadly, on making sure our safety systems don't penalize people who did nothing wrong. If something still feels off with your account, reply here.

@hello363459 @tallcowyt Not a single step of this error is acceptable. And Discord is just acting like "no one was really effected" which is complete bullshit. They're lying to cover their asses for PR sake. THAT is the real issue here.

@GeneFirewind @tallcowyt not from what I’ve heard. My understanding is that images with grids got put directly into photoDNAs database and because it’s a perceptual hashing algorithm it doesn’t just flag 1:1 copies, it flags anything that is interpreted as the same hash.

Sooo since Discord Support responded to my original post, I would like to respond to their response: 1) A Discord employee replied to my original post saying that it was “misinformation” and that “this has nothing to do with AI” which, no offence to them, but I think this is nonsense. For example, this official Discord article: discord.com/safety/how-dis… explains how Discord uses “machine learning”, which is a subset of Artificial Intelligence. Also, the employee misunderstood my post, I was trying to make a general point about the problems with AI moderation on Discord and they took it as if I was only referring to that one specific issue. (No hate to them obviously) 2) Adding on from that, I do believe that if Discord has to use AI for moderation, it has to be changed. The current system is fucked up and causes thousands of people to falsely lose their accounts when they have done nothing wrong. The incident this weekend was one example, but not just that, MILLIONS of people get false bans because of AI all the time!! This even happened to an owner of the unofficial Roblox server, a server which has 1m+ members and is a partnered server with Discord themselves! Link: x.com/notexttospeech… The best way to fix this going forward would be to, at the very least, give everyone an option to choose to have a human reviewer for all support tickets & appeals. I’m pretty sure Roblox already does this but I could be wrong, regardless of that it would be great if Discord and other companies like YouTube adopt this. A lot of the time, clankers are stupid and make unfair decisions, which can end up ruining people’s lives. A lot of people rely on their accounts for stuff like work or to communicate with friends and that can be taken away from them because of something that isn’t even their fault. 3) Discord claims that they’ve unbanned everyone, however I’ve seen a lot of people who are still banned because of sending grid images. Yeah… soooo… they can’t just be lying to us like that. Feel free to like & retweet to spread awareness and drop me a follow if you want. Also Discord please increase the max friends limit. 1000 is too low, it would be nice if it was 2000. And let us record audio in VCs and stages on mobile and iPhones. Thank you and have a great day!! @discord @discord_support @notexttospeech

@hello363459 @tallcowyt Their error: Use photoDNA to make training data (which was flawed); use machine learning for the AI from that data to train it on what is CSAM (now trained on flawed data); apply a false positive because of the bad training data; then BYPASS its rules-set and ban without review.

@hello363459 @tallcowyt There's no issues with photoDNA. It as you said is only a hashing algorithm. The problem comes from the training data using photoDNA to hash images for the machine learning for Discord's AI training data pool; which contained non-csam photos that just simply had grids.

@hello363459 @tallcowyt "As well as 8000 other people." Writing it was only 200 on their twitter post was the biggest idiot mistake they could because there are thousands of people reporting they've been banned for this and not reinstated. Discord lies constantly about the state of their platform.

@GeneFirewind @tallcowyt can you link me to the post saying 8k? If there is proof the 200 number is wrong I apologize for thinking it was 200.

@hello363459 @tallcowyt You have google just look it up dude... theverge.com/games/962156/d… literally every media outlet already knows it more than 200.

@hello363459 @tallcowyt Again you're not reading the damn information provided. The AI is exactly what went wrong. It took the training data that was given, and wrongfully applied it then OVERSTEPPED IT'S PERMISSIONS and BANNED WITHOUT HUMAN APPROVAL. Literally the AI is the faulty part of the system.

@GeneFirewind @tallcowyt yes, the AI isn’t what seemingly went wrong here. I’d assume an AI prefilters before it’s checked against the photoDNA database but I’ve heard *specifically* photoDNA was flagging these images and the model to even decide if it’s worth running pDNA was fine.

@hello363459 @tallcowyt Your ability to read a conversation is breathtaking. At no point did anyone call photoDNA AI. They are using machine learning to teach a moderation AI to detect images using photoDNA. Good lord where did you learn to read.

@GeneFirewind @tallcowyt yes the “ai system” used is photoDNA which has nothing to do with AI. It is a perceptual hashing algorithm, its not machine learning based. Even if it’s third party they can still easily track the account bans, not saying that’s accurate but your reasoning makes no sense

@hello363459 @tallcowyt Add in the fact that they "claimed" only 200 users when we've already confirmed via the Discord reddit that well over 8,000 confirmed users have been banned for it and probably many more because that overlay is a very small subset of users that use both reddit and discord.

@GeneFirewind @tallcowyt dude there are multiple comments from people who verifiably work at AI research companies saying photoDNA isn’t ML nor AI. In this case you are the one who doesn’t understand any of it

@BlackbluueVT @tallcowyt When you traceroute the signals sent by Discord when messages are sent you can see the data being sent to about 12 different "providers" many of which still are unnamed by Discord as to who/why/when/what/where that data is going/doing/being seen by.

@BlackbluueVT @tallcowyt Well they are actively partnered with OpenAI (images), use Sentropy (whom they bought), but also still use Community Sift (Keywords Studios) for text chats. On the voice side they use ToxMod (Modulate), and a few other undisclosed AI moderation tools for account verifications.

@Seraphinaf5 It's 71 for those not paying attention. {100 + 1} -> (101) - 40 -> (61) + 10 -> (71)

@tallcowyt @hello363459 "He knows a guy that works at Nintendo." People love to make comments in fields they've never touched. This is spot on, machine learning is a process in AI training. Some folks just really don't understand any of this and it's become more and more clear (and scary tbh).

@hello363459 I can appreciate how respectful youre being right now but machine learning is a subset of AI which is what Discord explicitly mentioned in their article about combatting CSAM

@BlackbluueVT @tallcowyt 1) Discord doesn't run the system, it's a third party 2) It absolutely bans people and does so with large error margins I've collected enough data on false bans over the last year to see it then doesn't allow the tickets to be seen by mod staff, seemingly to cover its tracks.

@tallcowyt I would actually argue the opposite. it's much better to have a false positive ban than to allow a false negative offense go through. and to my knowledge, if discord's system works as intended no AI will ever directly ban people. it only flags content for human review.

@DmVariant @tallcowyt And the fact that is was doing something outside it's installed parameters to begin with is a tell tale sign that the training data has been compromised and the action set that it was trained to follow is also flawed.

@tallcowyt We need to stop using ai as a scary buzzword. Ai for quick detection of things like CSAM is a GOOD use of machine learning. We shouldn’t stop use because it might have an error (which according to Discord was with it banning instead of suspending for manual review)

@DmVariant @tallcowyt Uh, as a developer, no. AI can barely figure out how many fingers belongs on a given persons hand when generating an image, I'm not going to trust it to detect what an object is in a vacuum. No account should be permanently or temp banned without human interaction on the account.

@hello363459 @tallcowyt So first of all, the AI system that is used for scanning images for CSAM is ran by a third party not by Discord themself. Which leads me to believe their "200" accounts is a bullshit madeyupy number like the "70,000" compromised accounts last year that ended up being 2,285,000.

@tallcowyt yeah but the ML wasn’t what went wrong. I’ve heard from someone who knows a discord backend dev that it was specifically photoDNA that started false flagging which is a hashing algorithm, not any form of AI. I guess they could be lying but I don’t see why they would

@smxgame @amuse_komimaru @smx_jp @chompochompo @HyperPotions @MDKOfficial @Quiqman @kazejapanakane @musicbyjmcx Now if only we could get more cabs. Sad that our order ended up canceled.

@amuse_komimaru @smx_jp @chompochompo @HyperPotions @MDKOfficial @Quiqman @kazejapanakane @musicbyjmcx To our new players, thank you for playing #StepManiaX! We hope you are having fun! Please make sure to follow @smx_jp! We have passionate fans running the account and will keep you up to date!

Celebrate the arrival of a #StepManiaX DX in Japan at @amuse_komimaru with the help of @smx_jp! It's StepManiaX Update #76! Available worldwide, enjoy music from @chompochompo's @HyperPotions & @MDKOfficial, @Quiqman feat. @kazejapanakane, and @musicbyjmcx. Out now!

@TakigawaYusaku @NicoleA18060980 They're not illegally parked, it's just a rural street with 2 way traffic in 1 lane. They exist all over major cities, the roads weren't designed for the massive cars we have now.

@NicoleA18060980 こんだけ路駐多かったら、車の影から歩行者が出てくる可能性を考えて止まれる速度で徐行すべきだけどな 結局ぶつかってる時点でスピード出しすぎなんよ

突然の飛び出し…でもドライバーの神ブレーキで少女が守られた。反射神経すごすぎ。