This page has archives. Sections older than 5 days may be automatically archived by Lowercase sigmabot III.
editor similarity
Hi, you mentioned some method for estimating the distance between editors in "metric space". I'd be curious what you mean by this and where I can find more information. Thank you! DMH223344 (talk) 15:29, 21 June 2024 (UTC)[reply]
@DMH223344: It's a bit of a long story. Years ago, I read about some work using machine learning to identify socks. It has been rattling around in my mind ever since. Unfortunately, I don't have access to a bunch of A100s and the enwiki database is too big, so machine learning is not an option. But the fact that the system could apparently discover features that enabled it to detect 90%+ socks suggests that it is possible to do some kind of proximity ranking of accounts, albeit in the very high dimensional space of a neural network. Since then, I've been wondering what those features were (the network is a black box so I have no idea) and whether looking at distances between editors in much lower dimensional spaces might still be able provide clues about sockpuppets. I've just started looking at this, and it's a bit of a rabbit hole, but it might have some potential. My comment was based on this test output (I've anonymized it here). I suggested a match based on the low Wasserstein distance between the editors in a particular space (I'll omit the details). I really have no idea whether it is a good match because my test set is small right now, I just happened to be looking at the test output at the time (although the Editor Interaction Analyser suggests it might be a decent match). It's possible to construct all sorts of spaces from editor data and I don't know which ones could be useful. Also, there are many things that I haven't gotten around to doing and aren't clear to me yet, like the relationship between proximity and dimensionality, how many samples per editor are needed (I have to pull them out of the database), can this approach 'predict' the outcomes of previous SPI reports etc. Sean.hoyland (talk) 17:19, 21 June 2024 (UTC)[reply]
interesting! one thing we should be very aware of with any method like this is the false positive rate. It's of course essential to have an idea of how well we've controlled this rate using the proposed method. There are of course many ways we can do that, depending on what data we have and how it was collected.
some brief comments without know much of the details of what you're describing: I would *not* expect a neural network based approach to work well for this task. More generally, I would not expect a supervised approach to work well. I also dont think gpus are required, although I'm guessing some approaches might not be open to us without access to many cpus and large working memory.
Feel free to share any other details you might have, even if you consider them overwhelming or disjointed thoughts. DMH223344 (talk) 17:50, 21 June 2024 (UTC)[reply]
In general, I would say that a method which tries to identify pairs of user accounts that are socks will not work unless we make some strong assumptions or come up with a clever way to identify candidate pairs. Otherwise our FPR will likely be huge. DMH223344 (talk) 17:53, 21 June 2024 (UTC)[reply]
I would not have expected a ML based approach to work well for this task either because I assumed the signal to noise ratio would be problematic. And yet it does work, quite well it seems. There are several papers. The fact that these systems are doing something (opaquely) that works is encouraging in the sense that it shows there are features there and not just in our imaginations when we see patterns connecting users. I'm trying a no assumptions, just do math approach. Sean.hoyland (talk) 18:17, 21 June 2024 (UTC)[reply]
I dont disagree that an ML based approach would work. It was the supervised framing that I was unsure of. but of course I could be wrong! DMH223344 (talk) 19:04, 21 June 2024 (UTC)[reply]
Technical question
Hey, how many edits per account are required so that your software can actually (relatively reliably) group them together? Can a “full picture” be created out of multiple low-edit accounts? FortunateSons (talk) 09:14, 24 June 2024 (UTC)[reply]
I really have no idea. At this stage I don't have any evidence that it could reliably group identical twins who shared a room, edited together, with a passionate interest in rare Mongolian hats. I'm in the stumbling around in the dark, bumping into things phase where I keep realizing 'oh, to do that, I need to be able to do this first, but how?' My normal approach to problem solving is to not think about it and do something else. Puzzlingly this works quite well for mysterious reasons, albeit slowly. As for the question 'Can a “full picture” be created out of multiple low-edit accounts', maybe, but I guess only if there is high confidence that combining them boosts a signal for a single source rather than introduces noise. Sean.hoyland (talk) 10:37, 24 June 2024 (UTC)[reply]
That's unfortunate, but if I find two editors with an obsessive interest in Gugu hats, I'll let you take a swing at it. :)
I can definitely relate to this style of problem solving, and wish you the best of luck with the work.
I'm asking because I seem to have made an acquaintance trying to make me join into their topic-specific discussions through improper means, and while I'm appropriately dealing with the messages themselves, I was curious whether the half dozen accounts with ~10 edits each are enough to create a profile that can be compared to others with an above-random chance of success? Some of them caught joint checkuser blocks and some haven't, so... FortunateSons (talk) 11:15, 24 June 2024 (UTC)[reply]
Yeah, that’s the guy that some got CU’d for. I had only skimmed the case before, and didn’t connect the name and the person. FortunateSons (talk) 12:57, 24 June 2024 (UTC)[reply]
I would be interested to know which ones have checkuser blocks, to see how they were logged and categorized. Inconsistencies in logging and categorization can be a bit of a weak link, making potentially valuable information invisible. Sean.hoyland (talk) 12:50, 24 June 2024 (UTC)[reply]
I’m happy to provide the usernames of all who have send me messages through e-mail, assuming that’s permitted? FortunateSons (talk) 12:55, 24 June 2024 (UTC)[reply]
In the ArbCom case they said "Private evidence (including emails) can be sent to the Committee at arbcom-en@wikimedia.org". I think they still want people to let them know about inappropriate canvassing emails. I assume it's okay for you to list the usernames here. There's no loss of privacy. But what do I know? Very little apparently, so it might be worth verifying with someone who knows what they are doing first. Maybe ScottishFinnishRadish knows. Sean.hoyland (talk) 13:04, 24 June 2024 (UTC)[reply]
I’m going with an abundance of caution and will sent them through mail for now, if you find a categorisation error or two, I would have no issue with you doing with it what you wish (in line with policy).
If an admin later says that it was not appropriate, I would expect you to then delete the mail :) FortunateSons (talk) 13:23, 24 June 2024 (UTC)[reply]
No problem. Always happy to delete emails. Email received. Sean.hoyland (talk) 14:04, 24 June 2024 (UTC)[reply]
Perfect, if something productive comes of it, I would really appreciate a quick update either here or through mail. :) FortunateSons (talk) 14:05, 24 June 2024 (UTC)[reply]
I think I can probably see some of the blocked accounts...maybe... e.g. [1][2][3][4][5] Sean.hoyland (talk) 13:11, 24 June 2024 (UTC)[reply]
Hello, Sean.hoyland. Please check your email; you've got mail! It may take a few minutes from the time the email is sent for it to show up in your inbox. You can at any time by removing the {{You've got mail}} or {{ygm}} template.
