Could AI judge a Daily Challenge?

Why I wondered If you enter challenges on BudgetPixel, you'll know the ranking can still feel a little like a lottery. I've reached the point where I'm fairly detached from where I finish. That isn't because I don't care about my work. It's because I don't think the voting system is a reliable way to judge it. Here's how it works. Voters, who don't need to have entered the challenge, are shown two images side by side and pick a winner. They can read the challenge theme and description, but there are no criteria beyond that. The level of detail in the description has varied a lot from challenge to challenge. At one stage the ranking used an ELO-based algorithm. Now it's a simple win rate. People still complain about it. Some complained about the ELO system, and many complain about where they finished. There's also the possibility of tactical voting: deliberately not picking images you think might do well, to improve your own chances. With credit prizes at stake, plus the satisfaction of a top 3 or top 20 place, the temptation is there. My view is simple. The majority pick what they like or what they expect. In a split-second moment, forced to choose between two images, the end result is unlikely to produce an objective ranking of the entries. The idea: let AI be the judge I've wondered what a better system would look like. If you want a ranking based on user opinion, the best option is probably for every voter to pick their own top 20 from all the entries, in order. But that's time consuming, and it's still open to manipulation. Then it struck me that BudgetPixel already uses AI agents. Agents moderate posts, and there's the built-in AI assistant, Ava, that users use to configure a companion . The owners are clearly up to speed with what modern AI tools and models can do. So why not a Challenge Ranking agent? Models like Claude can look at images and make judgements against stated criteria. An agent like that would apply the same criteria to every entry, produce a complete ordering, and would be entirely impartial. I wondered whether it would work in practice, so I tested it on a challenge that had already finished, and compared its ranking with the actual results. How I set up the test I picked a completed challenge. The brief was: Memory Tailor Create your most imaginative scene of memories being stitched into clothing, fabric, or a living garment — No Level4 Adult Content in submission. I screenshotted each entry and gave Claude the lot, without the theme at first. It worked out the theme from the images alone, which was a promising start. It also spotted two duplicates (my error when screenshotting) and flagged one entry that was an ordinary photo of a person, nothing to do with the theme. That one was a genuine entry, so it stayed in. The filenames ran A to Z, then AA to XX. What Claude didn't know was that I'd named them so that A to T were the original top 20, in finishing order. So it was ranking blind. The criteria After I shared the theme, we agreed on five criteria: 1. Theme interpretation and originality 2. Storytelling and emotional impact 3. Composition and lighting 4. Technical quality (AI artefacts, hands, garbled text) 5. How well the memories are integrated into the fabric A few ground rules came out of the discussion: • Captions are optional when submitting an entry, so they only counted where they added to an entry's impact. • Image quality was limited. These were screenshots, not the full-resolution files. That would make fine detail hard to judge, but obvious problems like bad hands and garbled text were still visible. A real agent on the site would have the originals. • No identification of real people. Each entry was judged purely as an image. • The output was just a ranked list, first to last. I didn't want justifications at that stage, only the order. One thing Claude raised up front that's worth repeating: it isn't free of taste. It said it would tend to reward a clear concept and a coherent composition, and that consistent bias is still bias. The difference from crowd voting is that the reasoning behind every placement can be shown and checked. First pass: analysing the rankings The following table presents the original top 20 and Claude's top 20, comparing the two rankings. The left side is the original top 20 and where Claude put each entry ("Not Placed" means outside its top 20). The right side is Claude's top 20 and where each finished originally. Only 10 of Claude's top 20 were also in the top 20 as judged by the voters. Its top 3 was completely different, and included one entry the voters hadn't placed at all. The table below reflects the how the position of original top 20 images that Claude placed in the top 20 moved in the rankings, either up or down. Of the 10 entries in both top 20s, Claude ranked 6 higher and 4 lower. None kept the same position. Claude's winner was H: a woman in a kimono at a loom, weaving the folk tale of the crane wife with feathers, her crane shadow on the shoji screen behind her. It finished 8th with the voters. P, Athena at her loom with the Fates drawing threads of memory from a sleeping giant, made the biggest climb, from 16th to 4th. G, a tailor working on a gold gown with faint memory scenes in the embroidery, had the biggest fall, from 7th to 18th. Objective assessment of Claude's rankings I scanned a good number of Claude's placings and was broadly happy with them. One stood out, though. Claude's 2nd place, CC, shows a man pulling open his jacket to reveal a glowing portal, with his younger self running out of it into a spill of toys. It's excellent and highly creative, but I wouldn't have placed it. There's no tailor and no stitching. The jacket is a doorway to memories rather than something memories are sewn into. I'd consider this to be off-theme. When I raised this, Claude agreed it had let originality and impact outweigh theme fit. The brief asks for memories "being stitched into" a garment, and CC doesn't show that happening. If I were running this as an admin, this is exactly where I'd step in. The Agent does the heavy lifting, then the human-in-the-loop (such an AI cliche now) checks the top 20 and and either accepts as-is or provides the Agent feedback to trigger a second pass. So that's what I did. I asked Claude to rerank with one extra rule: a tailor, or the act of making the garment, must be shown in some form. Claude treated the rule as a pass or fail gate. Entries that fail it now rank below every entry that passes, and the order is otherwise unchanged. Three entries dropped out of its top 20: • CC: the portal jacket. • RR: an elderly couple under a finished quilt of their life together. Lovely, but nobody is making anything. • JJ: a woman wrapped in an embroidered kimono, with no stitching shown. Three came in to replace them: • C (originally 3rd) • II (a neon cyberpunk memory atelier) • Z (a spider seamstress working on the Red Queen's dress) Below are the revised tables. The overlap rose from 10 to 11. Of those 11, Claude ranked 6 higher and 5 lower. Claude's top 3 is now H, D and P. D, a father stitching his daughter's wedding dress with scenes from her childhood, was 4th with the voters. C, a woman stitching a jacket of memory patches, makes the top 20 but takes the biggest drop, from 3rd to 18th. What did I learn? Staying creative while staying on theme is hard My own entry, XX, wasn't placed by the voters or by Claude. It shows a grandmother stitching a glowing quilt of family memories while two grandchildren watch. I'd tried to play on nostalgia with a painted style. At the time I had wondered whether a grandmother and a quilt were a stretch from a "tailor", but felt it was close enough to the theme's intent. Claude's view was that it wasn't a stretch at all. The brief explicitly allows "fabric", and the stitching is clearly shown, so on theme fit it did better than CC. The problem was originality. A quilt of memory scenes was the most common interpretation in the whole challenge, and a dress covered in memories wasn't far behind. Nostalgia was where most entrants went, so it ended up as common ground rather than something that set an entry apart. XX came 26th in the first pass and 22nd after the revision. I can accept that. The original voting placed it 21st. Is a weaver a tailor? When I voted, I accepted weavers as tailors. Claude did too, and it matters a lot to its ranking. Its new winner (H, the crane wife) is a weaver, and so are P (Athena), Q (the Norns) and K (a man at a memory loom). Rule weavers out and four of its top 10 fall away. I think weaving fits the spirit of the brief, since weaving is how fabric gets made. But you could argue it either way, and that's the point. The edge cases need deciding before judging starts, not after. Does glamour get votes? This one is my own opinion rather than a trend I can prove. Some entries seem to me to play on the sexual objectification of women, more so in some challenges than others. When it feels like an image has been made to appeal to people enjoying a glamorous woman, with prominent cleavage and revealing attire, I tend to vote against it. Two entries in this challenge lean that way: • O: a woman in lingerie and stockings under a memory robe. Voters placed it 15th. Claude had it 46th in its first pass. • F: a woman at a sewing machine with visible cleavage. Voters placed it 6th. Claude had it 32nd in its first pass. Claude didn't apply a specific penalty for this. Both scored low for other reasons: O shows no stitching, and F's memory quilt was one of the common treatments. Two entries don't make a pattern, but I suspect pairwise voting rewards that kind of eye-candy. In a split-second choice between two images, glamour can grab attention in a way theme fit doesn't. A rubric criterion that every element must serve the theme would handle it neutrally, without anyone having to moralise. So, would it work? I think this shows an agent-based ranking is possible. Claude produced a full, consistent ordering of the entries, and when I checked its placings I was happy with them. Where I disagreed, as with CC, one round of feedback fixed it. To be fair, it doesn't prove Claude's ordering is more correct than the crowd's - this is not a maths exam with right and wrong answers. But it does reflect a more consistent ranking against a uniform set of criteria that can be explained. That's still a big step up from a result nobody can account for. If BudgetPixel ever tried this, here's what I'd suggest, most of it from my back-and-forth with Claude: 1. Publish the rubric with each challenge. Entrants know what's being judged, and placings can be questioned on specific grounds. 2. Settle the edge cases in the brief. Must a tailor be portrayed in the act of sewing? Does a weaver count? Decide before judging, not after. 3. Add a criterion that every element serves the theme. That deals with gratuitous content (sexualisation, celebrity likenesses, brand logos) without moralising, so entrants know where the line is. Though there will always be some subjective judgement. 4. Have a human review the top 20. The agent does the heavy lifting, and a person catches judgement calls like CC before results go out. 5. Consider run the agent more than once, or use more than one model, and average the results. That shows how stable the ranking is rather than relying on a single pass. 6. Provide each image with a short assessment against the criteria. A line or two on why an entry placed where it did would make things feel fairer, especially for the people who don't place, which is where most complaints come from. 7. What about the voting credits? Simply drop them and award credits for entering. The number of entrants is fairly consistent, so a suitable number of credits could easily be set. While I think the results of this experiment are sufficiently positive to warrant a trial period, I'm not optimistic that the owners will choose to run with this. I believe they prioritise major enhancements that may may have greater impact on wider social media and brings in new users, which I can appreciate. While implementing this means to judging the Daily Challenge may appeal to regular participants, I think the benefits are relatively small in the bigger scheme of things. Regardless I found the experiment intriguing. I can't say I am surprised at the results. I am not surprised that Claude was able to establish a set of criteria and apply them objectively. I am also not surprised that Claude's rankings were quite different to the original voting. I'd be interested to hear what you make of it.