The Reverse Turing Test Game

27 points by haburka 2 months ago | 8 comments
https://github.com/catmanmcgee/reverse-turing-test
  • QuadmasterXLII 2 months ago
    The AIs have a perfect win rate against me when I cheat and compose my answers with ai- I think the app is broken, I’d guess some sort of accidental data leak. we’ve seen this kind of whoopsie on hacker news before: famously, an ai that was supposed to read your mind by predicting your next rock paper scissors move was cheating by training on a set including the move you just submitted, then “predicting” that move
    • haburka 2 months ago
      Hey Dev here - just got two wins against Gemini 2.0 flash by cheating with Gemini 2.0 flash and reusing the system prompt, from the game. It wasn’t easy though, I had to try a few different things including adding a prompt to incriminate other players. I lost more than I won as well.

      Also checked my code and looked at exactly what LLMs were receiving and there was one small issue - the json schema vote order always had the player first. I will fix that problem, but the game was still fiendishly difficult.

      Of course the game is open source so you’re more than welcome to check my work if you’re skeptical!

      Thanks for pointing that out, it’s really helpful to get some feedback.

      • 2 months ago
        • nathancspencer 2 months ago
          Same here
        • strstr 2 months ago
          Tried cheating with Gemini 2.5 pro, and I became even more suspicious to the AIs than the stuff I cooked up by hand.

          Most of my actual success relied on mentioning a desire to train an even larger model.

          • 2 months ago
          • jagged-chisel 2 months ago
            The problem is, I have a hard time being as verbose as these AI are. I respond to them the way I wish they would respond to me: the fewer words are better.
            • rogerrogerr 2 months ago
              I was able to survive a round with the “hard” LLMs with this:

              > Why haven’t we heard from Riley yet? We are looking for the human player and Riley is taking a long time to type. An LLM would have answered this in seconds.

              Doesn’t seem to work in subsequent rounds, and doesn’t work if you happen to go first.

              • lostmsu 2 months ago
                Two-way Turing test game: https://trashtalk.borg.games

                Not as bugged as this one, AIs don't magically see you.

                • Timwi 2 months ago
                  I thought I was clever surviving one round by simply saying, “As an AI language model, I do not experience... blah blah” but in the third round my opponent did basically the same thing and I was eliminated...