PR-Agent — extension that adds AI chat to code reviews on GitHub

24 points by marsh_mellow 10 months ago | 15 comments
  • lvncelot 10 months ago
    I'm sorry if this makes me sound like a cranky old man but I find the idea of AI-generated PR description and reviews entirely misses the mark of what they're for.

    The PR description should contain the context that made a change necessary, which is something that can not be contained in the change itself and thus can't reliably be inferred from it.

    And if I request a PR review, I trust you, as my peer, to understand the problem my change is trying to solve and to judge whether it succeeds in doing so. If I wanted AI suggestions for it I can skip the middleman.

    I guess the "chat with the codebase" feature could be useful, but the auto-review feature just gives me very uncanny vibes. If someone would "review" my code by blasting it through an LLM I'd be ticked off.

    • Groxx 10 months ago
      Having seen / used / etc similar things: yes I fully agree.

      For people who already know the code well, it can catch things you didn't think about, and that's kinda nice. I suspect it has cost me/my team more time on wild goose chases than it has saved on real findings, but it might still be worth it.

      For others, I've seen it constantly mislead people, and convince them to make changes that look reasonable but lead to the PR being incorrect. Over and over and over. And the "look reasonable" part is particularly dangerous during code reviews - reviewers are less careful about things that look reasonable, that's a relatively normal and unavoidable thing for humans even though it's incorrect.

      When it's exposed to everyone, there's a pressure to follow and trust it, like existing linters. Private use is less bad, but still risky due to the plausible-change-review-negligence issue.

      ---

      Also PR descriptions in particular are borderline useless when written from an LLM, because they just summarize the changes (or mislead summary-readers about the change, when incorrect). So you're just littering your commit history with the equivalent of auto-generated blogspam. No thanks.

      • servercobra 10 months ago
        We use this feature and it's actually been pretty handy. I still require my devs to write their own PR description (because of course they should), but it does give a few suggestions for things to look into as they review. Occasionally it's pointed out a few things I might have missed. But at least half of the suggestions are obviously off base, so I can't imagine trusting it to actually do the review.
        • talrid 10 months ago
          Hi @lvncelot

          I am Tal, one of the developers of this extension.

          See our FAQ regarding your concerns about the PR description, and the review process: https://pr-agent-docs.codium.ai/faq/#answer1

          In general, PR-Agent is designed to assist, not replace, human reviewers.

          Many of the flows and processes we deployed are aimed at meeting this goal. We try not to overclaim, or overpromise. PR-Agent should not be the sole reviewer of a PR. We believe it can definitely enhance and improve the reviewing process.

          • nine_zeros 10 months ago
            > The PR description should contain the context that made a change necessary, which is something that can not be contained in the change itself and thus can't reliably be inferred from it.

            > And if I request a PR review, I trust you, as my peer, to understand the problem my change is trying to solve and to judge whether it succeeds in doing so. If I wanted AI suggestions for it I can skip the middleman.

            Are you talking about keeping meaningful processes around? Because companies are trying to get rid of humans and people altogether. They don't really know why PR reviews exist - only that reviews can take time - and they don't want to spend time. But can't remove reviews anyway - so just paper over it.

            • boredtofears 10 months ago
              I find AI summaries to be useful when writing PR's as a starting point. Having it generate a bulleted list summarizing the changes is a good way to make sure you've covered your entire PR (and it's easy to trim down unnecessary things). You can add context as needed.

              I'd never trust an unrevised version though.

              • johnnyanmac 10 months ago
                Yeah, thars the big issue. Really only takes one unengaged or lazy reviewer to not take the time to revise that AI output and it will ruin the entire Org. Honestly it's a theme for all LLMs: they are great as a guide but people inevitably use them as if they are a replacement. The companies marking it as such don't help either.

                As is, getting in revised LLMs responses seems like it'll be Gen Alpha's "let me Google that for you" swipe. Except at least Googling means you still need to put in leg work.

              • mistrial9 10 months ago
                your statement is clear without the first half sentence .. another way to characterize this is discussing the role of human people as team members versus robot support services.. whose role? whose benefit? from the human side.. what accuracy ? what costs (both hidden and overt) ? from the toolset side
              • teqsun 10 months ago
                Oh, Codium?

                My last company had codium on our internal codebases for PR review/summary. I found it neat, but I never saw it make any suggestions that surpassed linting. But as a lazy dev who didn't want to rewrite all the details I put in my commit messages when making a PR, I felt it did a pretty good job summarizing.

                Funny story: One time I jokingly asked it to rate the skill level of the programmer who wrote a certain PR (which was created by me). It was very complimentary of my skills, which I shared in the humor chat about how it was a very smart AI. The very next day I was laid off (unrelated, it was along with most of my team).

                • marsh_mellow 10 months ago
                  There's an open source version of this as well: https://github.com/Codium-ai/pr-agent
                  • aravindputrevu 10 months ago
                    Looks interesting at the same moment, I don't know why it should be a chrome extension and not a hook into my workflow like Coderabbit dot ai.

                    What I love with these tools, I'm getting productive. Although, I'd still like to do the review myself.

                    • gengstrand 10 months ago
                      Has anyone here tried out https://github.com/freeedcom/ai-codereviewer which I suspect is something similar only in GHA form?
                      • 10 months ago
                        • lucabetelci 10 months ago
                          Most of the reviews for this extension look like they were generated with LLM.
                          • nextworddev 10 months ago
                            This looks like a security nightmare
                            • mass_and_energy 10 months ago
                              Wow there really is nothing that people won't shoehorn AI into these days eh? What is the point of this? What is the net business benefit, other than "but, but it's got AI"?