Someone at work accidentally enabled the copilot PR screening bot for everybody on the whole codebase. It put a bunch of warnings on my PRs about the way I was using a particular framework method. It’s suggested fix? To use the method that had been deprecated 2 major versions ago. I was doing it the way that the framework currently deems correct. A problem with using a bot which uses statistical likelihood to determine correctness is that historical datasets are likely to contain old information in larger quantities than updated information. This is just one problem with having these bots review code, there are many more. I have yet to see a recommendation from one which surpassed the quality of a traditional linter.

which uses statistical likelihood to determine correctness is that historical datasets are likely to contain old information in larger quantities than updated information. They should make some kind of layered models, where the user sets weight to layers. But in any case, this is not what I necessarily meant, just that a big project relying upon unpaid maintainers is flawed, especially when somebody makes real buck on it. There have been plenty of cases of state actors putting in backdoors. Those were human, most likely, and not some bots.

Or, hear me out, we can acknowledge that the quantity of information and experience necessary to review code properly far exceeds the context windows and architecture of even the most well resourced LLMs available. Especially for big projects. You can hammer a nail with the blunt end of a screwdriver, but it’s neither efficient nor scalable, even before considering the option of choosing the right tool for the job in the first place.