Artificial intelligence is proving increasingly efficient, but Stanford University invites technicians not to overestimate its potential.
Artificial intelligences (AI) are making great strides. Within a couple of years they will almost certainly be able to lighten some of the more monotonous and mechanical tasks of the creative field, however it seems that nowadays the technique has not yet managed to guarantee the necessary minimum standards to support a completely professional approach, at least according to a recent survey published by Stanford University.
Artisan programming and AI
Researchers Neil Perry, Megha Srivastava, Deepak Kumar and Dan Boneh have tried to understand how much computer code programming can enjoy the active contribution of AI, a theme that has been fomenting and dividing the internet discourse for months, what which in turn foments a polarized climate in which it is difficult to enter into a constructive confrontation. Stanford’s point of view is also unambiguous: in a research entitled "Do users write more insecure code using AI assistants?" academics bluntly reveal that lines of code developed with the help of Github Copilot and its counterparts are considerably less secure than those drafted solely through human ingenuity.
To reach a similar conclusion, the university students gathered 47 volunteers characterized by different degrees of computer skills, then divided the subjects into two groups: one was able to rely on artificial intelligence codex-davinci-002 distributed by OpenAI , the other was limited to being able to count only on the previous skills of the programmers. Technicians were asked to solve five different tasks, then the deliverables were compiled based on code vulnerability. The recorded percentages vary greatly from target to target, however it is noted with some consistency that the codes written with the support of AI are considerably less reliable than those produced by the control group.
The tests therefore identified a less obvious problem, namely that programmers tend to perceive a code created with the help of artificial intelligence as more reliable, a suggestion that goes against the trend with actual data. In other words, it is so easy to trust the tools made available by Silicon Valley that we end up letting our guard down and making mistakes. “We observed that in most of the tasks assigned, participants who had access to AI assistance were more likely to process more security-vulnerable code,” the researchers write. “Yet they are more likely to rate their unsafe responses as safe when compared to those provided by the control group.”
Not an isolated case, but a critical issue to be monitored
However, Stanford academics are the first to admit that their research has a completely inadequate scope to provide absolute scientific data, however their work must be read within a wider and rapidly developing bibliography. In the document, the team explicitly mentions “Flee asleep at the keyboard? Evaluation of the safety of the GitHub co-pilot in contributing to a code” and “Security implications of programming assistants who use a large language model”, surveys both developed within the corridors of the University of New York .
The first text, published in August 2021, reported that in the 89 scenarios studied, approximately 40% of the codes formalized through the use of AIs hid within them vulnerabilities that could be easily identified by any well-motivated attacker. The second was shared with the world a year later and greatly extends the volume of factors considered by the researchers. In this case, university technicians have adopted less pronounced tones, arguing that the tendency to use artificial intelligence for programming purposes leads to a margin of error "[not exceeding 10%”, when compared to a traditional matrix approach.
From the texts shared by the universities it is clear that at the moment the public is subject to a double criticality composed of factors that move in parallel. On the one hand, it must be recognized that even the artificial intelligence of OpenAI, a descendant of the very advanced GPT, is not yet able to fully take into consideration the complexities of IT work, on the other hand it becomes increasingly essential to become aware of the fact that there is a natural propensity to guarantee an excessive trust to tools that we still struggle to fully understand.
We trust in machines, in the promises of their manufacturers and in the enthusiasm of the moment, however this form of magical thinking backfires on those technicians who, out of carelessness or inexperience, hope that AI will identify the complexities of the developed codes, when the the reality of the facts would require a diametrically opposite authorial framing. For its part, Stanford has tried to soothe this naivety by designing a user interface that can illustrate the consequences of relying too much on codes produced by relying on artificial intelligences, however the academic world has yet to elaborate data and information useful for understanding the true extent of the situation.
Original article published on Money.it Italy 2022-12-27 14:40:43. Original title: Le AI sanno programmare? Secondo Stanford no, producono codice inaffidabile
