Mozilla has introduced two new features on MDN that make use of artificial intelligence, to a mixed response. Should we trust AI Help and AI Explain, both of which are powered by GPT-3.5?

MDN is the free-to-use resource that documents the open web platform and provides developers with the information they need to build projects on the web platform. MDN Web Docs was originally created by Mozilla, and is now managed and maintained by Open Web Docs.

Writing about the new, AI-powered features, Steve Teixeira, Mozilla's chief product officer, explained that Mozilla and MDN:

"see a path forward to “overlay” useful AI-driven helpers on top of MDN’s canonical web dev documentation to aid developers in new ways."

This was the reasoning behind MDN launching AI integrations with its body of web developer reference documentation, specifically as two features, AI Help and AI Explain, both powered by GPT-3.5.

AI Help lets MDN readers ask questions with a conversational interface, and the tool offers concise answers with MDN articles related to their questions for contextual help, while AI Explain can be used to explore and understand code blocks and parts of code examples embedded in MDN documentation pages, describing the underlying purpose and behavior of the code.

Teixeira said that:

"Our internal work on these tools has us hugely optimistic about their potential to save developers tons of time as they seek answers and learning resources."

However, when the tools were rolled out, users responded pointing out cases where the answers given were factually incorrect. One response on GitHub gave an example of an AI Explain explanation of how the grid property works, pointing out that the answer incorrectly says the code creates two columns whereas it actually creates one, and misidentifies areas as columns. The user continues:

"it seems like this feature was conceived, developed, and deployed without even considering that an LLM might generate convincing gibberish, even though that's precisely what they're designed to do."

Mozilla has described this and similar feedback as "enormously helpful". They say the MDN team is now investigating these bug reports, but they've:

"elected to be cautious in our approach and have temporarily removed the AI Explain tool from MDN until we've completed our investigation and have high-quality remediations in place for the issues that have been observed."

The problem with GPT-3.5 is that it was trained on data up to September 2021 - and 20 months is long enough for it give outdated answers. Add to this the tendency of all LLM's to "hallucinate" - make up answers when they don't have appropriate information and elaborate the answers when the do, and you can see why you need to double check any information supplied as responses to questions. Using AI-based tools to summarize articles and to improve the grammer and syntax of written documentation shouldn't be beset with such problems and should be helpful, especially when working with documention that isn't in your native language.

The news that the plan is to continue to make use of AI-based tools on MDN continues to receive criticism, with respondents suggesting Mozilla is more interested in making money from MDN content than in ensuring the answers provided are accurate.

More Information

Mozilla Statement On AI On MDN

Related Articles

MDN Plus - A Premium Service

MDN Web Docs Has New Platform and New Funding

MDN Web Docs Call For Participation

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info