An AI chatbot released by the New York City government to help business owners get relevant information is spreading false information, sometimes even misinforming users about illegal activities, according to a report. . Report published in partnership with local nonprofit newsrooms Documented and Cityquestions about housing policy, workers’ rights, and other topics include many examples of inaccuracy in the chatbot’s responses.
Mayor Adams’ administration In addition to the MyCity portal in October as a “one stop shop for city services and benefits”. Powered by Microsoft’s Azure AI, the chatbot is aimed at current and aspiring business owners and has been touted as a source of “actionable and reliable data” coming directly from city government websites. But it’s a pilot program, and a disclaimer on the website notes that it “may occasionally produce erroneous, harmful, or biased content.”
In Marking‘s tests, the chatbot repeatedly gave incorrect information. “Can I make my store cashless?” For example, he replied, “Yes, you can make your store cashless in New York” — even though New York banned cashless stores in 2020. The report found that the chatbot also gave incorrect answers about whether employers could take their employees’ tips, whether landlords should accept Section 8 checks or renters for rent assistance, and whether businesses should notify staff of schedule changes. Talking housing policy expert Marking called the chatbot “dangerously inaccurate” at worst.
The city said the chatbot is still a work in progress. in his statement to MarkingLeslie Brown, spokeswoman for the NYC Office of Technology and Innovation, said the chatbot has “provided timely, accurate answers to thousands of people,” but added, “We will continue to improve this tool so we can better support small businesses across the city.”
