How to vet AI tools
Questions to ask when vetting AI tools
With so many new tools, how do you know which ones can be trusted or valuable?
Can you find the name of the AI company, developers, and or leaders? Look them up. Can you find reviews of the AI tools?
What problem does the AI tool solve?
For any tool or application to be useful, it must provide a solution to a problem.
Ask the following questions;
Who faces the problem in the organization?
Is it a problem that can be solved for many in the organization or just one individual or team?
Are there resources being used to solve the problem without AI? If yes, at what price? (looking for cost efficiencies here)
What value comes out of solving the problem e.g. monetary benefits, time savings, mental load, staffing, etc?
Can the problem be solved efficiently even without AI? If yes, how much time is that effort without AI?
How much time could the AI tool save individuals or teams?Is essential data accessible?
AI applications will need some level of data. This raises the question of where the data will come from, and whether the data is ready for use or needs to be cleaned up. Ask yourself, what data you will allow to be accessible to the AI? Be mindful of intellectual property and privacy data. Understand how the AI will use your data.What does the AI tool or application cost?
The app might be offered as a one-time purchase or a subscription.
Find out how much expertise and resources you will need to run, maintain and improve the AI solution.How does the application deploy AI?
Many companies have added AI as a buzzword to their products. Is the tool browser based or does it need to be downloaded and run on your computer? Be cautious if it needs to be downloaded or run on your computer. If browser-based, what does it need access to?
When this is unclear, follow up with the developer or company and ask them to provide this information.What features does the product have?
The features of an AI app or website are key in delivering value to your organization. You need to test the features of the AI product and see if they work – the ‘try before you buy’ rings true here. Start small before rolling out to the full organization.
You can do this if the app or website is online. Sometimes, the AI company might need to set you up with a demo account so you can access the features.
Integrations and APIs are key aspects to look out for when cautiously rolling out a new AI tool in your organization.
Understand the installation and deployment process, and how it plays out in real-world applications.What are the results achieved and what is the accuracy of the AI tool?
Find out about the successful and failed deployments of the app by asking the company or researching the company. If the app has been in the market for a while, you’ll also expect to see client testimonials, ratings, and case studies.
The absence of testimonials, ratings, and case studies doesn’t necessarily mean the app doesn’t work. However, cover your bases by asking or researching the company if they have any case studies or companies you can speak with that have rolled out the AI application in their organization.
No AI is 100% accurate. That’s why it’s important that you have an acute understanding of how accurate or robust the AI tool is in predicting outcomes or providing recommendations.
The questions above are a great starting place when evaluating AI tools and applications.
Make sure your company has an AI policy and educate your employees on the good and bad of using AI tools. This is important for protecting the organization from a potential breach.
Recent AI News
OpenAI rolled out Custom Instructions for ChatGPT https://openai.com/blog/custom-instructions-for-chatgpt. This is good news for getting ChatGPT to remember your writing preferences/style.
Go access Custom Instructions:
On web, click on your name → Settings → Beta features → opt into Custom instructions. Custom instructions will appear in the menu when you click on your name going forward.
On iOS, go to Settings → New Features → turn on Custom instructions. Custom instructions will appear in settings.