This service, which was unveiled at the AWS re:Invent conference, is presently in preview.
Model selection and evaluation, according to AWS VP of Database, Analytics, and Machine Learning Swami Sivasubramanian, “is something that’s repeated periodically, not just at the beginning.”
Involving more people in the evaluation of AI models is the aim.
Elements of the Bedrock model evaluation
The two components of the Bedrock model evaluation process are automated and human.
The robustness, accuracy, and toxicity of a model can be evaluated by developers for tasks like text classification, Q&A, summarization, and text generation.
On the basis of these assessments, the system then generates a report.
Users can work with their own team or an AWS human evaluation team for human evaluation; AWS provides specialized pricing and timelines for those collaborating with its assessment team.
Model evaluation using unique datasets
AWS offers test datasets, but also lets users use their own data for model evaluation.
This aids companies in comprehending the models’ performance in their particular use cases.
Vasi Philomin, the vice president of AWS for generative AI, said that a better grasp of model performance can direct development more successfully and assist businesses in determining whether models adhere to responsible AI standards prior to deployment.
Additional metrics are detected by human evaluation.
Metrics like empathy and friendliness that automated systems might overlook can be found by human evaluators.
While benchmarking models is not required of all AWS customers, it may be helpful for those who are unsure about which models to use.
AWS will only charge for model inference used in evaluations during the preview period.
Benchmarking on Bedrock is intended to give businesses a means of measuring a model’s influence on their projects, not to evaluate models in general.
Additionally, Titan Image Generator has been revealed.
Titan Image Generator is another image-generation tool that Amazon has released.
AWS customers can now grab it in preview on Bedrock, the company’s AI development platform.
Titan Image Generator is one of the generative AI models in Amazon’s Titan lineup. It can both create new images and modify pre-existing ones by utilizing text descriptions.
This places it against competitors such as OpenAI, Microsoft, and Google.