Morgan Stanley uses AI evals to shape the future of financial services

By integrating GPT-4 into their workflows, Morgan Stanley Wealth Management has transformed how financial advisors access the firm's extensive knowledge base and respond to client inquiries. Currently, over 98% of advisor teams actively utilize AI @ Morgan Stanley Assistant, an internal chatbot designed to answer financial advisors' questions, facilitating seamless internal information retrieval.

The implementation of AI in financial services required a solid foundation of trust in the technology's ability to deliver significant value while adhering to the firm's stringent quality and reliability standards. To address this, Morgan Stanley established an evaluation framework to assess every AI use case prior to deployment. These evaluations measure model performance against real-world scenarios and guide ongoing improvements through expert feedback.

The initial goals for their AI use cases included faster information retrieval to save advisors time, automation of repetitive tasks such as summarizing research reports, and enhanced insights tailored to client needs. The team conducted summarization evaluations to assess GPT-4's effectiveness in condensing large volumes of content into concise summaries, allowing for prompt refinements and improved output quality.

Morgan Stanley's eval framework is dynamic, evolving as the team gains insights. They introduced translation evaluations for multilingual clients and collaborated closely with OpenAI to refine retrieval methods, ensuring the AI could manage an expanding document library. This evolution has significantly increased the number of questions the AI can effectively answer, enhancing the overall efficiency of advisor-client interactions.

Building on the success of AI @ Morgan Stanley Assistant, the team launched AI @ Morgan Stanley Debrief, a meeting summary tool powered by Whisper and GPT-4. This tool converts Zoom recordings into actionable outputs, such as client notes and draft follow-ups, which are seamlessly integrated into CRM systems. Advisors can review and adjust AI-generated outputs, ensuring a balance between automation and human oversight.

To meet the rigorous compliance standards of financial services, Morgan Stanley incorporated quality assurance into their eval framework, conducting daily tests to identify potential weaknesses. OpenAI's zero data retention policy further addresses security concerns, ensuring the confidentiality of Morgan Stanley's proprietary data. The focus on quality and reliability has resulted in trusted solutions that employees actively use, with nearly all advisor teams adopting AI tools daily, leading to increased engagement and efficiency in client interactions.