Consultant for Generative AI Testing and Evaluation

Job Posting Organization:
Mercy Corps is a leading global organization founded on the belief that a better world is possible. Established to address humanitarian crises and support communities in need, Mercy Corps operates in over 40 countries worldwide. The organization is dedicated to empowering individuals and communities to overcome adversity and build resilience through innovative solutions and partnerships. With a mission to drive sustainable program impact, Mercy Corps leverages technology and data to enhance program quality and effectiveness, ensuring that its global workforce has the necessary infrastructure and support to fulfill its critical missions.

Job Overview:
The consultancy role focuses on supporting the IT department by designing and implementing a user-testing protocol for Generative AI tools within Mercy Corps. The consultant will evaluate existing tools and identify enterprise-level use cases for Generative AI, specifically assessing the utility of the Digital Library Chatbot against these use cases. The consultant will also develop and execute a comparative testing format for at least two commercially available Generative AI solutions, contributing to the organization's goal of enhancing program impact through technology.

Duties and Responsibilities:
The consultant will be responsible for defining a clear testing process for enterprise AI tools based on previous work on Generative AI testing frameworks. They will conduct both guided and independent testing on the Digital Library Chatbot and recommend a procurement strategy for testing commercially available enterprise Generative AI solutions such as CoPilot, ChatGPT, and Amazon Q. Additionally, the consultant will perform a comparative analysis of these enterprise solutions, including financial comparisons, and provide a detailed report on the results of user testing and recommendations for enterprise-wide rollout.

Required Qualifications:
Candidates must possess a Bachelor's degree or higher in a relevant field, with direct subject matter experience considered in lieu of formal education. Strong experience in technology evaluation at the enterprise level, participatory research, and the application of Generative AI in aid and development contexts is essential. A minimum of 3-5 years of relevant experience is required, including at least one year specifically related to IT in aid or development organizations or Technology for Development.

Educational Background:
A Bachelor's degree or higher in a relevant field is required. Candidates with substantial direct subject matter experience may be considered even if they do not meet the educational requirements. This flexibility allows for a diverse range of applicants with practical experience in technology evaluation and Generative AI.

Experience:
The position requires 3-5 years of relevant experience, with at least one year specifically related to IT in aid or development organizations or Technology for Development. This experience should include a strong background in technology evaluation and the use of Generative AI in practical applications.

Languages:
While the job description does not specify mandatory languages, proficiency in English is likely essential given the nature of the consultancy and the requirement for communication with various stakeholders. Additional language skills may be beneficial but are not explicitly mentioned as a requirement.

Additional Notes:
The consultancy is projected to start in early May and conclude on June 30, 202

• The level of effort is estimated to be approximately 30-45 days within a 60-day period, with flexibility based on the consultant's daily rate. Working hours are flexible, but some overlap with US Mountain time zone is required during the week. The consultant will report to the Director of Data Science in the Technology for Development team and will collaborate closely with IT and other Mercy Corps elements.