Rating the applications and CVs

There is autonomous Agent handling the recruitment process for applying the job. The Agent takes in the CV, helps the applicant to see what is missing and gives rating how the applicant CV applies to the open job position. Agent answers questions about the position and organization. If the Agent does not have answer, it asks from the hiring manager and updates the database. The final step is to rate the applications so that hiring manager could see immediately which applications matches the job position the best.

Human still makes the decisions – as EU AI Act states

I was thinking quite a bit where to put the rating logic in the architecture. I had three choices

1. Add the rating functionality in the Copilot Studio Agent Generative Answers
2. Create Agent Flow (or Power Automate) that independently rates the applications and CVs
3. Create a separate Agent that rates the applications and CVs

I chose to add the rating functionality in the Agent Generative Answers, since in that context I will have already the open job position description and requirements and the applicants CV content. If I would use flow, then I would need to get the job position requirements and description and use AI Hub and create separate prompt for that. I decided not to create another Agent since in this exercise I won’t be trying multi agent orchestration and want to test how to put all the functionality under one Agent.

I started with below prompt – already got answer where the LMM wanted to ask more details and needed to instruct not to ask more information.

"From knowledge source get 'Job Title' " & Global.JobPositionSelected.DisplayName & ". Get the 'Job Description' and 'Requirements' and compare them with applicant CV Content. Rate the application with once of these choices 'Does not match', 'Match somehow', 'Match', 'Strong Match' or 'Perfect Match' and give justification why the CV matches the open job position. Don't ask more details, here is the Applicant CV Content: " & Global.CVcontent

I also noticed that I need to save the generative answer into a variable so I could pass it forward later on.

I notitced that rephrasing the prompt I was missing the actual comparison. Need to add it back to the prompt and tested again.

"From knowledge source get 'Job Title' " & Global.JobPositionSelected.DisplayName & ". Get the 'Job Description' and 'Requirements' and compare them with applicant CV Content. Rate the application with once of these choices 'Does not match', 'Match somehow', 'Match', 'Strong Match' or 'Perfect Match' and give justification why the CV matches the open job position. Describe what is missing and, what is found in CV compared to job description? Don't ask more details, here is the Applicant CV Content: " & Global.CVcontent

But this ends up with reply that the LLM cannot help me. I need to rephrase the prompt once again to get it working. The testing is quite slow, since I need every time run the agent from start to get reply.

"From knowledge source get 'Job Title' " & Global.JobPositionSelected.DisplayName & ". Get the 'Job Description' and 'Requirements' and compare them with applicant CV Content. Rate the application with once of these choices 'Does not match', 'Match somehow', 'Match', 'Strong Match' or 'Perfect Match' and give justification why the CV matches the open job position. Notify also if there are some skills missing in the applicant CV Content which are asked in the 'Requirements'. Don't ask more details, here is the Applicant CV Content: " & Global.CVcontent

Manual testing is slow

Yes, now I have working prompt which gives answer that I’m happy with. Next I need to create the Agent Flow which extracts the rating and updates into the Dataverse row.

This means that you need to rephrase the prompt many times and test the Agent during the development continuously. I have not figured out yet a way to test separate action in the middle of a topic nor figured out how to record my Agent testing and rerun it like in Power Automate you can automate test with previous inputs.

Then I was thinking how to save the Dataverse column choice value from the Copilot Studio Agent. I did not want to try anything fancy since I was not sure would it even work. I decided to create Agent flow since I need to get the choice id with the value and Power Automate is my specialty.

I create one variable and used expression to get the correct term from the generative answer. Then added correct id for each choice. The rest of the Agent Flow is getting the Job Application with user name (List Rows) and then updating the value for the Rating column with the variable MatchLevel.

if(contains(triggerBody()?['text_1'], 'Perfect Match'), 5,

if(contains(triggerBody()?['text_1'], 'Strong Match'), 4,

if(contains(triggerBody()?['text_1'], 'Match somehow'), 2,

if(contains(triggerBody()?['text_1'], 'Does not match'), 1, 3

)

)

)

)

After that I need to call the Agent Flow in the Topic after getting the answer.

And after testing I see that the Rating column has value everytime any user has uploaded a CV.