Content Relevance Evaluator

100% remote

Full Time

Contract/Temporary

Administrative

Austin, TX

1 - 2+ years

$44.14 - $44.83 / hour Pay Rate

Function

Administrative

location

Austin, TX

Experience

1 - 2+ years

pay

$44.14 - $44.83 / hour Pay Rate

Apply now

Responsibilities

Follow policies and confidentiality dictations to safeguard data and information

qualifications

Dependable, with a respect to confidentiality and policies
Great attention to detail

required qualifications

Reporting
Search Engines
Document Management
Python Scripting
Search Query

job highlights

3 Month Contract
100% Remote

Review query–document (or image, video, answer) pairs and evaluate their relevance using a standardized rubric.
Interpret user intent by analyzing queries, context, and search history to determine the most relevant outcomes.
Rate assistant answers for accuracy, descriptiveness, and completeness, including identifying when the system fails to provide a correct or useful response.
Conduct research on terminology and topics to confirm accuracy and contextual correctness of results.
Formulate queries (in natural language or KQL) to test relevance and surface appropriate documents.
Document evaluations clearly and consistently, providing feedback that can guide system improvements.
Collaborate with internal teams by sharing insights and suggesting improvements based on evaluation patterns.

Strong analytical and critical thinking skills with the ability to assess nuanced differences in relevance.
Excellent attention to detail and consistency in applying evaluation rubrics.
Strong research ability to validate information and terminology across diverse topics.
Proficiency in formulating effective search queries; familiarity with KQL is a plus.
Solid written communication skills to clearly document findings and provide feedback.
Ability to work independently and manage tasks with minimal oversight.
Comfortable using and evaluating AI/chatbot systems.
Preferred: Prior experience with relevance labeling, content evaluation, or quality rating.
Bonus: Basic scripting skills in Python to automate or streamline evaluation tasks.

To be considered for our current job openings, we kindly ask that you complete your registration by clicking the “apply now” button.

At Premier Talent Partners, we are a people-centered recruitment firm with a strong focus on Skills-Based hiring. We specialize in placing individuals in roles ranging from entry-level to VP level across various fields, including Customer Success, Sales & Marketing, Finance & Accounting, Administrative, HR, Operations, and Technology. Our expertise spans contract/temporary, temporary-to-permanent, permanent, and payrolling services, and we are dedicated to promoting equitable hiring practices on a national scale.

Please note that compensation for the specific role will be determined based on your unique skills and experience. If you have an interest in exploring other positions similar to this one, please be aware that compensation rates and benefits packages may vary accordingly.
We do consider applications from qualified candidates with prior arrests and conviction records, in accordance with applicable regulations.
It is imperative that you have the legal authorization to work in the United States, as we do not provide sponsorship.

Apply now