At Partly, we are committed to delivering the highest quality data in the industry. Our custom inter-annotator application is a cornerstone of this commitment, ensuring accuracy, consistency, and continuous improvement across our data standardisation.

‍

Ensuring Quality Through Collaboration

Our team of Quality Assurance (QA) professionals work collaboratively to standardise complex data. The inter-annotator application monitors how often these experts agree on their assessments (agreement rate) and how much of the data has been re-evaluated by multiple experts (coverage). This dual focus allows us to maintain a comprehensive overview of our data quality.

‍

Inter-Annotator Agreement and Coverage

Agreement Rate

The agreement rate is a critical metric that indicates the level of consensus among our QA team members. A high agreement rate means:

• Consistency in Annotations: Ensures that data is interpreted uniformly across the team.
• Reliability of Data: Provides confidence in the accuracy of our QA outputs.
  ‍

Coverage

Coverage assesses how much of our data has been re-examined. Our focus on coverage guarantees that:

• Comprehensive Quality Checks: All relevant data undergoes thorough review.
• Detection of Anomalies: Increases the likelihood of identifying and correcting any discrepancies.
  ‍

Robust Disagreement Review Process

Disagreements among annotators are natural in complex QA tasks. We turn these instances into opportunities for improvement through a structured review process:

• Identification of Discrepancies: Our application flags any inconsistencies for prompt attention.
• Resolution and Alignment: Our vehicle specialists review the opinions and refine guidelines to enhance future agreement.

This process not only resolves current issues but also strengthens our team's cohesiveness and expertise.

‍

Continuous Improvement and Quality Enhancement

Our commitment to quality is an ongoing journey. We continuously strive to elevate our processes by:

• Updating Training Programs: Incorporating insights from the disagreement reviews to enhance team skills.
• Refining Annotation Guidelines: Adjusting protocols to reflect the latest standards and team consensus.
• Monitoring Progress: Regularly assessing agreement rates and coverage to track improvements over time.

‍