Kevin Veigas

AI data operations · Evaluation · Technical QA

Human data for AI, measured — not assumed.

I’m Kevin Veigas. I run annotation, review and evaluation operations for AI, and build open-source tools that show a team which labels to trust, where review effort pays off, and when a dataset is ready to ship.

See the work Get in touch

Based in Mangaluru, India · Authorized to work in Canada · Open to remote roles worldwide

Live simulation

Adaptive label collection

Labels collected
0
At 5 labels per item
0
Saved
—
Settled
0
Expert review
0
  • Agreeing label
  • Dissenting label
  • Settled
  • !Expert review
An illustration of DataQual’s stopping rule. Each item keeps asking annotators for labels until a worker model is 95% confident (minimum two); items still contested after five go to an expert. Measured results are under DataQual.
  • 15,000+computer-vision images validated at 99%+ dataset quality
  • ~85% → 93%annotation accuracy after a calibration program I ran
  • 15annotation team members led and coached
  • 19production AI systems monitored in Grafana
  • ~30%shorter QA and rework cycle
  • ~35%faster issue response through drift monitoring

From my AI data operations contract at Maneva Inc., 2025.

Selected work

Two systems for the human-data loop

DataQual decides which labels to trust and when to stop collecting them. OpsPilot runs the campaign that produces them. Both are open source, tested, and published with the evidence behind every claim.

01 Annotation quality Python · FastAPI · React

DataQual

Evidence-driven annotation quality, review prioritization and adaptive label collection.

Turns raw multi-annotator labels into agreement statistics with intervals, consensus under different assumptions (majority vote vs. a from-scratch Dawid–Skene worker model), annotator reliability with credible intervals, explainable disagreement flags and a ranked review queue. Its newest module decides, item by item, whether to stop paying for labels, ask for more, or escalate to an expert.

  • 25–63%fewer labels at 7–9 labels per item, within 0.8 accuracy points (exploratory)
  • +27.4 ptsaccuracy over majority vote when some annotators are adversarial
  • 100%label parity with the Crowd-Kit reference on a real benchmark
Live demo Source

Evidence · 12 synthetic scenarios · 5 seeds each

How many labels does each item really need?

Redundancy
Stop at confidence
  • Adaptive: stop when confident
  • Collect every label
Show the data

Labels per item, averaged over seeds 100–104 (simulator 1.1.0). Source: docs/evidence/adaptive-collection.

DataQual's Label collection view: 80 of 100 items settled, 20 need more labels, none need an expert, with a table of the most uncertain items.
The Label collection view: settled items stop costing money; the rest show exactly how many more labels they need.
02 Campaign operations Python · FastAPI · React

OpsPilot

An operations cockpit for human-data campaigns, from intake to a defensible delivery.

Covers the loop a campaign manager owns: calibration-gated eligibility, quality-aware allocation with a recorded reason, a 10-state task machine, keyboard-first QA, escalations, a Monte Carlo completion forecast and five delivery gates. Each operation commits atomically with its audit record. The API is idempotent, so a client can safely retry after a timeout.

  1. Intake
  2. Calibrate
  3. Allocate
  4. Execute
  5. Adaptive QA
  6. Escalate
  7. Deliver
  • ½the undetected errors of uniform review at the same review budget (1.8% vs 3.5%)
  • 43%fewer reviews than uniform review for the same delivered quality
  • 40 / 40synthetic workforces where adaptive QA beat uniform review at the same budget
Live demo Source

Evidence · 40 paired synthetic workforces · 12 annotators, 10 days each

Review where the errors are

Trust-based QA spot-checks proven annotators at 5% and reviews struggling ones in full. Compared with reviewing everyone at the same rate, it ships half the errors for the same effort.

  • Adaptive QA (dot = one world; large dot = mean)
  • Uniform review (line = expected; dots = measured)
Show the data

Synthetic workforces: a quarter of annotators weak (mean accuracy 0.85), the rest strong (0.97); perfect reviewers assumed. Source: docs/evidence/ADAPTIVE_QA.md.

OpsPilot's Today view: a next-best-action card telling the manager to resolve one open escalation, campaign health marked critical with reasons, and a completion forecast.
The Today view leads with one next action and the evidence behind it, not a wall of charts.

Approach

How I work

  1. 01

    Evidence before claims

    Every result ships with its method, seeds and source data, so anyone can reproduce it and see exactly what it does and doesn’t show.

  2. 02

    Make uncertainty visible

    Intervals, posterior confidence and an explicit “not enough evidence” state, instead of one opaque quality score that hides how little is known.

  3. 03

    Design for the operator

    Next-best actions, plain-language reasons, keyboard-first review and an audit trail a manager can defend to a client.

Experience

Where I’ve done this work

  1. Jun 2026 – PresentRemote

    Human Data (BizOps) · micro1

    AI training operations and data review

    • Build multi-application business workflows across Excel, PowerPoint, Word, Teams and Outlook as realistic training tasks for frontier AI agents.
    • Review submitted workflows against detailed rubrics, verifying UI states, task completion, selections and full-desktop context.
    More detail
    • Identify workflow inconsistencies, task defects and rubric failures, documented as structured reviewer feedback.
    • Work as both contributor and reviewer, iterating captured workflows on evaluator feedback to improve consistency.
  2. 2026 – PresentRemote · Fellowship

    AI Training Fellow, Project Seal · Handshake AI

    AI evaluation and adversarial research

    • Design adversarial prompts that expose reasoning failures in frontier models, focusing on information buried in tables, PDFs and images, and on conflicting authoritative sources.
    • Document each failure pattern in structured, evidence-based formats for the research team’s evaluation pipeline.
    More detail
    • Prioritize novel, high-signal failure modes over repeats of known weaknesses.
  3. Freelance · Project-basedRemote

    Egocentric Robotics Annotation Coordinator · Outlier AI

    Egocentric video annotation for robot learning

    • Annotate first-person (egocentric) video of people performing physical tasks to build robot-learning datasets: segmenting demonstrations into action phases, labeling hand–object interactions and contact, and writing natural-language task descriptions.
    • Apply detailed project guidelines to edge cases such as occluded hands, ambiguous action boundaries and failed attempts, flagging uncertain frames rather than guessing.
    More detail
    • Check annotations for temporal accuracy and guideline consistency, turning recurring issues into clearer labeling decisions.
    • Engaged project by project, as robotics annotation work becomes available.
  4. Aug – Oct 2025Remote, Canada · Contract

    AI Data Operations Specialist · Maneva Inc.

    Annotation quality and production AI monitoring

    • 15,000+ images validated
    • 99%+ dataset quality
    • 85→93% annotation accuracy
    • 19 systems monitored
    • Ran cross-functional calibration and quality-alignment workflows that lifted annotation accuracy from about 85% to 93% and cut QA cycle time by about 30%.
    • Validated 15,000+ computer-vision images with dataset checks and anomaly-detection routines, sustaining 99%+ dataset quality.
    More detail
    • Monitored 19 deployed AI systems through Grafana dashboards with pipeline-health, performance and blocker reporting, contributing to about 35% faster issue response.
    • Led and coordinated a 15-member annotation team, using performance visibility and quality coaching to meet production benchmarks.
    • Used SQL in QA and analysis, translating implementation details into dashboards, documentation and reporting.
  5. Dec 2023 – Feb 2024Remote

    Associate Project Coordinator, AI/ML Operations · ParallelDots

    Computer-vision delivery for FMCG clients

    • 4 client accounts
    • 3 time zones
    • ~20% shorter cycle time
    • Coordinated shelf-monitoring and product-recognition implementations across four FMCG accounts and three time zones, gathering requirements and communicating model outputs.
    • Analyzed delivery bottlenecks with ML engineers and coordinated model retraining, cutting overall cycle time by about 20%.
    More detail
    • Supported implementation quality monitoring for systems running at 90%+ model accuracy, aligning client feedback and acceptance criteria with technical teams.
    • Tracked requests, decisions and follow-ups in Jira and Confluence using Agile, Scrum and Kanban.

Skills & education

What I bring

AI data operations

  • Dataset validation
  • Annotation operations
  • Reviewer workflows
  • Data quality
  • Model monitoring
  • Anomaly detection

AI evaluation

  • Rubric-based evaluation
  • Preference ranking
  • Calibration
  • Inter-annotator agreement
  • Disagreement analysis
  • Adversarial red-teaming

Technical QA

  • SQL
  • Python
  • Grafana
  • Root-cause analysis
  • Issue reporting
  • Workflow validation

Project operations

  • Agile delivery
  • Stakeholder communication
  • KPI reporting
  • Documentation
  • Jira · Confluence
  • Cross-functional coordination

Education

  • MSc, AI & Technology ManagementNexford University · online, U.S.-accredited
    In progress
  • Postgraduate Certificate, IT Project ManagementConestoga College · Distinction, Dean’s Honour List
    Dec 2024
  • BE, Computer ScienceVisvesvaraya Technological University · First Class
    Sep 2023

Certifications

  • CAPM — Certified Associate in Project Management (PMI)
  • Certified ScrumMaster (CSM) — Scrum Alliance
  • AI Product Management — Duke University
  • Managing Machine Learning Projects — Duke University
  • Deep Learning Specialization — Coursera / Andrew Ng
  • Machine Learning — Stanford University
  • Competitive Strategy & Managerial Economics
  • Excel for Business — Illinois Tech

Contact

Let’s talk about your human-data pipeline.

I’m looking for roles in AI data operations, evaluation, technical QA and AI project operations. Based in Mangaluru, India · Authorized to work in Canada · Open to remote roles worldwide.

kveigas825@gmail.com LinkedIn Résumé (PDF)

Case study · 01

DataQual

The problem

Annotation teams need to know whether a dataset is reliable enough to use, where disagreement comes from, and where scarce review and labeling budget should go. Too often the answer is a spreadsheet or a single quality number that hides annotator uncertainty, consensus assumptions and the difference between an error and a genuinely ambiguous item.

What I built

Agreement & coverage

Percent agreement, pairwise agreement on genuinely shared items and Krippendorff’s alpha, all with bootstrap intervals.

Consensus

Majority vote, weighted vote and a from-scratch Dawid–Skene EM model, compared side by side with every item where they disagree.

Annotator reliability

Beta-Binomial shrinkage with leave-one-worker-out priors, 95% credible intervals and Dirichlet-smoothed confusion evidence.

Disagreement diagnostics

Explainable rules separate probable defects, ambiguity or policy gaps, mixed evidence and insufficient evidence, each with a recommended action.

Review prioritization

A ranked, paginated queue with a decomposable score, compared against four baselines on a deterministic simulator.

Adaptive label collection

Per-item stop / ask-for-more / escalate decisions from a smoothed worker model, with no gold used for planning.

Validation

  • Reference parity. 100% hard-label parity with the pinned Crowd-Kit Dawid–Skene implementation on a real requirements-annotation benchmark (posterior MAE ≈ 7.7×10⁻¹¹).
  • Adaptive collection. Stopping at 95% confidence matched the accuracy of using every label in all 12 synthetic scenarios, with up to 14% fewer labels at 3–5 labels per item and 25–63% fewer at 7–9 (exploratory).
  • Worker model. Up to +27.4 accuracy points over majority vote when some annotators are adversarial.
  • Engineering. 131 backend tests at 92.2% branch-inclusive coverage, Ruff and Pyright clean, frontend unit tests and a production build in CI.

Product

Coverage and agreement view with percent agreement, Krippendorff's alpha with confidence interval, and pairwise agreement table.
Coverage and agreement, with intervals and support counts.
Consensus view comparing majority vote and Dawid–Skene labels.
Consensus sensitivity: where majority vote and Dawid–Skene disagree.
Annotator view with reliability estimates and credible intervals.
Annotator reliability with credible intervals.
Review workspace with ranked candidates and an evidence panel explaining the top candidate.
Review workspace: every rank comes with its evidence.
Label collection view: 80 of 100 items settled, 20 need more labels, none need an expert.
Label collection: settled, needs more labels, or expert review.
Disagreement diagnostics view classifying flagged items.
Disagreement diagnostics with support and recommended actions.

Stack

Python 3.12 · FastAPI · DuckDB · Parquet · NumPy/SciPy · React · TypeScript · Vite · Pytest · Pyright · Ruff · Vitest · Playwright · axe

Limits I state up front

Synthetic benchmarks with five seeds are not enough for broad claims. The worker model assumes annotators err independently, and its posterior is not a calibrated error probability. The public demo has no authentication: never upload confidential data.

Case study · 02

OpsPilot

The problem

Human-data programs (annotation, RLHF preference ranking, AI evaluation) can fail operationally even when the guidelines are clear. A manager juggles qualification, capacity, routing, production state, review backlogs, escalations, SLA pressure and delivery — usually across dashboards that don’t agree and actions nobody can audit.

What I built

Decision brief

The Today view leads with the next best action, plain-language SLA reasons, a completion forecast and a delivered-quality outlook.

Quality-aware allocation

Work goes only to annotators with every required skill, a passed calibration and capacity on the day; urgent work goes to the strongest QA tier. Each allocation records why.

Adaptive, trust-based QA

A Beta posterior per annotator sets the tier (trusted 5%, standard 20%, probation 50%, at risk 100%). Inclusion probabilities are stored, so quality is estimated with design weights.

Execution & escalations

A 10-state task machine rejects invalid transitions; rework is capped at three attempts, then escalated with a recorded decision.

Forecast & delivery

A Monte Carlo forecast gives P50/P90 dates. Five mandatory delivery gates each show their evidence and a link to fix them.

Operational safeguards

One transaction per business operation, idempotency keys on every mutation, UTC-correct timestamps and a non-destructive demo bootstrap.

Validation

  • Budget-matched benchmark. Over 40 paired synthetic workforces, adaptive QA shipped 1.8% undetected errors versus 3.5% for uniform review at the same budget (better in 40 of 40), and needed 43% fewer reviews to match its quality.
  • Correctness under failure. A failure injected mid-workday leaves tasks, reviews and audit records unchanged.
  • Engineering. 74 backend tests at 95% coverage, Ruff and Pyright clean, 41 frontend unit tests, and Playwright flows with axe accessibility checks on all nine views.

Product

Quality insights view with the adaptive sampling policy, delivered-accuracy estimate and annotator trust tiers.
Quality insights: trust tiers, delivered accuracy and review effort.
Allocation view with qualified capacity and allocation reasons.
Allocation with a recorded reason for every assignment.
Keyboard-first QA review queue.
Keyboard-first QA review and escalations.
Delivery readiness view with five gates and their evidence.
Five delivery gates, each with its evidence.

Stack

Python 3.12 · FastAPI · SQLAlchemy 2 · Pydantic 2 · SQLite · React 18 · TypeScript · Vite · Tailwind · Pytest · Vitest · Playwright · axe

Limits I state up front

All data is synthetic. The QA estimators assume reviewers are correct, the forecast has not been backtested on real throughput, and SLA status is rules-based rather than predictive. Authentication, multi-tenancy and hosted backups are not implemented.