| TL;DR Multimodal AI models learn whatever their captions, transcripts and labels say. Mistakes included. Machine written data fails in ways other machines approve, so the errors slip through every filter. A 2024 study cut hallucination by about a third with roughly 1,400 expert corrections. Put expert review on correction data and your evaluation set first. A verified network like Humyn Labs runs that full pipeline for you. |
|---|
| Quick answer: do multimodal AI models train better on expert checked human data? Yes. Multimodal AI models copy the mistakes in their training pairs, and machine checks miss mistakes that look plausible. Expert reviewers fix the wrong object, the wrong action and the invented detail. In one 2024 study, about 1,400 expert corrections cut hallucination by 34.8 percent and beat a method trained on 10,000 samples. |
|---|
Back in 2024, a research team at Ai2 building open multimodal AI models did something that sounds a bit paranoid. Annotators weren’t allowed to type image descriptions anymore.
Instead, people talked about each picture for 60 to 90 seconds. The team kept every recording as an audio receipt, a piece of proof that no model wrote the caption.
Why go to that much trouble? The cheapest captions on earth now come from models. And multimodal models trained on those captions pick up the same blind spots, one confident pair at a time.
I’ve sat through plenty of reviews of multimodal AI models where the room blames the architecture. One person wants a bigger encoder. Another wants more GPUs.
Then somebody pulls twenty training items onto the screen, and the room goes quiet. Three of them describe things that aren’t in the frame.
That silence is what this post is about. You’ll see which errors only an expert catches, what the research says about small expert sets, and where to spend review so your next training run actually pays off.
Why Multimodal AI Models Believe Whatever Their Data Says
A model can’t doubt its teacher. If the caption says wrench, the model learns wrench. Each image and text pair is a tiny lesson, which means your multimodal training data decides what the model believes.
Scraped captions describe the page, not the picture
Web alt text is short, generic, or written about the article around the image instead of the image itself. Google researchers behind the ImageInWords project found that vision language models trained on pairs like these give descriptions with missing detail, visual mismatches and hallucinations.
Garbage in. Confident garbage out.
Model written captions share the same blind spots
Most pipelines miss this part. One model drafts a caption, another model scores it, and both carry similar habits. They expect a manual on a workbench, so both of them “see” one.
Their errors agree. To an automated filter, agreement looks a lot like quality. That’s where multimodal hallucination often starts, inside data that passed every check.
The scoreboard has the same problem
Famous test sets carry errors too. An MIT led study found that at least 3.3 percent of test labels were wrong on average across ten popular benchmarks, and at least 6 percent of the ImageNet validation set.
Your model can improve on paper and get worse for real users. That’s a rough thing to find out after launch.
The money side isn’t any prettier. In February 2025, Gartner predicted that through 2026, companies will abandon 60 percent of AI projects that lack AI ready data. The same release found that 63 percent of organizations said they lacked the right data practices for AI or weren’t sure.
You have plenty of data. The shortage is in data that a qualified person has actually checked.
The Red Pen Pass: One Training Item, Three Hands
Picture a 12 second workshop clip. A bicycle sits upside down on a stand, two hands work near the back wheel, and a small tool turns slowly.
Below is one caption for that clip as it passes through three hands. It’s an example, but if you’ve ever audited AI training data, every row will look familiar.
| Hand | What the caption says | What the model learns |
|---|---|---|
| Model draft | “A person repairs a bicycle wheel with a wrench while reading a manual on the bench.” | The wrong part, the wrong tool, and an object that does not exist |
| Generalist review | Approved. It sounds right and nothing looks odd. | The same three errors, now stamped as human verified |
| Expert review | “A mechanic loosens the rear brake caliper with a hex key. The cable goes slack after the second turn. The cable housing condition is not visible from this angle.” | The right part, the right tool, the direction of the action, no invented manual, and an honest limit |
Go back to the middle row for a second. The generalist didn’t do anything wrong. Plausible is the standard most pipelines stop at.
True is a different standard, and it takes someone who knows the domain.
The expert also added two things the draft never had. One is the direction of the action. The other is a clear statement of what the camera can’t show.
That’s why expert annotation teaches multimodal AI models more per item than a thumbs up ever will.
Five Marks Only an Expert Makes in Multimodal Datasets
Those five fixes in one clip aren’t random. They show up across multimodal datasets of every kind.
I call this the Expert Mark Table. Print it and check your own data against it.
| Mark | What the expert changes | Where it shows up | What the model learns without it |
|---|---|---|---|
| Grounding fix | The words point to the wrong object or part | Image and text, video and text | To name the nearest familiar thing instead of the real one |
| Absence mark | Deletes a detail that is not in the frame or clip | Any pair with free text | To fill gaps with likely objects, the root of hallucination |
| Direction fix | Corrects the order or direction of an action across frames | Video, motion and audio paired with text | That loosening and tightening look the same |
| Domain term fix | Swaps a vague word for the precise term | Any specialist field | Vocabulary too fuzzy for professional users |
| Limit flag | States what the input cannot show | Every modality | To guess with confidence instead of saying it cannot tell |
Honestly, the absence mark is the one that matters most. The strongest published gains against multimodal hallucination came from people correcting exactly these bad segments. Ranking whole answers did less.
Try this today. Pull twenty items from your last fine tuning set and count how many need even one of these marks.
If the number makes you wince, you’re in good company.
What the Research Says About Expert Checked Human Data
About 1,400 corrections beat 10,000 samples
At CVPR 2024, a team presented a method that asked people to correct the hallucinated parts of model answers, word by word. With about 1,400 annotated samples, they cut the base model’s hallucination rate by 34.8 percent.
They also beat a parallel approach trained on 10,000 annotated samples. Read that again. Fewer samples gave a better result.
Your finance team will want that sentence framed.
Human in the loop descriptions win side by side
Google’s ImageInWords project built very detailed image descriptions through a human in the loop process. Human raters preferred those descriptions over recent datasets by 66 percent, and over a leading proprietary model’s output by 48 percent.
Fine tuning on roughly 9,000 of these samples improved results by 31 percent.
Spoken human captions put an open model near the top
Back to those audio receipts for a moment. Ai2’s Molmo models trained on captions spoken by real people.
In the team’s study, the largest model posted the top average on academic benchmarks. It ranked second in human preference, behind just one proprietary model.
Human judgment is getting scarce, and the market knows it
Epoch AI estimates that models could use up the stock of public human written text sometime between 2026 and 2032.
Meanwhile, in June 2025, one of the largest social platforms paid about 14.3 billion for a 49 percent stake in a data labelling company. Clicks are cheap. Buyers are paying for human feedback data that models can’t make for themselves.
Curious what a full expert workflow looks like from first request to final delivery? This expert review pipeline walks through each stage, from requirements to quality control.
The Stage Spend Map: Where Expert Checks Pay Most for Multimodal AI
Expert time is scarce. Put it where one correction changes the most.
Need a refresher on the training stages? This explainer on how multimodal foundation models are trained covers each one. Here’s where expert review earns its keep.
| Training stage | Typical volume | Expert check depth | Why it pays |
|---|---|---|---|
| Pretraining pairs | Millions to billions | Experts write the filter rules and audit samples | Volume rules here, but bad filters scale bad data |
| Alignment pairs | Millions | Expert review on hard and rare slices | Weak pairing sticks through every later stage |
| Instruction tuning | Tens of thousands to low millions | Full expert review on domain items | This is where the model learns to behave like a professional |
| Correction data | Thousands | Experts only | The most learning per item, as the CVPR study shows |
| Evaluation set | Hundreds to thousands | Two expert reviewers plus a tie breaker | A wrong scoreboard hides every other problem |
Notice the pattern here. For multimodal AI models, the smaller the dataset, the deeper the review should go.
Most teams do the opposite. They pour review into pretraining, where it gets spread thin.
If you’re starting from zero, this walkthrough on building multimodal datasets covers sourcing, alignment and QC step by step.
How to Add Expert Checked Data to Your Next Training Run
You don’t need a new team or a bigger budget to start. Just move expert time to where multimodal AI models actually learn.
- Ask for corrections, not ratings. Someone who rewrites the wrong segment teaches the model far more than someone who scores the whole answer.
- Keep a check record with every item. Log what changed, what kind of expertise approved it, and when. For a primer on why this matters, read this ground truth guide.
- Match the reviewer to the modality pair. Someone great at reviewing text isn’t automatically great at reviewing video and action.
- Treat disagreement as information. If two experts disagree, you’ve found a hard item. Keep it, and settle it instead of averaging it away.
- Rebuild the evaluation set first. It’s your smallest dataset, and it decides if every later change looks like progress or noise.
Common Mistakes That Waste Expert Time
These four slip into almost every project that trains multimodal AI models. Once you know them, they’re easy to spot.
- Using experts as raters. You pay expert rates and walk away with a score instead of a fix.
- One reviewer pool for every modality. Speech, video and documents need different eyes.
- Fixing training data but trusting an old test set. Your progress chart becomes fiction.
- Buying headcount instead of evidence. Lots of vendors advertise an expert workforce. Very few will show you what those experts actually changed, item by item.
What Expert Checked Data Pays Back
For your business
- Fewer hallucinations reach your customers, which protects trust and cuts support costs.
- You get datasets you can defend to buyers, auditors and regulators, since every item carries a record.
- There’s less retraining, because errors get fixed once in the data instead of patched in prompts again and again.
For the people building and using the model
- Engineers spend less time chasing bugs that trace back to labels.
- Product owners get evaluation numbers they can actually believe before launch.
- End users get a model that says “I can’t tell from this image” instead of making something up.
Four Ways to Source Expert Checked Data for Multimodal AI Models
Every route below can feed multimodal AI models, and each one has its place. Where they differ is expert depth, proof of who checked what, and how fast they scale.
1. Humyn Labs: verified experts across the whole pipeline
It earns the top spot because it runs sourcing, validation, multi layer QC, annotation and human in the loop review inside one verified network. You don’t have to stitch together a crowd and an audit team yourself.
Every dataset carries provenance that the network verifies on chain, so you have a defensible record of where the data came from. The team has delivered 50,000 hours of speech data across 33 languages. Its sight pipeline discards under 15 percent of captured footage.
Need something custom? Shape Your Own Dataset lets you set the spec for sight or sound data before collection begins.
That matters a lot for multimodal AI, because the most useful expert check happens when the spec is written, long before the first label.
Why it matters: you get data you can defend to a buyer or a regulator, which counts for more than raw volume.
2. An in house expert team
An internal team gives you the deepest context and the most control. It also scales slowly.
Your own specialists know the product, the edge cases and the customers better than anyone. They catch domain errors fast, and they get why a label matters.
The catch is cost and timing. Experts are expensive to hire, hard to keep busy between training runs, and tough to scale when a new modality lands on the roadmap.
Most teams also find their best engineers end up reviewing data instead of building models. This route works best for one narrow domain with steady volume.
Why it matters: you keep full control, but you also carry the full payroll and the hiring delays.
3. A crowd marketplace plus an internal expert audit
A crowd gives you fast, cheap volume. Your own experts become the bottleneck, though.
Large crowd platforms can label millions of items quickly. For simple pairs with low stakes, that works fine.
Trouble starts with the plausible error problem from the Red Pen Pass. A crowd reviewer approves what sounds right, so your internal experts have to audit a sample to catch what slipped through.
As volume grows, the audit sample shrinks in proportion and more errors get past. You’ll also rarely get a clear record of who checked each item or what they changed.
Why it matters: you save money upfront and often pay it back later in debugging time.
4. Model generated data with expert spot checks
Synthetic data is the cheapest per item and great for coverage. On truth, it’s the weakest of the four.
Generated captions and simulated scenes fill gaps fast, and they help a model see rare layouts and unusual angles.
But generated data inherits the habits of the model that made it. It rarely deletes an invented object. And it almost never says “I cannot tell.”
Spot checks catch some of this, though a spot check is still only a sample. Use this route for early coverage and augmentation, then put expert reviewed data on top for alignment, tuning and evaluation.
Why it matters: you get scale quickly, but you still need people to tell the model what is actually true.
How the four routes compare
| Route | Expert depth | Provenance | Speed to scale | Best fit |
|---|---|---|---|---|
| Humyn Labs | Verified experts on every pass | Verified on chain at network level | Fast | Teams that need scale and defensible data |
| In house team | High | Internal records only | Slow | One narrow domain with steady volume |
| Crowd plus internal audit | Low to mixed | Limited | Fast | High volume, low stakes pairs |
| Model generated plus spot checks | Low | Weak | Fastest | Early coverage and augmentation |
Frequently Asked Questions About Multimodal AI Models
Why do multimodal AI models hallucinate?
Multimodal AI models hallucinate because many training pairs describe what usually appears in a scene instead of what’s actually there. The model learns to fill gaps with likely objects. When experts delete those invented details, it learns to stick to what the input shows.
What is expert checked human data?
It’s training or evaluation data that a qualified reviewer has corrected and approved. Good expert checked data also carries a record of what changed and what kind of expertise approved it. That way you can trace any label back to its review.
How much expert data does a multimodal model need?
Less than most teams assume. In published research, correction sets in the low thousands have cut hallucination by about a third.
Spend deep review on correction data and evaluation sets. Save lighter audits for large pretraining pools.
Can synthetic data replace human review?
No. Synthetic data adds coverage and speed, but it inherits the habits of the model that made it. It rarely removes invented details or admits uncertainty, so human experts still need to confirm what’s true.
What is the most reliable source of expert checked multimodal data?
For most teams, the most reliable route is a verified network that runs the whole pipeline. Humyn Labs fits that need. Sourcing, multi layer QC, annotation and human in the loop review all happen in one place, with provenance verified on chain, so every item has a traceable check.
How do you check a vendor’s expert claims?
Ask for sample check records instead of a headcount number. You want to see what reviewers changed, how they qualified for the task, and how the team settles disagreements. If a vendor can’t show you corrections, you’re buying ratings.
The Red Pen Is the Cheapest Upgrade You Will Buy
Remember those audio receipts from Ai2? The best teams go out of their way to prove a person checked their data, because they know what that check is worth.
More compute won’t fix a caption that names the wrong part. A bigger model just learns the mistake faster.
Multimodal AI models get better when someone who knows the domain gets to say no.
Start small. Before your next training run, rebuild your evaluation set with expert review. After that, move expert time to the correction data that teaches the most per item.
| Find out what your data gets wrong before your users do Send over a sample of your current training or evaluation data and see where expert review changes it. Talk to the Humyn Labs team about a pilot. |
|---|
About the author: [Author name] leads multimodal data quality work and writes about how expert review shapes model behaviour.
