Rapidata/Flux-2-pro_t2i_human_preference
Viewer
•
Updated
•
44.9k
•
587
•
7
AI & ML interests
RLHF, Model Evaluation, Benchmarks, Data Labeling, Human Feedback, Computer Vision, Image Generation, Video Generation, LLMs, Translations
Recent Activity
View all activity
LinoGiger
published
a
dataset
5 days ago
LinoGiger
updated
a
dataset
5 days ago
LinoGiger
published
a
dataset
10 days ago
LinoGiger
updated
a
dataset
10 days ago
RapidataAI
published
a
dataset
11 days ago
LinoGiger
updated
a
dataset
24 days ago
jasoncorkill
posted
an
update
27 days ago
Post
4679
Do you remember https://thispersondoesnotexist.com/ ? It was one of the first cases where the future of generative media really hit us. Humans are incredibly good at recognizing and analyzing faces, so they are a very good litmus test for any generative image model.
But none of the current benchmarks measure the ability of models to generate humans independently. So we built our own. We measure the models ability to generate a diverse set of human faces and using over 20'000 human annotations we ranked all of the major models on their ability to generate faces. Find the full ranking here:
https://app.rapidata.ai/mri/benchmarks/68af24ae74482280b62f7596
We have release the full underlying data publicly here on huggingface: Rapidata/Face_Generation_Benchmark
But none of the current benchmarks measure the ability of models to generate humans independently. So we built our own. We measure the models ability to generate a diverse set of human faces and using over 20'000 human annotations we ranked all of the major models on their ability to generate faces. Find the full ranking here:
https://app.rapidata.ai/mri/benchmarks/68af24ae74482280b62f7596
We have release the full underlying data publicly here on huggingface: Rapidata/Face_Generation_Benchmark
LinoGiger
updated
a
dataset
27 days ago
jasoncorkill
updated
a
dataset
27 days ago
jasoncorkill
published
a
dataset
27 days ago
caldarons
updated
a
dataset
about 1 month ago
caldarons
published
a
dataset
about 1 month ago
jasoncorkill
posted
an
update
5 months ago
Post
3277
"Why did the bee get married?"
"Because he found his honey!"
This was the "funniest" joke out of 10'000 jokes we generated with LLMs. With 68% of respondents rating it as "funny".
Original jokes are particularly hard for LLMs, as jokes are very nuanced and a lot of context is needed to understand if something is "funny". Something that can only reliably be measured using humans.
LLMs are not equally good at generating jokes in every language. Generated English jokes turned out to be way funnier than the Japanese ones. 46% of English-speaking voters on average found the generated joke funny. The same statistic for other languages:
Vietnamese: 44%
Portuguese: 40%
Arabic: 37%
Japanese: 28%
There is not much variance in generation quality among models for any fixed language. But still Claude Sonnet 4 slightly outperforms others in Vietnamese, Arabic and Japanese and Gemini 2.5 Flash in Portuguese and English
We have release the 1 Million (!) native speaker ratings and the 10'000 jokes as a dataset for anyone to use:
Rapidata/multilingual-llm-jokes-4o-claude-gemini
"Because he found his honey!"
This was the "funniest" joke out of 10'000 jokes we generated with LLMs. With 68% of respondents rating it as "funny".
Original jokes are particularly hard for LLMs, as jokes are very nuanced and a lot of context is needed to understand if something is "funny". Something that can only reliably be measured using humans.
LLMs are not equally good at generating jokes in every language. Generated English jokes turned out to be way funnier than the Japanese ones. 46% of English-speaking voters on average found the generated joke funny. The same statistic for other languages:
Vietnamese: 44%
Portuguese: 40%
Arabic: 37%
Japanese: 28%
There is not much variance in generation quality among models for any fixed language. But still Claude Sonnet 4 slightly outperforms others in Vietnamese, Arabic and Japanese and Gemini 2.5 Flash in Portuguese and English
We have release the 1 Million (!) native speaker ratings and the 10'000 jokes as a dataset for anyone to use:
Rapidata/multilingual-llm-jokes-4o-claude-gemini
-
- 7 replies
jasoncorkill
posted
an
update
6 months ago
Post
2438
Imagine you could have an Image Arena score equivalent at each checkpoint during training. We released the first version of just that:
Crowd-Eval
Add one line of code to your training loop and you will have a new real human loss curve in your W&B dashboard.
Thousands of real humans from around the world rating your model in real time at the cost of a few dollars per checkpoint is a game changer.
Check it out here: https://github.com/RapidataAI/crowd-eval
First 5 people to put it in their loop get 100'000 human responses for free! (ping me)
Crowd-Eval
Add one line of code to your training loop and you will have a new real human loss curve in your W&B dashboard.
Thousands of real humans from around the world rating your model in real time at the cost of a few dollars per checkpoint is a game changer.
Check it out here: https://github.com/RapidataAI/crowd-eval
First 5 people to put it in their loop get 100'000 human responses for free! (ping me)
jasoncorkill
posted
an
update
7 months ago
Post
3996
Benchmark Update:
@google
Veo3 (Text-to-Video)
Two months ago, we benchmarked @google ’s Veo2 model. It fell short, struggling with style consistency and temporal coherence, trailing behind Runway, Pika, @tencent , and even @alibaba-pai .
That’s changed.
We just wrapped up benchmarking Veo3, and the improvements are substantial. It outperformed every other model by a wide margin across all key metrics. Not just better, dominating across style, coherence, and prompt adherence. It's rare to see such a clear lead in today’s hyper-competitive T2V landscape.
Dataset coming soon. Stay tuned.
Two months ago, we benchmarked @google ’s Veo2 model. It fell short, struggling with style consistency and temporal coherence, trailing behind Runway, Pika, @tencent , and even @alibaba-pai .
That’s changed.
We just wrapped up benchmarking Veo3, and the improvements are substantial. It outperformed every other model by a wide margin across all key metrics. Not just better, dominating across style, coherence, and prompt adherence. It's rare to see such a clear lead in today’s hyper-competitive T2V landscape.
Dataset coming soon. Stay tuned.
-
- 5 replies
jasoncorkill
posted
an
update
7 months ago
Post
2881
🔥 Hidream I1 is online! 🔥
We just added Hidream I1 to our T2I leaderboard (https://www.rapidata.ai/leaderboard/image-models) benchmarked using 195k+ human responses from 38k+ annotators, all collected in under 24 hours.
It landed #3 overall, right behind:
- @openai 4o
- @black-forest-labs Flux 1 Pro
...and just ahead of @black-forest-labs Flux 1.1 Pro, @xai-org Aurora and @google Imagen3.
Want to dig into the data? Check out our dataset here:
Rapidata/Hidream_t2i_human_preference
What model should we benchmark next?
We just added Hidream I1 to our T2I leaderboard (https://www.rapidata.ai/leaderboard/image-models) benchmarked using 195k+ human responses from 38k+ annotators, all collected in under 24 hours.
It landed #3 overall, right behind:
- @openai 4o
- @black-forest-labs Flux 1 Pro
...and just ahead of @black-forest-labs Flux 1.1 Pro, @xai-org Aurora and @google Imagen3.
Want to dig into the data? Check out our dataset here:
Rapidata/Hidream_t2i_human_preference
What model should we benchmark next?
jasoncorkill
posted
an
update
7 months ago
Post
5544
🚀 Building Better Evaluations: 32K Image Annotations Now Available
Today, we're releasing an expanded version: 32K images annotated with 3.7M responses from over 300K individuals which was completed in under two weeks using the Rapidata Python API.
Rapidata/text-2-image-Rich-Human-Feedback-32k
A few months ago, we published one of our most liked dataset with 13K images based on the @data-is-better-together 's dataset, following Google's research on "Rich Human Feedback for Text-to-Image Generation" (https://arxiv.org/abs/2312.10240). It collected over 1.5M responses from 150K+ participants.
Rapidata/text-2-image-Rich-Human-Feedback
In the examples below, users highlighted words from prompts that were not correctly depicted in the generated images. Higher word scores indicate more frequent issues. If an image captured the prompt accurately, users could select [No_mistakes].
We're continuing to work on large-scale human feedback and model evaluation. If you're working on related research and need large, high-quality annotations, feel free to get in touch: info@rapidata.ai.
Today, we're releasing an expanded version: 32K images annotated with 3.7M responses from over 300K individuals which was completed in under two weeks using the Rapidata Python API.
Rapidata/text-2-image-Rich-Human-Feedback-32k
A few months ago, we published one of our most liked dataset with 13K images based on the @data-is-better-together 's dataset, following Google's research on "Rich Human Feedback for Text-to-Image Generation" (https://arxiv.org/abs/2312.10240). It collected over 1.5M responses from 150K+ participants.
Rapidata/text-2-image-Rich-Human-Feedback
In the examples below, users highlighted words from prompts that were not correctly depicted in the generated images. Higher word scores indicate more frequent issues. If an image captured the prompt accurately, users could select [No_mistakes].
We're continuing to work on large-scale human feedback and model evaluation. If you're working on related research and need large, high-quality annotations, feel free to get in touch: info@rapidata.ai.
jasoncorkill
posted
an
update
8 months ago
Post
3318
🚀 We tried something new!
We just published a dataset using a new (for us) preference modality: direct ranking based on aesthetic preference. We ranked a couple of thousand images from most to least preferred, all sampled from the Open Image Preferences v1 dataset by the amazing @data-is-better-together team.
📊 Check it out here:
Rapidata/2k-ranked-images-open-image-preferences-v1
We're really curious to hear your thoughts!
Is this kind of ranking interesting or useful to you? Let us know! 💬
If it is, please consider leaving a ❤️ and if we hit 30 ❤️s, we’ll go ahead and rank the full 17k image dataset!
We just published a dataset using a new (for us) preference modality: direct ranking based on aesthetic preference. We ranked a couple of thousand images from most to least preferred, all sampled from the Open Image Preferences v1 dataset by the amazing @data-is-better-together team.
📊 Check it out here:
Rapidata/2k-ranked-images-open-image-preferences-v1
We're really curious to hear your thoughts!
Is this kind of ranking interesting or useful to you? Let us know! 💬
If it is, please consider leaving a ❤️ and if we hit 30 ❤️s, we’ll go ahead and rank the full 17k image dataset!
-
- 6 replies
jasoncorkill
posted
an
update
8 months ago
Post
3095
🔥 Yesterday was a fire day!
We dropped two brand-new datasets capturing Human Preferences for text-to-video and text-to-image generations powered by our own crowdsourcing tool!
Whether you're working on model evaluation, alignment, or fine-tuning, this is for you.
1. Text-to-Video Dataset (Pika 2.2 model):
Rapidata/text-2-video-human-preferences-pika2.2
2. Text-to-Image Dataset (Reve-AI Halfmoon):
Rapidata/Reve-AI-Halfmoon_t2i_human_preference
Let’s train AI on AI-generated content with humans in the loop.
Let’s make generative models that actually get us.
We dropped two brand-new datasets capturing Human Preferences for text-to-video and text-to-image generations powered by our own crowdsourcing tool!
Whether you're working on model evaluation, alignment, or fine-tuning, this is for you.
1. Text-to-Video Dataset (Pika 2.2 model):
Rapidata/text-2-video-human-preferences-pika2.2
2. Text-to-Image Dataset (Reve-AI Halfmoon):
Rapidata/Reve-AI-Halfmoon_t2i_human_preference
Let’s train AI on AI-generated content with humans in the loop.
Let’s make generative models that actually get us.
jasoncorkill
posted
an
update
8 months ago
Post
2389
🚀 Rapidata: Setting the Standard for Model Evaluation
Rapidata is proud to announce our first independent appearance in academic research, featured in the Lumina-Image 2.0 paper. This marks the beginning of our journey to become the standard for testing text-to-image and generative models. Our expertise in large-scale human annotations allows researchers to refine their models with accurate, real-world feedback.
As we continue to establish ourselves as a key player in model evaluation, we’re here to support researchers with high-quality annotations at scale. Reach out to info@rapidata.ai to see how we can help.
Lumina-Image 2.0: A Unified and Efficient Image Generative Framework (2503.21758)
Rapidata is proud to announce our first independent appearance in academic research, featured in the Lumina-Image 2.0 paper. This marks the beginning of our journey to become the standard for testing text-to-image and generative models. Our expertise in large-scale human annotations allows researchers to refine their models with accurate, real-world feedback.
As we continue to establish ourselves as a key player in model evaluation, we’re here to support researchers with high-quality annotations at scale. Reach out to info@rapidata.ai to see how we can help.
Lumina-Image 2.0: A Unified and Efficient Image Generative Framework (2503.21758)