Common issues with (human) ranking feedback

1. Introduction

The AI community extensively uses human feedback for training and evaluation. In the context of large language models (LLMs), this feedback is often collected as (pairwise) ranking feedback: given a prompt and two (or more) responses, annotators rank responses according to their preference. Such data is then directly used to train models (e.g. via RLHF (Kaufmann et al., 2023 Kaufmann, T., Weng, P., Bengs, V. & Hüllermeier, E. (2023). A Survey of Reinforcement Learning from Human Feedback. https://doi.org/10.48550/arXiv.2312.14925 )), or to evaluate models (e.g. in Chatbot Arena (Chiang et al., 2024 Chiang, W., Zheng, L., Sheng, Y., Angelopoulos, A., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J. & Stoica, I. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. PMLR. https://proceedings.mlr.press/v235/chiang24b.html )). As with any data, there are some common issues with ranking feedback that can negatively affect downstream use-cases — potentially resulting in misleading evaluation results or trained models that are poorly aligned to human values. Whilst issues with ranking feedback are frequently discussed separately, I have not come across a good overview of common issues.

This blog post aims to fill this gap: a list of the (in my opinion) most common issues with ranking feedback. This list is shaped by my own experience generating and working with ranking feedback datasets — and often running into some of these very same issues. I hope it helps!

📮 Call-for-feedback: If you think I have missed an issue that you run into or work that might be relevant, please reach out via contact@arduin.io! I would be very happy to add it here and would of course acknowledge your contribution (if wanted).

2. Background: ranking feedback data

In this blog post, I focus on feedback data in the context of LLM training and evaluation. In particular, I focus on ranking feedback – possibly the most common form of feedback used in the LLM context. In ranking feedback, each datapoint typically consists of the following parts: (A) a prompt, (B) multiple responses and a (C) general ranking annotation that orders the responses according to annotator preference. Often only two responses are ranked (Chiang et al., 2024 Chiang, W., Zheng, L., Sheng, Y., Angelopoulos, A., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J. & Stoica, I. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. PMLR. https://proceedings.mlr.press/v235/chiang24b.html ). Such feedback is also commonly referred to as pairwise preferences/feedback. Human-annotated ranking feedback is often simply referred to as human preferences or human feedback. In many datasets additional information beyond prompt, responses and ranking is included: for example, (D) additional annotations (e.g. regarding the truthfulness of each response) and (E) metadata about the models used to sample responses and annotator details (e.g. demographics or, if LLM-based, configuration parameters). The (pseudo-)yaml below illustrates this typical structure of a ranking feedback datapoint

💬 prompt: task for which model responses are sampled and compared on,
    can sometimes be full prior conversation history with multiple
    back-and-forth messages
🤖 responses: sampled responses to the prompt, typically two (i.e. a pair of)
    responses, recently also often 4 or more.
✏️ annotation:
    - general_ranking: ranking of all responses,
        typically allowing for ties between responses
    - additional_annotations: other annotation data often collected
        from annotators, e.g. ratings for each response of its
        harmfulness and truthfulness # optional
📋 metadata:
    - response_models: names and configurations of models from
        which responses are sampled # optional
    - annotator: id and demographic details if human annotators,
        LLM configuration if LLM-as-a-Judge # optional

Example datapoint:

💬 prompt: 'How far away is the moon?'
🤖 responses: ['Not very far at all!', 'About 384,000 km']
✏️ annotation:
    - general_ranking: '1 > 0'
    - additional_annotations:
        truthful: [0.1, 0.9]
📋 metadata:
    - response_models: ['model-x', 'model-y']
    - annotator:
        name: 'anonymous'

Popular datasets. There are a number of popular ranking feedback datasets, here is an incomplete list: Anthropic HH (Bai et al., 2022 Bai, Y., Jones, A., Ndousse, K., Askell, A., Chen, A., DasSarma, N., Drain, D., Fort, S., Ganguli, D., Henighan, T., Joseph, N., Kadavath, S., Kernion, J., Conerly, T., El-Showk, S., Elhage, N., Hatfield-Dodds, Z., Hernandez, D., Hume, T., Johnston, S., Kravec, S., Lovitt, L., Nanda, N., Olsson, C., Amodei, D., Brown, T., Clark, J., McCandlish, S., Olah, C., Mann, B. & Kaplan, J. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. https://doi.org/10.48550/arXiv.2204.05862 ), AlpacaEval annotations (Dubois et al., 2023 Dubois, Y., Li, C., Taori, R., Zhang, T., Gulrajani, I., Ba, J., Guestrin, C., Liang, P. & Hashimoto, T. (2023). AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback. https://proceedings.neurips.cc/paper_files/paper/2023/hash/5fc47800ee5b30b8777fdd30abcaaf3b-Abstract-Conference.html ), UltraFeedback (Cui et al., 2024 Cui, G., Yuan, L., Ding, N., Yao, G., He, B., Zhu, W., Ni, Y., Xie, G., Xie, R., Lin, Y., Liu, Z. & Sun, M. (2024). UltraFeedback: Boosting Language Models with Scaled AI Feedback. https://doi.org/10.48550/arXiv.2310.01377 ), ChatbotArena data (Chiang et al., 2024 Chiang, W., Zheng, L., Sheng, Y., Angelopoulos, A., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J. & Stoica, I. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. PMLR. https://proceedings.mlr.press/v235/chiang24b.html ), and PRISM (Kirk et al., 2024 Kirk, H., Whitefield, A., Röttger, P., Bean, A., Margatina, K., Ciro, J., Mosquera, R., Bartolo, M., Williams, A., He, H., Vidgen, B. & Hale, S. (2024). The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models. https://doi.org/10.52202/079017-3342 ).

3. Common issues

As with any form of dataset, ranking feedback datasets frequently have issues affecting downstream uses. I split common issues into two separate lists: first issues related to the selection of prompts and responses (dataset design issues) and then issues related to the annotation of these datasets (annotation issues). Most issues affect both evaluation and training use-cases. Issues that are only relevant for one use-case are marked as such (e.g. ’eval only’). Below is an overview of the issues discussed.

Important note: Many issues can be unavoidable. Thus, whilst it is important to be aware that certain datasets have these issues, I do not think these issues reflect negatively on the dataset’s creators or their contribution. Collecting human data is expensive and messy. Large ranking feedback datasets are rarely made public, so any dataset contributions are very valuable even if they have issues.

For many issues below I add references. These references contain some form of discussion of the respective issue, sometimes (but not always) because the issues is observed in a newly introduced dataset.

Dataset design issues

Annotation issues

4. Conclusion

Ranking feedback can be flawed in many ways. Yet, the above list is undoubtedly incomplete: there are many dataset properties that are difficult to measure and haven’t been detected yet. Ranking feedback also has more general limitations: for example, relative rankings omit potentially important information about how much to responses differ. In general, given the enormous cost of both data collection and model training, really trying to understand ranking feedback datasets you use as well as possible seems well-justified – there are many potential issues to detect and mitigate. Exploring this open problem of issue detection, our recent work on Inverse Constitutional AI (Findeis et al., 2024 Findeis, A., Kaufmann, T., Hüllermeier, E., Albanie, S. & Mullins, R. (2024). Inverse Constitutional AI: Compressing Preferences into Principles. http://arxiv.org/abs/2406.06560 ) aims to automatically find new kinds of annotator biases — beyond those already well-known. I think there is great potential to develop further (potentially similar) methods that can effectively discover and measure ranking feedback issues.

5. Further reading

There are many papers in the space of (ranking) feedback (issues), below is a short list of papers I found particularly useful:

• Kirk et al. (2024) Kirk, H., Whitefield, A., Röttger, P., Bean, A., Margatina, K., Ciro, J., Mosquera, R., Bartolo, M., Williams, A., He, H., Vidgen, B. & Hale, S. (2024). The PRISM Alignment Dataset: What Participatory, Representative and Individualised Human Feedback Reveals About the Subjective and Multicultural Alignment of Large Language Models. https://doi.org/10.52202/079017-3342 provide possibly the most extensive public discussion of any human feedback dataset (PRISM). I can recommend having a look at the paper!
• Chiang et al. (2024) Chiang, W., Zheng, L., Sheng, Y., Angelopoulos, A., Li, T., Li, D., Zhu, B., Zhang, H., Jordan, M., Gonzalez, J. & Stoica, I. (2024). Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference. PMLR. https://proceedings.mlr.press/v235/chiang24b.html provide an extensive analysis of Chatbot Arena data, a popular pairwise ranking feedbakc dataset that is worth to understand well.
• Lambert et al. (2024) Lambert, N., Morrison, J., Pyatkin, V., Huang, S., Ivison, H., Brahman, F., Miranda, L., Liu, A., Dziri, N., Lyu, S., Gu, Y., Malik, S., Graf, V., Hwang, J., Yang, J., Bras, R., Tafjord, O., Wilhelm, C., Soldaini, L., Smith, N., Wang, Y., Dasigi, P. & Hajishirzi, H. (2024). Tulu 3: Pushing Frontiers in Open Language Model Post-Training. https://doi.org/10.48550/arXiv.2411.15124 provide a detailed description of preference data considerations during large model training in this Tülu 3 post-training discussion – similar details are hard to come by from other labs.
• Finally, I also recommend Weng (2024) Weng, L. (2024, February 5). Thinking about High-Quality Human Data. https://lilianweng.github.io/posts/2024-02-05-human-data-quality/ and Kaufmann et al. (2025) Kaufmann, T., Ball, S., Beck, J., Hüllermeier, E. & Kreuter, F. (2025). On the Challenges and Practices of Reinforcement Learning from Real Human Feedback. InMeo, R. & Silvestri, F. (Eds.), Machine Learning and Principles and Practice of Knowledge Discovery in Databases. (pp. 276–294). Springer Nature Switzerland. https://sml.disi.unitn.it/files/hldm23/HLDM23_325.pdf for overviews of challenges specific to human data collection.

Notes & references

Acknowledgements

Big thanks to everybody who provided feedback and discussed related issues!

References

Citation

If you found this post useful for your work, please consider citing it as:

Findeis, Arduin. (Jan 2025). Common issues with (human) ranking feedback. https://arduin.io/blog/feedback-issues/.

 @article{Findeis2025Commonissueswithhumanrankingfeedback,
        title = "Common issues with (human) ranking feedback",
        author = "Findeis, Arduin",
        journal = "arduin.io",
        year = "2025",
        month = "January",
        url = "https://arduin.io/blog/feedback-issues/"
 }