Methodology for analyzing the individual comments to the NTIA’s AI RFC - Imbue

Methodology for analyzing the individual comments to the NTIA’s AI RFC

37 min read

Last updated 16 Jun 2026

Summary: This article describes the methodology and motivation for the analysis decisions of our investigation of the individual submissions to the NTIA’s AI Accountability Request for Comment. For our discussion of results and implications for policy makers, please see here.

Specifically, this report describes how we created the system to perform this analysis and how we validated its results. Throughout this exercise, we also critiqued our own work and investigated the limitations of what we ultimately produced. We hope that by communicating both the strengths and weaknesses of our system, this will result in greater confidence in how our system actually behaves and contextualize our conclusions about people’s concerns regarding AI Accountability. We believe this transparency is important when evaluating systems and it’s one piece of the puzzle to building safer AI systems.

We’ve uploaded our code to compute the results here.

We believe that, when using complicated AI systems to analyze data, those presenting an analysis ought to be responsible for also presenting sufficient data about their methodology that enables the reader to verify the results. In particular, this means providing not only the raw data, but the justifications of the various decisions made, as well as the procedures and methodologies that were used.

Here, we present what we believe represents a positive example of using large language models (LLMs) to analyze a relatively large and unstructured dataset – the set of all individual submissions NTIA request for comment on AI accountability.

We had two goals for this project:

The focus of this report is on the second (meta-level) question – how can we use these powerful models in order to create an analysis that we can actually trust? For more on the first question (what was actually said in the report), see our other post here.

The rest of this document is broken down as follows:

Methodology

We selected the following 15 questions after several rounds of iteration as we felt that in their entirety they would cover a majority of concerns within the submissions:

We attempted to answer these questions for every individual (non-corporate/non-entity) submission to the RFC that was:

  1. Written in English (excluded 3).
  2. Not a duplicate (excluded 14)
  3. Not consisting only of image attachments (excluded 2)
  4. Longer than 75 characters (excluded 30)
  5. Shorter than 20,000 characters (excluded 15)

This resulted in a final dataset of 1,198 assessed submissions. For each valid submission, we answered each of the 15 questions by querying GPT-4 with the following prompt:

textInstructions: Identify the label of the correct answer for each question below about the contents of the provided document.

The following document was provided in response to The National Telecommunications and Information Administration (NTIA) requests for comments on Artificial Intelligence ("AI") system accountability measures and policies. NTIA will rely on these comments, along with other public engagements on this topic, to draft and issue a report on AI accountability policy development, focusing especially on the AI assurance ecosystem. Some documents may include the text of parsed attachments.

DOCUMENT:
-----------------------
{{ document }}
-----------------------

Question: {{ question }}
A) Yes
B) No
C) Unsure

We used a temperature of 0.2, and made at least 3 queries, taking the most common answer as the actual answer.

Validation

We validated our results by creating a “gold” set of labels by answering each of the 15 questions for 18 hand-selected submissions. We selected these submissions for diversity, and ensured that even for the most-skewed questions, we had at least some examples of both positive and negative answers. We selected 18 submissions as a compromise between the cost of doing additional submissions and the need to have a sufficiently large and diverse sample to get meaningful results.

In addition to the responses from GPT-4, we used the crowd sourcing platform Prolific to collect a large amount of responses from human participants as well. In total we recruited 406 participants, each of whom answered questions about 6 submissions: 1 taken from the gold set and 5 others. Participants were allowed to participate in multiple surveys on different sets of submissions; the most involved participant completed 11 such surveys. We excluded submissions longer than 2,000 characters from human labeling to avoid having to force people to read extremely long documents. In total, human participants rated 929 unique submissions. We took several steps to ensure we conducted our human data labeling in a fair and ethical manner, see Appendix A for details.

We performed quality assurance on the human responses at multiple overlapping levels:

In practice, we found that all these steps contributed to increasing the quality of the human labeling data, and our final dataset contains a high level of agreement among participants, with a high level of confidence – the most common answer to any question about any submission was “Unsure” only around 1% of the time. The final instructions given to participants can be seen in Appendix B.

Our final results on our gold set are presented below in Table 1. For each question, we present 3 metrics of reliability:

  1. Agreement: the percentage of golden submissions for which the most common answer among human participants or the LLM matches the golden answer.
  2. Number of false positives: the number of golden submissions where the golden answer is “No” but the most common answer among human participants or the LLM is “Yes”.
  3. Number of false negatives: the number of golden submissions where the golden answer is “Yes” but the most common answer among human participants or the LLM is “No”.

Additionally, we present the same metrics for the LLMs, using the most common answer across 3 independent LLM runs as the modal LLM answer.

This data allows us to draw two conclusions:

  1. The LLM is slightly closer to the golden answer than human survey participants although the difference is small and not necessarily statistically significant.
  2. Both human participants and LLMs are biased and make more false positive errors (reporting that the author of a submission is concerned when they are not) than false negatives.

We also observe that the reliability of both human and LLM answers varies strongly between questions. To reinforce this finding, we also report two metrics computed across all submissions, golden or otherwise:

Overall, the high agreement between our gold label set, the scores made by other human participants, and the LLMs, gave us enough confidence in our methodology to feel comfortable presenting them in our post summarizing our takeaways from analyzing the NTIA RFC dataset.

Experimentation

Our process did not begin with any of the details described above in the sections about questions, methodology, and validation. Rather, they each evolved over time as we iterated towards creating a set of results in which we had sufficient confidence to feel comfortable sharing more broadly.

Initial exploration

We started the process by exploring what was possible.

We created an initial question list, evaluation set, and the LLM infrastructure to ask our questions across all submissions. Through this exercise we started to better understand what questions we should ask. We also realized a need to look closely at the data, the importance of having a representative evaluation set, and the need for human labeling to help scale up the creation of this evaluation. This process gave us a rough outline of all the experiments that we felt were worth running on our system to see how far we could push performance.

Filtering the dataset

After this initial exploratory work, our next step was to better understand how exactly we should handle particular intricacies of the dataset and which submissions we could not analyze. In particular, we elected to filter out submissions that did not match our criteria above (too short, too long, non-English, etc) which reflect fundamental limitations of our approach based on LLMs and human labeling. Because the large majority (95%) of submissions passed these filters, we were able to manually review the remainder to ensure that their exclusion did not substantially affect the final conclusions. This work also allowed us to notice issues like duplicate submissions or submissions that contained attachments, which we handled by adding the text of each attachment to the submission.

Deciding on the exact questions

Much of our work and experimentation was around selecting the questions themselves. Through our own internal preliminary investigation of the questions, we removed any questions where the answer was subjective or the wording of the question itself led to ambiguity. We found that even short answer or non-binary multiple choice questions led to disagreement among humans, enough so that we struggled to build enough confidence in the LLM responses.

We developed the questions and the golden set iteratively, using a combination of human labellers, language models, and our own judgements.

Through this iterative procedure, we were able to arrive at a set of questions that was sufficiently unambiguous to function as an objective “gold” data set. This is what allowed us to benchmark both the LLMs as well as future human participants against a set of data that we could actually trust was correct.

Collecting human data

We iteratively developed each of the quality assurance datasets as we began collecting small scale versions of our human label data. For example, in one of the first runs, we noticed that there was a large variability in how long participants took to read submissions and answer questions. This led us to explicitly instruct participants about how much time they should spend reading, which largely alleviated the issue.

Improving LLM performance for accurate responses

Throughout the project, we constantly experimented with various prompts and methods of eliciting answers from the LLMs for all the questions we considered running. We often evaluated their performance on a small number of examples by hand when iterating, and on a larger number of examples as we started developing the ground truth and other human data.

We conducted our LLM experiments with off-the-shelf language models, specifically OpenAI’s GPT-4 and Anthropic’s Claude-2, so that our experiments and results could be replicated by others.

Alternative approaches we tried

Early on, we tried to investigate RFC submissions by asking language models to summarize their content. While this worked to some extent, we found that summaries often hid important content and did not accurately reflect the original commenter’s concerns. We eventually abandoned this approach in favor of our final question-based approach.

Limitations

While we have tried to be as thorough as reasonably possible with this analysis, there remain many limitations at both the object level (e.g., what we can conclude about people’s concerns about AI) as well as at the meta level (e.g., how confident we can be in our conclusions).

Limitations of the analysis

In our view, the primary limitations of the analysis itself are as follows.

  1. A heavily biased, limited sample of respondents. Given the fact that this data was collected in response to a request for comments (that most people probably didn’t hear about or did not have the time or expertise to craft a response, especially those people who are most likely to be disadvantaged by AI in the future).
  2. The limited format of a request for comment. We can only really answer questions that are explicitly addressed in the comments themselves, and have no ability to interact with the individuals who created those submissions. We cannot ask follow up questions.
  3. The limited knowledge of the respondents. If our true objective is to actually regulate, we want to know what the real risks and future harms and dangers of AI are. There is no guarantee that asking random individuals will give us very good coverage on those questions.
  4. The inherent limits of summarizing data. Any analysis that is shorter than the original dataset must have thrown away some of the data, and depending on what that data is, we could easily be losing some very interesting and valuable insights.

These limitations are extremely serious. One should not construe our analysis as saying something about the broader questions about AI risks, dangers, and harms because the underlying dataset simply does allow such broad statements.

Limitations of our methodology

At the object level, our analysis suffers from additional limitations: did we ask the right questions? Did we phrase them correctly? Do those questions cover the issues that were actually raised in the responses? Are the questions sufficiently unambiguous? Even taking as fixed the questions we ultimately decided to ask, there are also limitations in how confident we can even be in our answers to those questions.

However, unlike the limitations on our analysis, the limitations on the methodology are somewhat less serious. However, there are factors that could have introduced noise or error into our answers:

  1. Our questions, models, and humans are each biased in their own way. There may be bias from the selection of our participants on Prolific.
  2. We cannot ask the exact same questions to human labelers and language models. Unlike human participants, the LLM prompt contains additional instructions that human participants did not see.
  3. Some questions may not be possible to convert to a binary yes/no answer. Some questions faced difficulties in conversion leading to ambiguity in answers.
  4. Some responses had to be filtered out in order to make analysis practical. This poses a fundamental challenge to the scalability of this type of work, as the cost is directly proportional to the number of individuals who submitted comments.

There are certainly other limitations of our methodology, though overall, we feel relatively confident that, at least for the questions we ended up with and the dataset we were working with, we were able to answer questions in a way that seems likely to have been relatively accurate. However, it is critical to remember that this was only possible because of our iteration on those questions.