[RFC] Collecting Community Input on Qualification of LLVM Tools & Libraries

[petbernt]

This RFC builds on recent discussions in the <LLVM Qualification WG https://llvm.org/docs/QualGroup.html>. Its goal is to collect community perspectives on expectations and priorities for qualification of LLVM tools and libraries.

Motivation

Qualification of compilers, related tools, and runtime libraries is a recurring challenge in domains where functional safety standards such as IEC 61508 and ISO 26262 apply. A number of companies and projects use LLVM in these contexts, but approaches to qualification vary widely, and there is currently little shared guidance.

Over the past few months, a small group of contributors has been meeting once a month to explore whether and how LLVM could better support such use cases. We are still at an early, exploratory stage, without a defined scope or deliverables yet, though we have discussed some possible directions (see <meeting materials https://llvm.org/docs/QualGroup.html#meeting-materials>).

At the same time, we would also like to gather input from the broader community on expectations, priorities, and areas of interest. Input from the community will help us understand which areas truly matter to LLVM users.

Please keep in mind that we’d like to listen to expectations and requirements from the LLVM community, but we don’t have a dedicated funding or staff working full time on this, so there are limitations to what we can achieve. We cannot commit to delivering all requests. Depending on your feedback, we’ll consider whether to refine scope, identify pilot areas, or simply gather more knowledge.

Questions for Feedback

We would especially welcome your thoughts on the following:

Useful outputs

What kinds of outcomes would be most valuable?

(e.g., lightweight templates, traceability guidance, process examples, references to standards)

Relevant LLVM tools

Which tools are most relevant for qualification in your view?

(e.g., clang, clang-tidy, code coverage, others)

Relevant LLVM libraries

Which libraries are most relevant for qualification in your view?

(e.g., compiler-rt, libc++, libc, libc++abi, or other runtime components)

Areas of focus

Should qualification efforts focus more on compilers, runtime libraries, or supporting tools?

(e.g., compiler vs. runtime libraries, testing vs. documentation, qualification processes vs. supporting tools)

Expectations of such an effort

Given that this is a volunteer-driven, exploratory discussion, what would you realistically expect from it?

Scope and Boundaries

• The purpose of this RFC is to collect perspectives and map expectations for qualification of LLVM components upstream.

• It does not propose starting qualification work or deliverables for a specific project; nevertheless, inputs are welcome from specific projects, as they can help define and shape what value this group can provide to them.

• Any future direction will depend on interest, feasibility, and available resources.

Note: Safety standards distinguish between tools and runtime libraries. Tools (e.g. compilers, analyzers) are qualified under tool qualification guidelines, while libraries are treated as software components that run in the safety-critical system and require measures like coverage analysis. For this RFC, we welcome input on both, even though the approaches differ.

Next Steps

• Feedback here will help us understand whether and how LLVM could support broader LLVM community expectations on upstream qualification efforts for LLVM tools and libraries, and where collaboration might be most useful.

• Please share your views, whether it’s a single priority, a tool you think matters most, or broader guidance. Even short comments are valuable.

[uwendi]

Just a quick note for readers of this RFC: this is an important conversation for anyone who depends on LLVM components in structured development processes.
If you’ve worked with LLVM in safety-critical or high-assurance contexts, your input on what’s missing (or already working well) would be very valuable. Even brief examples or insights could help shape practical next steps for the group.

[smithp35]

Our proprietary toolchain, which included a fork of clang, ( Arm Compiler for Embedded FuSa ) went through the qualification process. We’ve now separated out functional safety qualification from the development team so these are my thoughts based on prior experience.

The RFC seems aimed at collecting input from using llvm tools or runtime libraries in Functional Safety. I recommend that there’s a question, possibly a separate RFC, for what costs the community is willing bear when it comes to qualification. For example I expect that adding traceability annotations to tests is OK, but expecting the wider community to maintain them is another. It could also be useful to try and find if there is any other wider community, security comes to mind, that could benefit from similar outputs.

Useful Outputs

As an open-source project, I expect that a qualification approach will lean towards a black-box tool/library meets its specification. For our proprietary toolchain that used clang as its compiler we used the tools documentation as the qualified scope and a specification that could be tested. Clang, and in particular lld has large gaps in documentation that are either outsourced to the GNU documentation or are not documented at all beyond source-code/commit-log/reviews. Improving documentation would be valuable to the wider community as well as allowing vendors to test the behaviour against the documentation.

A community position on builtins, the implementations of these are like library functions rather than a translation of user-code, but they are defined in the compiler and have no source code. Do these require tool or library qualifications. In any case some indications of what builtins are available for a target, and whether there are any tests for them in llvm-project would be valuable.

Tools (such as via clang-tooling, or built in to clang) for users to derive qualification data, or identify areas that need additional human intention would be very helpful. For example:

• Which language extensions, including attributes and pragmas have been used by the program.
• Which builtins (that may need additional checking) have been used implicitly/explicitly by the program.
• What dynamic memory allocation sizes have been used throughout the program. This could be used to build a custom allocator suitable for use in a safety critical environment.

A tool that could do some kind of traceability analysis on the llvm test suites would also be valuable. For example given a command-line option, language extension, language feature, can it find the existing tests that intersect it. This may need annotations added to the existing tests (see How Ferrocene qualified the Rust Compiler) https://www.youtube.com/watch?v=_ITnWoPvMKA

I expect that safety-related software doesn’t use all of LLVM’s features, and realistically not all of its Targets. Could the safety community identify subsets of functionality to concentrate one. For example no exceptions, a set of supported targets.

Relevant Tools:

For our proprietary toolchain Arm Compiler for Embedded, we used clang (targeting Arm and AArch64) but this was combined with our own proprietary linker and binutils. For future projects I expect that the tools would include lld and the “user-facing” llvm-* utilities, where “user-facing” is loosely defined as the llvm equivalents of the GNU binutils. The most important of the binutils would be those with outputs that are incorporated into the safety-related system, for example llvm-ar and llvm-objcopy. IIUC an assessment would need to be made for all tools used in the llvm build and test process too, although not to the same degree as clang. To date we have not included a debugger, relying on third-parties for that.

Relevant LLVM Libraries:

In terms of priority I think (libc++ and libc++abi), or at least a subset of it that is most likely to be used by automotive developers, is the most valuable. For Arm Compiler for Embedded we had our own proprietary qualified C-library, but no such equivalent for C++. I expect many LLVM users to be in the same situation. We decided to not support exceptions and RTTI to keep the qualification costs down.

For future products we are interested in LLVM lib C and the compiler-rt builtins. The rest of compiler-rt would not be expected to be used in production.

Of major interest would be a qualified bare-metal profiling runtime to be used with clang’s MC/DC code coverage.

Areas of Focus:

In terms of what is most valuable, then I would say runtime libraries as these are significantly more work than tools. However I recommend that you concentrate at first on what is most easily achievable upstream to get some momentum. For example improving tool and library documentation to act as specifications would benefit the wider community.

Expectations of such an effort:

I think anything like a qualified open-source release from the LLVM foundation of either the tools or libraries is unrealistic. I expect that the community will encourage contributions that benefit everyone, will be willing to accept contributions that are neutral to everyone else, but will likely push back if it involves changing existing behaviour. As an example the GN build system support, this is maintained by the people that use it, and those that don’t use it can ignore it.

I would like to see some work identified that:

• Can be done incrementally.
• Is beneficial to more than just functional safety. For example a SBOM for llvm could benefit security as well as safety.
• Does not impose additional requirements on the wider llvm-community’s existing practice.

This can be contributed over time by parties interested in sharing qualification costs while minimising the impact on the remainder of the community.

Additional tools that can derive qualification information, such as traceability, would likely be the easiest thing to add without facing pushback from the community on process.