Question about "missing" call coverage

Clang Frontend Using Clang
1

Hello everyone,

I have a question about call coverage as e.g. recommended in ISO 26262.

With call coverage I mean coverage refers to the percentage of executed software sub-programs or functions with respect to each implemented call of these sub-programs or functions in the software.

So far, I haven’t been able to find much information on this. Is call coverage already being measured but just not exported, or is it not measured at all? Does anyone have insights or experience with this? Or do I miss something?

Thanks in advance!

2

Isn’t that a different name for function-coverage?
That is covered:
https://clang.llvm.org/docs/SourceBasedCodeCoverage.html#interpreting-reports

3

When I read the text like this, it does sound similar (perhaps a little confusing). However, function coverage behaves as I would expect. If every function that exists has been called at least once, function coverage = 100%. Call coverage, or function call coverage (FCC) as I have also heard it referred to, also measures whether all individual calls have been called at least once.

For example:

int foo() { return 42; }

int bar() { return 666; }

int main() {
  auto res = foo();
  res = bar();
  res = foo();

  if (false) {
    res = foo();
  }
  return 0;
}

here the function coverage would be 100% and the FCC for the function “foo” would be 66.67%, since only 2 of 3 possible calls were actually called.

4

That is indeed not supported. Branch coverage is most likely the closest as it will drop the coverage where the function is called. Though from what I can find online, other coverage metrics like the MC/DC are also accepted.

Back to your question. That metric is not measured at all.

5

Hi @uthmanna

We implemented a simple downstream solution for Call Coverage that looks at existing region coverage and produces a metric in the report view for the subset of regions that contain calls. IIUC, you want to know the percentage of executed calls at the call-site. Nothing is produced in the source-file view, where it is less useful (at least for our embedded use cases).

Because it’s subsumed by region coverage, and we have more aggressive metrics like MC/DC, I think you’re the first to really ask about it. We could explore upstreaming that work.

1 Like
6

@evodius96 I think it’s an extension of function coverage that also measures whether the function is executed at all points where it is used. That could be problematic if, for example, there is a location missing where the input might be nonsense.
I’m not sure what you mean by downstream solution. If I understand correctly, this coverage doesn’t really help me at this point, except that I know where there is a gap. I would like to have the information from the function’s perspective. I think it’s a property of the function to have call sites and to know whether they were called.

I can now write a plugin that runs during compilation and writes out the data. I then have to spend a lot of time evaluating it again afterwards. I find that a bit tedious and would prefer an integrated solution.

I am also willing to implement it if it is considered okay to include another important metric. The question for me now would be: do the coverage data and the model already provide this information? Or would this also have to be added?

Thank you very much for your answer!

7

It’s a solution my company implemented in our downstream toolchains that has not yet been upstreamed. In other words, it is not available to the upstream community for anyone to use.

Our downstream solution is simple and not to the level of detail you’re looking for. In the embedded use cases we deal with, it’s sufficient to have an overall metric that tells the users the percentage of call-sites actually executed across the whole of the application. Mechanically, this information is already tracked in LLVM’s region coverage. We just implemented a means to specifically distinguish regions that contain calls.

However, I see where the level of detail you’re asking for would be useful: being able to look at an individual function and prove that all calls to that function were executed. I don’t think that this would be difficult to implement. The counter data is already tracked via region coverage. It seems to me that clang would have to build a table in order to tie each function to the set of coverage regions that contain a call to that function. Note that this may be difficult with indirect, pointer-based function calls.

During visualization, llvm-cov would extract that data for each function and then visualize the region counters tied to that function’s corresponding call-site regions.

-Alan