0

I'm trying to query our instance of GitLab for flakey tests.

If I navigate to a URL of this format, https://our-gitlab-instance.com/the-group/the-project/-/pipelines/365771/tests/suite.json?build_ids[]=1671439, we get some nice JSON which lists stats about the tests.

I'm specifically looking for flakey test cases which looks something like this:

{
  "status": "success",
  "name": "The name of the test",
  "classname": "the/path/to/the/test.pw.ts",
  "file": null,
  "execution_time": 77.511,
  "system_output": null,
  "stack_trace": null,
  "recent_failures": {
      "count": 18,
      "base_branch": "develop"
  },
},

Initially, I was trying to use GraphQL:

query {
  project(fullPath: "path_to/my_project") {
    pipelines {
      nodes {
        testSuite {
          testCases {
            nodes {
              recentFailures {
                baseBranch
                count
              }
            }
          }
        }
      }
    }
  }
}

GraphQL seems to provide the right fields, however, testSuite requires an argument buildIds of type [ID!]!.

I can get the data I need with this query:

query {
  project(fullPath: "path_to/my_project") {
    pipeline(iid: "261264") {
      testSuite(buildIds: "1671439") {
        testCases {
          nodes {
            recentFailures {
              count
              baseBranch
            }
          }
        }
      }
    }
  }
}

But this involves creating a separate query for each job. As I'm wanting to enumerate over all builds over a given timeframe, this seems like a lot.

I have not yet found any REST endpoint that provides this information.

Are there any other options I haven't considered?

Improve this question

1 Answer 1

Reset to default
0

I have found a solution that involves three API calls. This solution is good example of an N + 1 problem.

  1. Fetch each failed pipeline:
GET :gitlab_url/api/v4/projects/:project_id/pipelines?status=failed&per_page=:per_page&page=:page&created_after=:created_after&source=parent_pipeline

This is an authenticated request that requires the PRIVATE-TOKEN header to be set to an access token with the required permissions.

This URL has the following params:

  • :gitlab_url - the URL of your GitLab instance
  • :project_id - the id of the project you are interested in
  • status=failed - only return failed pipelines
  • per_page=:per_page - the number of results to return (defaults to 20)
  • page=:page - the page to fetch
  • created_after=:created_after - return records after this date
  • source=parent_pipeline - by default, child pipelines are not included in the results. To return child pipelines, set source to parent_pipeline. I'm not yet sure if you need to query twice; once with the source params and once without
  1. Iterate over each pipeline, and it's jobs:
GET GET :gitlab_url/api/v4/projects/:project_id/pipelines?scope[]=failed&include_retried=true

This is an authenticated request that requires the PRIVATE-TOKEN header to be set to an access token with the required permissions.

This URL has the following params:

  • :gitlab_url - the URL of your GitLab instance
  • :project_id - the id of the project you are interested in
  • scope[]=failed - only return failed jobs
  • include_retried=true - include retried jobs in the response. Defaults to false.
  1. Fetch test cases for each job:
POST :gitlab_url/api/graphql

This is an authenticated request that requires the Authorization header to be set to an access token with the required permissions of the format Bearer :your_access_token.

Additionally, include the header Content-Type with value application/json.

This URL has the following params:

  • :gitlab_url - the URL of your GitLab instance

Set the body to the following:

query {
  project(fullPath: "productdev/insight") {
    pipeline(iid: ":pipeline_iid") {
      testSuite(buildIds: ":job_id") {
        testCases {
          nodes {
            name
            status
            classname
            recentFailures {
              count
              baseBranch
            }
            file
            executionTime
            systemOutput
            stackTrace
            attachmentUrl
          }
        }
      }
    }
  }
}

This query has the following params:

  • :pipeline_iid - This is the iid for the pipeline. this is not a spelling mistake; it is NOT the id.
  • job_id - The id of the job

This third query returns the JSON I am expecting:

{
  "status": "success",
  "name": "The name of the test",
  "classname": "the/path/to/the/test.pw.ts",
  "file": null,
  "execution_time": 77.511,
  "system_output": null,
  "stack_trace": null,
  "recent_failures": {
      "count": 18,
      "base_branch": "develop"
  },
},
Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.