Use libnm or popen(nmcli) to configure network programmatically?

I think both are viable, sensible options; nmcli's man page even says:

Typical uses include:

• Scripts: Utilize NetworkManager via nmcli instead of managing network connections manually. nmcli supports a terse output format which is better suited for script processing.

So, being called externally is definitely a design goal, and with --terse you will get output that the maintainers put thought into keeping stable for usage by computer programs parsing it.

I think libnm mostly makes more sense if

1. your C program already uses GObject anyways, and
2. you don't want to directly handle D-Bus calls yourself, but
3. also don't want to work very abstracted from the D-Bus API of NetworkManager.

Since the operations you describe are "large" anyways, meaning that the things they cause will take multiple milliseconds, maybe even seconds, the overhead of a single popen (which internally creates pipes, forks, invokes a shell, and tells the shell to execute a command) is really insignificant.

So, I'd go for invoking nmcli --terse {subcommand} from your C program. Whether you do that via the simple

FILE* output = popen("nmcli --terse foobar", "r");

which involves spawning a shell, or, more advanced, but maybe advantageous if you want to do things like use nmcli connection monitor,

char* args[] = {"nmcli", "--terse", "foobar"};
char* nmcli_path = "/usr/bin/nmcli";
struct io_pipe {
  int read_end;
  int write_end;
};
struct io_pipe pipe_fd;

int success = pipe(&pipe_fd); // check success >= 0!
int child_pid = fork();       // check new_pid >= 0!
if(child_pid > 0) {           // we're the parent!
  close(pipe_fd.write_end);   // we don't need the write end
  /* Do whatever you would have done with the FILE* coming out of popen here,
   * e.g., 
   * fd_set watched_fds;
   * FD_ZERO(&watched_fds);
   * FD_SET(io_pipe.read_end, &watched_fds);
   * fcntl(io_pipe.read_end, O_NONBLOCK);
   * struct timeval timeout = {.tv_sec = 5, .tv_usec = 0};
   * char* buffer = NULL;
   * while(select(1,            // how many file descriptors?
                  &watched_fds, // which read descriptors
                  NULL, NULL,   // which write, exception descriptors
                  timeout       // timeval specifying timeout.
                 )) {
   *   if(!buffer) {
   *     buffer = malloc(4096);
   *   }
   *   int howmuch = read(pipe_fd.read_end, buffer, 4096);
   *   // Do clever stuff with the things you read!
   * }
   * free(buffer);
   */
  int wstatus;
  waitpid(child_pid, &wstatus, 0); 
  /* see `man 2 wait` for what you can do with wstatus */
} else {                      // we're the child!
  close(pipe_fd.read_end);    // we don't need the read end
                              // we connect our stdout to the write end:
  dup2(STDOUT_FILENO, pipe_fd.write_end);
  execv(nmcli_path, args);    // we replace ourselves with nmcli 
}

is fully up to you.