Binary Exploitation - How to set GID of a SUID binary using shellcode?

According to setgid(2), the syscall will fail unless the calling process is privileged:

   EPERM  The calling process is not privileged (does not have the
          CAP_SETGID capability), and gid does not match the real group
          ID or saved set-group-ID of the calling process.

Unless you are running with the correct capabilities (specifically CAP_SETGID, provided by root), you cannot change your GID to a value which is not your real or saved GID. This limitation applies to any syscall that changes the GID. The manual page on credentials(7) is extremely helpful.

You can actually see where this is done in the kernel:

SYSCALL_DEFINE1(setgid, gid_t, gid)
{
    struct user_namespace *ns = current_user_ns();
    const struct cred *old;
    struct cred *new;
    int retval;
    kgid_t kgid;

    kgid = make_kgid(ns, gid);
    if (!gid_valid(kgid))
        return -EINVAL;

    new = prepare_creds();
    if (!new)
        return -ENOMEM;
    old = current_cred();

    retval = -EPERM;
    if (ns_capable(old->user_ns, CAP_SETGID))
        new->gid = new->egid = new->sgid = new->fsgid = kgid;
    else if (gid_eq(kgid, old->gid) || gid_eq(kgid, old->sgid))
        new->egid = new->fsgid = kgid;
    else
        goto error;

    return commit_creds(new);

error:
    abort_creds(new);
    return retval;
}