How do I create arrays within an array in C without a predefined size?

You can make use of a VLA (Variable length array).

You need to rearrange your code like

int noOfStudents = -1, noOfItems = -1;
int *grades;                                //is it used?
int i;

printf("Please enter number of students\n");
scanf("%d", &noOfStudents);

//fail check

int *a[noOfStudents];             // this needs to be proper.

//VLA

printf("Please enter number of items\n");
scanf("%d", &noOfItems);

//fail check

for (i = 0; i < noOfStudents; i++)
{
    a[i] = malloc(noOfItems * sizeof(a[i]));   //do not cast
}

You want a bi-dimensional array, to hold the list of lists of integer items. You can do that by declaring a pointer on integer pointers.

So you want to declare

int **a;

then

printf("Please enter number of students\n");
if (scanf("%d", &noOfStudents)==0 && noOfStudents<=0)  // bonus: small safety
{
    printf("input error\n");
    exit(1);
}
// now we are sure that noOfStudents is strictly positive & properly entered
a = malloc(sizeof(int*)*noOfStudents);

Then you have your array of pointers allocated, and the rest of your code is OK (don't cast the return value of malloc BTW)

A variant is:

a = malloc(sizeof(*a)*noOfStudents);

(so if type of a changes, the size follows, not that it matters here since they're all pointers)

Use pointer instead of array and allocates the memory of that pointer dynamically using malloc or calloc function.

Like this:

int *a;

a = malloc((sizeof(int)*noOfItems);

You may try the function malloc, which dynamically allocates memory and returns a pointer to it. Then you can cast the pointer to a pointer pointing to an array of a certain type.