segmentation fault because of sorting function

The clang compiler giving following warning message when compiling your code (the compiler, you are using to compile, must be giving similar warning message)

prg.c:28:14: warning: incompatible pointer types passing 'unsigned char [40]' 

to parameter of type 'unsigned char **' [-Wincompatible-pointer-types]
  sort_array(test,SIZE);
             ^~~~
prg.c:4:32: note: passing argument to parameter 'arr' here
void sort_array(unsigned char *arr[],int n)
                               ^
1 warning generated.

One suggestion - Don't ignore the warnings, they are there for some reason.

Warning message indicating that the type of test is not compatible with type of sort_array() parameter arr. The type of test is unsigned char [SIZE] whereas the type of arr is array of unsigned char pointers.

You want to sort the array test in sort_array(). There are two ways you can achieve this:

1. Either pass the pointer to the first element of array test to sort_array().
2. Pass the test array pointer to the sort_array().

The difference in pointer to first element of array and pointer to array is often confusing to the programmers.

Lets discuss first approach - pass the pointer to the first element of array test to sort_array()
When you are doing this

sort_array(test,SIZE);

that means you are passing the pointer to first element of array to the sort_array() because

&test[0] -> &(*(test + 0)) -> &(*(test)) -> &(*test) -> test

The type of element of test array is unsigned char, so the type of arr parameter of sort_array() should be unsigned char *. In this case the sort_array() prototype should be:

void sort_array(unsigned char *arr, int n);

In the sort_array(), you can access the elements of array like this - a[i], where i represent the i^th element of array test. The sort_array() :

void sort_array(unsigned char *arr, int n) {
    unsigned char tmp;

    for (int i = 0; i < n-1; i++) {
         for(int j = i + 1; j < n; j++) {
             if(arr[j] > arr[i]) {
                tmp = arr[i];
                arr[i] = arr[j];
                arr[j] = tmp;
             }
         }
    }
}

Note that in this approach, you need to pass the size of array to be sort. But this also gives flexibility that you can use the sort_array() to sort different sizes of array of type unsigned char [].

Now the second approach - Pass the test array pointer to the sort_array()

The type of test is unsigned char [SIZE], so pointer of its type will be unsigned char (*) [SIZE]. That means when you are passing the pointer to test array to sort_array() like this

sort_array(&test);

the type of arr parameter of sort_array() prototype should be:

void sort_array(unsigned char (*arr) [SIZE]);

Note that, in this case you don't need to pass the array size to sort_array() because the pointer knows the size of array it can point to.

When you do

sizeof (*arr);

You will get

40     // because the SIZE is 40

In sort_array(), you can access the elements of test array like this - (*arr)[i] where i represent the i^th element of array test. When using this approach, you can do:

#include <stdio.h>

#define SIZE 40

void sort_array(unsigned char (*arr)[SIZE]) {
    unsigned char tmp;

    for (int i = 0; i < SIZE; i++) {
         for(int j = i + 1; j < SIZE; j++) {
             if((*arr)[j] > (*arr)[i]) {
                tmp = (*arr)[i];
                (*arr)[i] = (*arr)[j];
                (*arr)[j] = tmp;
             }
         }
    }
}
int main(void) {

   unsigned char test[SIZE] = { 34, 201, 190, 154,   8, 194,   2,   6,
                               114, 88,   45,  76, 123,  87,  25,  23,
                               200, 122, 150, 90,   92,  87, 177, 244,
                               201,   6,  12,  60,   8,   2,   5,  67,
                                 7,  87, 250, 230,  99,   3, 100,  90};

   sort_array(&test);

   for(int i = 0; i < SIZE; i++)
   {
      printf("%u", test[i]);
      (i+1) != SIZE ? printf(" ,") : printf("\n");
   }

   return 0;
}

Output:

250 ,244 ,230 ,201 ,201 ,200 ,194 ,190 ,177 ,154 ,150 ,123 ,122 ,114 ,100 ,99 ,92 ,90 ,90 ,88 ,87 ,87 ,87 ,76 ,67 ,60 ,45 ,34 ,25 ,23 ,12 ,8 ,8 ,7 ,6 ,6 ,5 ,3 ,2 ,2

Note that when printing the sorted array, I am using %u format specifier. When you print the ASCII character using %c, printf() will print the ASCII character corresponding to decimal value test[i] but remember that ASCII character from 0 to 31 are non printable characters and your test array has values in this range.

The limitation of using pointer to array approach is that the arr parameter of sort_array() will expect the address of array of type unsigned char [SIZE]. If you pass address of an unsigned char array of different size other than SIZE then compiler will throw warning message.

Of course, the pointer to first element of array approach seems more appropriate in your case but its good to be aware of the concepts.