This is the declaration of the function and the compiler gives: "error: array type has incomplete element type" in both the declaration and definition. I don't see the error.
void calculate(float matrM[][], int origMatr[][]);
C doesn't have very powerful arrays, especially not when calling functions. Basically, the only data that gets sent to the function at run-time is the address of the first element. This means that the dimensions must be known in all directions except the topmost, in order for the compiler to be able to generate indexing code. Thus, you can't have two unknown dimensions, that makes indexing impossible and thus isn't allowed.
You can't declare an "any size" array like that, you must specify all dimensions except one, otherwise there's no way to compute the address of a random element in the array.
It's often better to break it down as:
void calculate(float *out, const float *in, size_t width, size_t height);
which will let you treat any memory block as float arrays, but requires you to specify the size, and the write the indexing manually:
out[y * width + x] = in[y * width + x];
Your "inner" dimensions must have a size, e.g.:
void calculate(float matrM[][5], int origMatr[][7]);
Otherwise, the compiler wouldn't know how to generate indexing code to access a particular element. Note also that these sizes must be compile-time constant expressions.
For more details, see How do I write functions which accept two-dimensional arrays when the width is not known at compile time? from the C FAQ.
If you have a modern C compiler (C99 would do) you can have the "extra" dimensions as expressions of other function parameters that precede
void calculate(size_t n, float matrM[][n], int origMatr[][n]);
you'd just have to be careful that you'd have equivalent parameters in your declaration (the one I gave) and the definition (the one that provides the implementation of the function).
You only get one "free" dimension (the first one) when you're passing an array to a function. The correct declaration will look more like:
void calculate(float matrM[][N], int origMatr[][M]);
N and M do not need to be constants, as of C 1999.
(float matrM[3][], int origMatr[3][])