The nm_mset method uses standard conventions for Ruby C extensions. The argc variable is a an integer, specifying number of parameters being sent. Then argv is a C array/pointer of type VALUE * which must contain all the Ruby-ish parameters (for an assignment, the last of these params is the value of the right hand side). Thirdly self is the Ruby Object receiver - but do remember that this might be either the class or an instance of it depending on how the method has been defined for Ruby.
To call it, you need to construct a VALUE * array with each parameter a VALUE, sent in the correct order.
I think you want to do something like this:
// Assuming you have following values already processed
// Assignments are not shown here - that is work for you!
VALUE self;
VALUE t;
int self_size;
int new_size;
//////////////////////////////////////////////////////////////////
//
// Assuming you set above values correctly, the following should
// work.
int i;
// Get reference to :*
VALUE asterisk = ID2SYM( rb_intern("*") );
// NB +1, we need to add call to self[ *right_params ] to end later
VALUE *left_params = ALLOC_N( VALUE, new_size + 1 );
for(i=0;i<new_size;i++) { left_params[i] = asterisk; }
VALUE *right_params = ALLOC_N( VALUE, self_size );
for(i=0;i<self_size;i++) { right_params[i] = asterisk; }
// equivalent to self[ *right_params ],
// but we push it straight into end of left_params ready for next step
left_params[ new_size ] = nm_mget( self_size, right_params, self );
// equivalent to t[ *left_params ] =
// Note +1 to size again, the last item is the value to assign
nm_mset( new_size + 1, left_params, t );
// Tidy up
xfree( left_params );
xfree( right_params );
return t;
printf?