No thing wrong with your code, you just used the function the wrong way.
To use a Function in Matlab you need to call it, NOT to run it directly.
This is your original function code
function [out] = detj(in1,in2,in3,in4)
%DETJ Summary of this function goes here
% Detailed explanation goes here %N is 2*50 Matrix
global N r s
out=zeros(2,2);
for m=1:2
for n=1:2
out(m,n)=(1/8)*(((N(1,in1)-N(1,in3))*(N(2,in2)-N(2,in4))-(N(2,in1)-N(2,in3))*(N(1,in2)-N(1,in4)))-r(1,m)*((N(1,in3)-N(1,in4))*(N(2,in1)-N(2,in2))-(N(2,in3)-N(2,in4))*(N(1,in1)-N(1,in2)))+s(1,n)*((N(1,in2)-N(1,in3))*(N(2,in1)-N(2,in4))-(N(2,in2)-N(2,in3))*(N(1,in1)-N(1,in4))));
end
end
end
To use your function I tried a simple code,
clc;clear;
global N
global r
global s
N=ones(2,50);
r=ones(2,2);
s=ones(2,2);
detj(1,2,3,4)
And I got answer as
ans =
0 0
0 0
So no thing wrong with your function, just you need to know how to use it.
Let me know if my answer is not clear else good luck.
detjwithout any input argumentdetjand where you createNandin1variables.