The problem with your function is that you have written it assuming that x is a single value, and then you have passed it a vector. Your code cannot cope with this.
If you try to run your code with a single value, eg: a=PdfMixUnfmBeta(15,0.4) your function should run.
Lets look at what is actually wrong with your code, when I tried running your code I was given the following error:
Output argument "y" (and maybe others) not assigned during call
This indicates that the lines which assign to y are never actually executed. This indicates that neither of the conditional statements (x < 0 and x >= 0) ever evaluate as true. The if statements expects a scalar logical value, but in your example it is provided with a vector of logical.
So, to fix this you either need to deal with your x values one at a time by wrapping it in a for loop, e.g.:
function y = PdfMixUnfmBeta(x_vec, alpha)
for x = x_vec
%do your function here
end
end
Alternatively, you can vectorize your code, which is by far the preferable solution:
y = zeros(1,length(x));
% where x is < 0 use this formula
y(x < 0) = (1-alpha) .*(1/2);
% where x is >= 0 use this formula instead.
y(x >= 0) = (1-alpha) .* (1/2) + (alpha .* 6 .* x .* (1-x));
In the above solution I use logical indexing and change some of the * to .*. You may find it useful to look up the difference between * and .* and to read up on vectorization.