creating an instance of another object in javascript

Simply you can create an instance of function and you know a function is an object. Why can't we create an instance of other user-defined objects?

It’s not quite correct to say “you can create an instance of function”. The new keyword is a bit misleading - it makes JavaScript look like it implements object-orientation using classes, when in fact it doesn’t.

What you’re actually doing with new A() is creating an object using the constructor function A. The new keyword tells the JavaScript interpreter to return an object from A - specifically the object referred to as this inside of A.

EDIT: How can I change b so that I can create an instance of that?

In your example, b is an object (var b={};). If you change b into a constructor function, then you can create objects using it. (By convention, constructor functions in JavaScript start with capital letters.)

So:

function B () {
}

var c = new B();

You can add things to the prototype object of B, and they’ll be accessible on c too (and on any other objects you create using B):

function B () {
}

B.prototype.NAME = 'B';

B.prototype.hello = function () {
    alert('Hello!');
}

var c = new B();
c.NAME // 'B'
c.hello() // alerts 'Hello!'

Short answer: The new operator requires its operand to have a special internal method [[Construct]] that generic objects do not have.

Long answer:

11.2.2 The new Operator
The production NewExpression : new NewExpression is evaluated as follows:
    1. Evaluate NewExpression.
    2. Call GetValue(Result(1)).
    3. If Type(Result(2)) is not Object, throw a TypeError exception.
    4. If Result(2) does not implement the internal [[Construct]] method, throw a TypeError exception.
    5. Call the [[Construct]] method on Result(2), providing no arguments (that is, an empty list of arguments).
    6. Return Result(5).

    The production MemberExpression : new MemberExpression Arguments is evaluated as follows:

    1. Evaluate MemberExpression.
    2. Call GetValue(Result(1)).
    3. Evaluate Arguments, producing an internal list of argument values (11.2.4).
    4. If Type(Result(2)) is not Object, throw a TypeError exception.
    5. If Result(2) does not implement the internal [[Construct]] method, throw a TypeError exception.
    6. Call the [[Construct]] method on Result(2), providing the list Result(3) as the argument values.
    7. Return Result(6).

You can also do

var b = new a.constructor();