update multiple column in table using same table data in postgres?

One method is to do the calculation in a CTE. This is most helpful if you have a unique id or primary key on the data:

with newvals as (
      select pc.*,
             (((((totalprice * commission  )/100) + shippingcharge + fixed_fee))+
            ((((totalprice * commission  )/100) + shippingcharge +
            fixed_fee)) * (service_tax) /100) as new_markeplace_fee
      from fk_pay_cal
     )
update fk_pay_cal
    set market_place_fee = new_markeplace_fee,
        settlment_value = totalprice - new_marketplace_fee
    from newvals
    where newvals.id = fk_pay_cal.id;

This behaviour is intended. Any reference to a column on the right side of the assignment uses the "old" value of that column. Otherwise you couldn't do things like update foo set x=y, y=x;

As I guess you do not want to repeat the formula, you need to use a two step approach to update this in a single query:

with calculated as (
  select pk_column, 
         (((((totalprice * commission  )/100) + shippingcharge + fixed_fee))+
            ((((totalprice * commission  )/100) + shippingcharge +
            fixed_fee)) * (service_tax) /100) as marketing_fee
  from fk_pay_cal 
  where pk_column = ... --<< make sure you only select those that you want to change
) 
update fk_pay_cal 
  set marketplace_fee = c.marketplace_fee,
     settlment_value = totalprice - c.marketplace_fee
from calculated c 
where c.pk_column = fk_pay_cal.pk_column;

You need to replace pk_column with the real name of the primary key column of your table.

Edit if you do not have a primary key (why?) then replace pk_column with ctid which is an internal identifier for a row that can be used in a situation like this (but it is not suitable as a general replacement for a primary key)

If you intend to update all rows in the table, then of course you need to remove the where clause inside the CTE.

Another option (which might be better in the long run anyway) is to create a function that encapsulates this formula:

create or replace function calc_fee(p_pay fk_pay_cal) 
  returns numeric
as
$$
  select (((((p_pay.totalprice * p_pay.commission) / 100) + p_pay.shippingcharge + p_pay.fixed_fee)) 
 + ((((p_pay.totalprice * p_pay.commission  )/100) + p_pay.shippingcharge + p_pay.fixed_fee)) * (p_pay.service_tax) /100);
$$
language sql
stable;

Then you can use that function in the update statement:

update fk_pay_cal 
  set marketplace_fee  = calc_fee(fk_pay_cal), 
      settlment_value = totalprice - calc_fee(fk_pay_cal);

As the function is declared stable Postgres will only call it once per row.

I assume that ID is the PK column

   with cte (
          update 
    fk_pay_cal 
set 
    marketplace_fee = (((((totalprice * commission  )/100) + shippingcharge + fixed_fee))+
            ((((totalprice * commission  )/100) + shippingcharge +
            fixed_fee)) * (service_tax) /100) 
    returning marketplace_fee,id)
    )
   update 
    fk_pay_cal set settlment_value = cte.marketplace_fee  from cte where cte.id=fk_pay_cal.id

or using ctid

 with cte (
          update 
    fk_pay_cal 
set 
    marketplace_fee = (((((totalprice * commission  )/100) + shippingcharge + fixed_fee))+
            ((((totalprice * commission  )/100) + shippingcharge +
            fixed_fee)) * (service_tax) /100) 
    returning marketplace_fee,ctid)
    )
   update 
    fk_pay_cal set settlment_value = cte.marketplace_fee  from cte where cte.ctid=fk_pay_cal.ctid