You understand that the underlying representation of List is an array, right? It doesn't have to be, but it is so that random access is constant time and the cost of adding is amortized by growing the array in chunks.
This means that removing from the "head" is going to be exactly the same cost as with an array as the same amount of work needs to be done.
Adding after a remove will be constant time since it's guaranteed that there will be space for the item that you're adding.
If you really need to have fast remove at either end and fast add at either end, but can live without constant time random access, you probably want to use a deque. If you're removing from the front and adding to the back, use a Queue which will be constant time for either remove or add.