It is hard to figure out what you are actually asking here, but the application is behaving exactly as I would expect.
Strings are immutable and the garbage collector doesn't take them out. isn't it
Both mutable and immutable objects may be garbage collected in Java.
The actual criterion that determines whether an object is actually garbage collected is it reachability. In simple terms, when the garbage collector figures out that the application can no longer use an object, the object will be deleted.
In both of your applications, objects of roughly the same size are being created once every 10 milliseconds. In each iteration, a new object is being created and its reference is being assigned to s, replacing the previous reference. This makes the previous object unreachable, and eligible for garbage collection. At some point, the Java VM decides to run the garbage collector. This gets rid of all of the unreachable object ... and the application continues.
I read that common Strings are not collected ever by the garbage collector, is that false?
This is false on two counts:
Strings created by new String(...), String.substring(...)1 and so on are no different from any other Java object.
Strings that are interned (by calling String.intern()) are stored in the string pool which is held in the PermGen heap2. However, even the PermGen heap is garbage collected, albeit on longer timescales that the heap in which objects are normally created.
(Once upon a time, the PermGen heap was not garbage collected, but that was changed a long time ago.)
As @MichaelBorgwardt correctly identified, you were confusing string objects (in general) with string objects that correspond to string literals. The latter are interned automatically, and end up in the string pool. However, they may still be subject to garbage collection. This can happen if the parent class is unloaded and nothing else references the literal.
1 - In Java 6 and earlier, there is a difference between strings created using new String and using String.substring. In the latter case, the original string and the substring would share the backing array that holds the string's characters. In Java 7, this changed. String.substring now creates a new backing array.
2 - From Java 7 onwards, the string pool is just a (hidden) data structure in the normal heap. From Java 8 onwards, the PermGen heap no longer exists.