0

How to join list of millions of values into a single String by appending '\n' at end of each line -

Input data is in a List:

   list[0] = And the good south wind still blew behind,
   list[1] =  But no sweet bird did follow,
   list[2] =  Nor any day for food or play
   list[3] =  Came to the mariners' hollo!

Below code joins the list into a string by appending new line character at the end -

String joinedStr = list.collect(Collectors.joining("\n", "{", "}"));

But, the problem is if the list has millions of data the joining fails. My guess is String object couldn't handle millions line due to large size.

Please give suggestion.

Improve this question
2
  • What error or exception you get? If is something related to the garbage collector, try to give the VM more memory. Commented Jan 15, 2020 at 15:37
  • it's not the String, it's you memory that is not enough. millions of Strings requires a lot of space Commented Jan 15, 2020 at 16:26

2 Answers 2

Reset to default
1

The problem with trying to compose a gigantic string is that you have to keep the entire thing in memory before you do anything further with it.

If the string is too big to fit in memory, you have only two options:

  1. increase the available memory, or
  2. avoid keeping a huge string in memory in the first place

This string is presumably destined for some further processing - maybe it's being written to a blob in a database, or maybe it is the body of an HTTP response. It's not being constructed just for fun.

It is probably much more preferable to write to some kind of stream (maybe an implementation of OutputStream) that can be read one character at a time. The consumer can optionally buffer based on the delimiter if they are aware of the context of what you're sending, or they can wait until they have the entire thing.

Preferably you would use something which supports back pressure so that you can pause writing if the consumer is too slow.

Exactly how this looks will depend on what you're trying to accomplish.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

0

Maybe you can do it with a StringBuilder, which is designed specifically for handling large Strings. Here's how I'd do it:

StringBuilder sb = new StringBuilder();
for (String s : list) sb.append(s).append("\n");

return s.toString();

Haven't tested this code though, but it should work

Improve this answer

5 Comments

Above code works, as it has a return statement instead of storing it in a string variable. If instead of return, ## String var = s.toString(); ## it doesn't work. Jvm memory size exceeds.
That’s not different to what the joining collector does internally.
@ShakthiRaj the memory needed by a String object does not change, whether you return the reference or store it into a local variable.
So, does the string memory size depends on jvm heap memory.? If that's the case, do java processing depends on jvm memory. Because I'm trying to do this process for millions of lines. Is there any other way to achieve this without making changes to memory even if it means of processing atleast lakhs of lines
@ShakthiRaj I don’t understand what you are asking. Of course, to hold a large string in memory, you need to have enough memory. If this example happened to work in one test, but failed after a subtle, irrelevant change, it means the string may barely fit into the memory, but since you want to do something more than just having a string in the memory, there is no way around adding more memory.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.