Best way to convert an InputStream object to String

Question

I have an InputStream object which contains several million file informations (Name, create date, author etc.) in XML format. I've already tried to convert it to String using IOUtils.copy method, but since the size of that information is pretty large it throws an java.lang.OutOfMemoryError, after running for few minutes.

Increasing JVM memory is not an option, since the number of files, which I collect info from, is increasing forever. So can someone suggest me what should I do to solve the issue?

What's the concrete implementation of the InputStream? Is it a ByteArrayInputStream, for example? — M A
– M A, Commented Sep 12, 2014 at 10:43
If the input is just too massive, your other options are (a) First trying to ETL it into a database or (b) using Hadoop or something similar — ControlAltDel
– ControlAltDel, Commented Sep 12, 2014 at 10:45
It just seems to be the wrong approach to convert the huge data into one string. What do you want to do with the string? — fgwe84
– fgwe84, Commented Sep 12, 2014 at 10:49
If you can't fit your object in memory, then avoid cases that requires you to store it in memory. Work with it as stream. — Alex
– Alex, Commented Sep 12, 2014 at 10:49
If the data is too large to store in memory and you can't increase memory then your options are limited - either process as a stream, extracting what you need, or persisting the data somewhere for later access. — Romski
– Romski, Commented Sep 12, 2014 at 11:35

folkol · Accepted Answer · 2014-09-12 11:36:48Z

2

The problem that you are having is the very reason stream based IO exists, it is simply not viable to slurp huge amounts of data into memory before consuming it.

Parse your stream as... a stream! See the Oracle tutorials for more information on stream based XML parsing using SAX.

XMLReader xmlreader =
    SAXParserFactory.newInstance().newSAXParser().getXMLReader();
xmlreader.setContentHandler(new ContentHandler() {
    ...
});

xmlreader.parse(new InputSource(myInputStream));

answered Sep 12, 2014 at 11:36

folkol

4,88324 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Best way to convert an InputStream object to String

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related