Java heap space::Out of memory error

Question

I am trying to add 50000 datas in MySql dB that contains only 2 columns(id, description). However I am able to add till 30000 data but when it tries to add more datas it gives me the following error

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOf(Unknown Source)
    at java.util.Arrays.copyOf(Unknown Source)
    at java.util.ArrayList.grow(Unknown Source)
    at java.util.ArrayList.ensureExplicitCapacity(Unknown Source)
    at java.util.ArrayList.ensureCapacityInternal(Unknown Source)

I have also tried adding arguments in the VM arguments under Run Configuration. Also tried with changing the setDomainEnv.cmd inside weblogic>bin .Below is the code. Anyone who can help me with this::

public class SequenceGenerator {
    private static final String CHAR_LIST = 
            "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890";
    private static final int RANDOM_STRING_LENGTH = 5;

    private static final String jsonFilePath ="D:/Assignments/Sequence/file.json";
    static HashMap hm =  new HashMap();
    static Set mapSet = null;
    int count=1;
    /**
     * This method generates random string 
     * and stores the values in a hashmap
     */
    public void generateRandomString(){


        while(count<=50000){/*Change 10000 to 30000*/
            StringBuffer randStr = new StringBuffer();
            for(int i=0; i<RANDOM_STRING_LENGTH; i++){
                int number = getRandomNumber();
                char ch = CHAR_LIST.charAt(number);
                randStr.append(ch);
            }
            hm.put(count, randStr);
            mapSet = hm.entrySet();
            //System.out.println(randStr.toString());
            count++;
        }

    }

    /**
     * This method generates random numbers
     * @return int
     */
    private int getRandomNumber() {
        int randomInt = 0;
        Random randomGenerator = new Random();
        randomInt = randomGenerator.nextInt(CHAR_LIST.length());
        if (randomInt - 1 == -1) {
            return randomInt;
        } else {
            return randomInt - 1;
        }
    }

    public static DataSource getMySQLDataSource() throws Exception {
        Properties props = new Properties();
        FileInputStream fis = null;
        MysqlDataSource mysqlDS = null;

        try {
            fis = new FileInputStream("D:/Assignments/Sequence/db.properties");
        } catch (FileNotFoundException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
        props.load(fis);
        mysqlDS = new MysqlDataSource();
        mysqlDS.setURL(props.getProperty("MYSQL_DB_URL"));
        mysqlDS.setUser(props.getProperty("MYSQL_DB_USERNAME"));
        mysqlDS.setPassword(props.getProperty("MYSQL_DB_PASSWORD"));
        return mysqlDS;
    }

    public static void main(String[] args) throws Exception 
    {
        SequenceGenerator sg = new SequenceGenerator();
        sg.generateRandomString();
        System.out.println("Current size of Heap::"+Runtime.getRuntime().totalMemory());
        System.out.println("Max size of Heap::"+Runtime.getRuntime().maxMemory());
        Connection con = null;
        Statement stmt = null;
        ResultSet rs = null;
        PreparedStatement pst = null;
        FileWriter fw = new FileWriter(jsonFilePath);
        try {
            con = getMySQLDataSource().getConnection();
            stmt = con.createStatement();
            System.out.println("Displaying key and value pair of HashMap..");
            Iterator mapIterator = mapSet.iterator();
            while(mapIterator.hasNext())
            {
                Map.Entry mapEntry = (Map.Entry)mapIterator.next();
                //int key = Integer.parseInt(mapEntry.getKey().toString());
                //String value = mapEntry.getValue().toString();
                pst = con.prepareStatement("insert into nodes_test values (?,?)");
                pst.setInt(1, Integer.parseInt(mapEntry.getKey().toString()));
                pst.setString(2, mapEntry.getValue().toString());
                pst.executeUpdate();
                System.out.println(Integer.parseInt(mapEntry.getKey().toString()));
            }
            fw.close();
        } catch (SQLException e) {
            e.printStackTrace();
        }finally{
            try {
                if(rs != null) rs.close();
                if(stmt != null) stmt.close();
                if(pst != null) pst.close();
                if(con != null) con.close();
                System.out.println("Free Memory::"+Runtime.getRuntime().freeMemory());
            } catch (SQLException e) {
                e.printStackTrace();
            }
        }

    }

}

You can do a batch update for every 500 insert and also introduce sleep after every batch so that JVM can collect garbage value. Also increase your JVM Heap space. — Sridhar DD
– Sridhar DD, Commented Jan 21, 2015 at 6:36
You should use batch execute in java so that you can do bulk insert. Here is the example my.vertica.com/docs/4.1/HTML/Master/14878.htm . You can add each insert with addBatch() and after each 500 entry you should do executeBatch() so that it will insert into db as bulk insert. After every executeBatch() you should provide 2 sec sleep so that JVM will release all the memory used by the loop. — Sridhar DD
– Sridhar DD, Commented Jan 21, 2015 at 6:48
How do I store 500 data at a stretch in addbatch. because each time i am iterating and adding in db. so how do i iterate first 500 datas and then add in db then add next 500 data and add in db — Soumya
– Soumya, Commented Jan 21, 2015 at 7:08
Batch will store each and every prepared statement with it(Batch will not insett on each iteration it will do bulk insert on executeBatch). you keep a variable and increment it on each iteration. if it is ==500 then execute the batch. and make the variable to ZERO. — Sridhar DD
– Sridhar DD, Commented Jan 21, 2015 at 7:12

StanislavL · Accepted Answer · 2015-01-21 06:21:46Z

1

I would suggest to store the String as soon as they are generated rather than storing in a HashMap. ust to avoid memory usage if you don't ned the data in your app after saving.

You can introduce a Connection parameter in the generateRandomString() method and pass it from the main() method. As soon as the random string is created you call store based on the connection. Also I would try to reuse prepared statement.

answered Jan 21, 2015 at 6:21

StanislavL

57.5k9 gold badges75 silver badges100 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

gknicker · Accepted Answer · 2015-01-21 07:24:20Z

Stop wasting memory. Just write the generated values directly to the database.

Here is your code without unnecessary memory usage. I made no other attempts to optimize the code.

public class SequenceGenerator
{
    private static final String CHAR_LIST =
        "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ1234567890";
    private static final int RANDOM_STRING_LENGTH = 5;
    private static final Random RANDOM = new Random();

    /**
     * Generates and returns a random string
     */
    public String generateRandomString() {
        char[] chars = new char[RANDOM_STRING_LENGTH];
        for (int i = 0; i < RANDOM_STRING_LENGTH; i++) {
            int number = RANDOM.nextInt(CHAR_LIST.length());
            chars[i] = CHAR_LIST.charAt(number);
        }
        return new String(chars);
    }

    public static DataSource getMySQLDataSource() throws Exception {
        Properties props = new Properties();
        MysqlDataSource mysqlDS = null;

        FileInputStream fis = new FileInputStream(
                "D:/Assignments/Sequence/db.properties");
        try {
            props.load(fis);
        } finally {
            fis.close();
        }
        mysqlDS = new MysqlDataSource();
        mysqlDS.setURL(props.getProperty("MYSQL_DB_URL"));
        mysqlDS.setUser(props.getProperty("MYSQL_DB_USERNAME"));
        mysqlDS.setPassword(props.getProperty("MYSQL_DB_PASSWORD"));
        return mysqlDS;
    }

    public static void main(String[] args) throws Exception {
        SequenceGenerator sg = new SequenceGenerator();
        Connection con = getMySQLDataSource().getConnection();
        try {
            PreparedStatement pst = con.prepareStatement("insert into nodes_test values (?,?)");
            for (int i = 0; i < 50000; i++) {
                pst.setInt(1, i);
                pst.setString(2, sg.generateRandomString());
                pst.executeUpdate();
                System.out.println(i);
            }
            pst.close();
        } finally {
            con.close();
        }
    }
}

Thanks it worked fine.. Can u please help me in optimizing this code.

Vishwajit R. Shinde · Accepted Answer · 2015-01-21 07:15:29Z

0

First find out your heap memory using following code

public static void main(String[]args){

    long totalHeapSize = Runtime.getRuntime().totalMemory();

    //Total heap size
    System.out.println("Total Heap Size is: " + totalHeapSize);
}

Now change your heap size if using below environment variable

SET _JAVA_OPTIONS = -Xms512m -Xmx1024m

OR
use below command

java -Xms512m -Xmx1024m TestData.java

answered Jan 21, 2015 at 7:15

Vishwajit R. Shinde

4752 gold badges5 silver badges18 bronze badges

Collectives™ on Stack Overflow

Java heap space::Out of memory error

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related