Java parsing txt file multithreading

Question

I have a method to parse txt file and saving datas to database. Method named: parseFileAndSaveToDB(String fileAddress); This file are big (100 000 rows) so inserting datas into DB are very long.

Can anyone help me with do this in multithreading?? I think that each thread can use this method with each files forexample:

parseFileAndSaveToDB("c:/file1");//thread 1
parseFileAndSaveToDB("c:/file2");//thread 2
parseFileAndSaveToDB("c:/file3");//thread 3
parseFileAndSaveToDB("c:/file4");//thread 4

But how to do it with multithreads?? And is it good idea?

We can help you to improve or to rectify your code/logic. We are not here to write code for you. — Ankur Shanbhag
– Ankur Shanbhag, Commented Feb 25, 2014 at 11:23
Are they writing in the same table? If yes, multithreading has no use at all. — Smutje
– Smutje, Commented Feb 25, 2014 at 11:23
It may have some use if their bottleneck was disk access to the files; by multithreading, they can overcome that, and then the database access would be the new bottleneck — Jorge_B
– Jorge_B, Commented Feb 25, 2014 at 11:26
@Smutje this depends on the table itself (of course) and on the RDBMS engine... Some engines do not require table-level locks on insertions :p — fge
– fge, Commented Feb 25, 2014 at 11:41

Ameer Tamboli · Accepted Answer · 2014-02-25 11:32:05Z

2

Please take a look at this link here which explains how to use ExecutorService in java. It can be something like:

ExecutorService executorService = Executors.newFixedThreadPool(10);

MyDataLoader myDataLoader1 = new MyDataLoader("c:/file1");
MyDataLoader myDataLoader2 = new MyDataLoader("c:/file2");
MyDataLoader myDataLoader3 = new MyDataLoader("c:/file3");

executorService.execute(myDataLoader1);
executorService.execute(myDataLoader2);
executorService.execute(myDataLoader3);

executorService.shutdown();

You can write MyDataLoader class like:

public class MyDataLoader implements Runnable {

    String fileName = null;

    public MyDataLoader(String fileName) {
        this.fileName = fileName;
    }

    public void run() {
        //Your logic to parse file and insert the data in DB.
    }
}

Hope this helps.

answered Feb 25, 2014 at 11:32

Ameer Tamboli

1,31812 silver badges21 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

RMachnik · Accepted Answer · 2014-02-25 19:47:02Z

1

This is quick draft. As the argument of program pass list of file names that should be read and store.

public class ParsingThread implements Runnable{
private String fileName;

public ParsingThread(String fName){
    fileName = fName;
}

@Override
public void run(){
    parseFileAndSaveToDB(fileName);

}
public void parseFileAndSaveToDB(String fileName){
    //your implementation
}



public static void main(String args[]){
    for(String fileName : args){
        Thread runAndStore = new Thread(new ParsingThread(fileName));
        runAndStore.run();
    }

  }
}

edited Feb 25, 2014 at 19:47

answered Feb 25, 2014 at 11:27

RMachnik

3,7121 gold badge36 silver badges52 bronze badges

2 Comments

hetmanJIIIS Over a year ago

He don't allow me to use in main method and in for loop this Thread, IDE talking that i have to use ParsingThread but if I use this, then i have to create method start() in ParsingThread method

RMachnik Over a year ago

Now it should work I edited my code sory for mistakes.

Saro Taşciyan · Accepted Answer · 2014-02-25 11:30:21Z

Very basic template:

public static void parseFileAndSaveToDB(String filePath)
{
    new Thread(new Runnable() {

        @Override
        public void run() {
            // # TODO : Open file with path "filePath"
            // # Read file
            // # Write to database
        }
    }).start();
}

public static void main(String args[]) {
    List<String> paths; // # TODO : fill this list with the file paths you have

    for(String filePath : paths) 
    {
        parseFileAndSaveToDB(filePath);
    }
}

But before that, i strongly recommend you read some articals on the consecpt of multithreading in java. You might find the following sources helpful:

Official Tutorial on Concurrency to begin with.
tutorialpoint.com with simple concept and demonstration samples
Vogella to dig deeper

Community · Accepted Answer · 2017-05-23 11:50:24Z

-1

Since I can't add comments yet I'll add an answer instead. I agree with Rafik991's answer and in the implementation you can start by counting the number of lines you have in order to distribute your data equally between your threads. check this link for fast implementations. i'm not sure if that's a good idea since you will have go through the file to get the right line but I think that it's what you are trying to do.

edited May 23, 2017 at 11:50

CommunityBot

11 silver badge

answered Feb 25, 2014 at 11:43

A.B

1145 bronze badges

Collectives™ on Stack Overflow

Java parsing txt file multithreading

4 Answers 4

Comments

2 Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related