Synchronization script in Java

Question

I am trying to create a java class that will connect to an online service, lookup a value and then update this value in my Database. It is a relatively simple task but since it is a large set of values i have the condition of executing this task in chunks of data (lets say 100).

So this ServiceSyncer class should(?) execute the following pseudo-code:

 while (select Objects where timestamp < today, limit 100 )  returns items {
 foreach Object {
 try {
  connect to Service for Object.attribute
  extract attribute value
  If attribute value has changed {
      update Object {
        set attribute value
        }
     }
   } catch exception {
     log exception
   }
     update timestamp = now()
   }
}

This loop will update all objects in chunks of 100 until all objects have been looked up. Objects that failed during lookup will remain failed until next time the syncer will execute again.

What would be conceptually a good approach on solving this problem?

**The above pseudo code looks less obsolete when trying to implement it.

Something like that will work, but i wonder if there is a better way

JAVA

    boolean keepProcessing = true;
    int startIndex = 0;
    int endIndex = 0;

    while (keepProcessing) {
        endIndex = startIndex + 100;
        if (endIndex > publist.size() - 1) {
            endIndex = publist.size() - 1;
            keepProcessing = false;
        }

        List<ObjectJPA> currentBatchList = publist.subList(startIndex, endIndex);

        for (int i = 0; i < currentBatchList.size(); i++) {
            // do staff
        }

        repository.save(currentBatchList);
    }

Not sure what you mean by "What would be conceptually a good approach on solving this problem?", seems you have the concept down. — Taylor
– Taylor, Commented Jun 29, 2015 at 14:01
@Taylor When i try implementing it i ran across problems; especially because i choose to do it in chunks of data. It seems it cant be done in one "step" — yeaaaahhhh..hamf hamf
– yeaaaahhhh..hamf hamf, Commented Jun 29, 2015 at 14:36
The problems are probably in the details of your implementation. Your pseudo code seems fine but implementing this kind of thing can get buggered by query details, transaction boundary issues, etc... — Taylor
– Taylor, Commented Jun 29, 2015 at 15:04

Ravindra babu · Accepted Answer · 2015-06-29 14:15:19Z

1

Since I do not have enough score to provide comment, I am submitting my solution as Answer instead of Comment.

I prefer different approach to update the state of your object

OPTION A:

1) You can get the state of the object from database in Service A.

2) When there is a change to object in your online service B, that service will simply notify your Service A through a service call (http call is preferred)

3) You will replace entire while loop with a service call. If there is a change in the object from Step 1, then only update modified object in your database from Service A.

OPTION B:

1) Maintain a distributed cache between these two services. If there is a change in object, then only update database with modified value.

answered Jun 29, 2015 at 14:15

Ravindra babu

39.1k11 gold badges258 silver badges222 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

yeaaaahhhh..hamf hamf Over a year ago

This should run as a synchronization script in a certain time and hour of the day. It is not intended to be used with notifications. It will just check the status whenever i want to check it and update it accordingly :/

Ravindra babu Over a year ago

Your Pesudo logic is fine. Add ThreadPoolExecutor or Executor service for quick processing. Getting object from online service and update database can be made part of a Callable task. stackoverflow.com/questions/30782611/… has one sample regarding Callable task

Ravindra babu Over a year ago

ThreadPoolExecutor executor = new ThreadPoolExecutor(1, 10, 30, TimeUnit.SECONDS, new ArrayBlockingQueue<Runnable>(100), new RejectionHandlerImpl()) will fine-tune your needs. And one more thing- what is the time taken to update one object in Database after comparing the same by fetching it from on-line service object?

Collectives™ on Stack Overflow

Synchronization script in Java

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related