Optimized way of checking for duplication in database using hibernate

Question

I am currently working on a program that is using Spring and Hibernate to retrieve data every 31 seconds (Spring scheduled service) and save to mysql tables (Hibernate).

I currently have a HashMap that is checking the data retrieved. If the data retrieved is in the HashMap, use that instance. If the data is not in the HashMap, add it to the HashMap and save this new entry through Hibernate.

My only concern with this design is that since this is a Spring schedule service that runs every 31 seconds, the HashMap object also get recreated every 31 seconds In that case, any entries previously added to the HashMap will not be on there anymore, and duplicated entries will be saved to the table.

I have came up with two ways to solve the issue:

Query for all the current entries on the table and use them to populate the HashMap, before the HashMap is used to check the new data retrieved. This way, the HashMap even after being recreated every 31 seconds, will still have all the unique entries.
Query the new data that just came in against all the entries on the table. This way, a HashMap is not needed. Whenever a new data comes in, just check it against all the entries through query.

Which of the two ways is more optimized as the number of entries on the table gets larger and larger? Please also feel free to suggest any alternative ways of doing this. Thanks.

My code:

@Scheduled(fixedRate = 31000)
public void saveCompanyData() {               

            // Retrieve data from online into Data
            ...
            ...

            // Hibernate
            SessionFactory factory = new Configuration()
            .configure("hibernate.cfg.xml")
            .addAnnotatedClass(Company.class)
            .buildSessionFactory();
            Session session = factory.getCurrentSession();
            session.beginTransaction();

            // Create HashMap
            HashMap<String, Company> companyMap = new HashMap<>();

            // Save to table if key not mapped
            String companyKey = Data.getCompanyName();
            Company company;
            if(companyMap.containsKey(companyKey)){
                company = companyMap.get(companyKey);
            } else {
                company = new company(Data.getCompanyName());
                companyMap.put(companyKey, company);
                session.save(company);
            }

            // Commit
            session.getTransaction().commit();

Second one sounds fair.

Antoniossss
– Antoniossss

2017-01-25 21:10:02 +00:00
Commented Jan 25, 2017 at 21:10 — Antoniossss
– Antoniossss, Commented Jan 25, 2017 at 21:10

Antoniossss · Accepted Answer · 2017-01-26 01:51:21Z

1

Second one sounds fair. Also given code does not make any sense as companyMap will always be empty so contains will always be false.

Also as you are refering to optimalization here, you should not create entity factory everytime you want to use it as this is expensive operation. It should be rather created only once, most probably at application startul, unless you got some fancy flow that allows hiberate.cfg to be altered on runtime.

edited Jan 26, 2017 at 1:51

answered Jan 25, 2017 at 21:12

Antoniossss

32.7k7 gold badges67 silver badges119 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Optimized way of checking for duplication in database using hibernate

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related