How to generate random records for elasticsearch

Question

How can I generate records for elasticsearch? I would like to generate at least 1 million records to test the memory size.

const now = new Date()
const startOfDay = new Date(now.getFullYear(), now.getMonth(), Math.random(now.getDate()))
const timestamp = startOfDay / 1000

const randomRecords = Array(10000000).fill(timestamp)

randomRecords.forEach((record, i) => {

    clientTest.index({
        index: 'test',
        type: 'test',
        id: '1',
        body: {
            [record]: `${record}${i}`,
        },
    }).then(function (resp) {
        logger.silly('Pushing of data completed', resp)

        return resp
    }, function (err) {
        console.trace(err.message)
    })

})

I guess your current problem is that you only end up generating one record, right? — Eirini Graonidou
– Eirini Graonidou, Commented Mar 22, 2018 at 10:22

Eirini Graonidou · Accepted Answer · 2018-03-22 10:42:16Z

1

For each record in your array, you set the id=1. This means, that for every iteration you overwrite the record with id=1, ending up saving one record.

So, you have two solutions:

use a counter that gets increased for every iteration, instead of the number 1, OR
use the bulk API, which also improves the performance of index operation. Note that you should also use an auto-increment (or at least unique) id for each record.

Please let me know, if you have further issues.

edited Mar 22, 2018 at 10:42

answered Mar 22, 2018 at 10:33

Eirini Graonidou

1,56616 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Palaniichuk Dmytro Over a year ago

Actually If wanna to use bulk API, and need to generate 1000 records, in one records ? client.bulk({ body: [ // action description { index: { _index: ${index}, _type: ${mytype}, _id: ${id} } }, { [title]: ${title} }, How to generate one request with diferents fields. and is it possible to recive actual size of data for elastickserch?

Eirini Graonidou Over a year ago

you can create a loop and generate the body of the request(the "params" part), so only construct the body, that you use as payload and in the end fire one and only bulk request.

rxjmx · Accepted Answer · 2018-03-22 11:49:13Z

0

Use should use the iterator i to increment the id field. If you use the same id in elasticsearch when indexing it will simply overwrite the field each time.

Change:

id: '1',

id: i,

This should work, but I would recommend using bulk api for this. So instead of indexing on each iteration. Therefore, make a bulk index collection before hand in and then bulk index it in one request.

Bulk API

answered Mar 22, 2018 at 11:49

rxjmx

2,1791 gold badge13 silver badges22 bronze badges

Comments

Palaniichuk Dmytro · Accepted Answer · 2018-03-22 16:41:17Z

Actually, it works for me also.

   export const pushTest = (message, type) => new Promise(async resolve => {
        try {
            let client = createClient()

            await client.index({
                index: type.toLowerCase(),
                type,
                body: message,
            },
            (error, response) => {
                logger.silly('Pushing of data completed', response)

                resolve(response)
            })
        } catch (e) {
            logger.error(e)
        }
    })

    for (let index = 0; index < 100; index++) {

        let messageTest = {
            'timestamp': {seconds: timestamp, nanos: 467737400},
            type: randomItem(alarmTypes),
            location: `Room_${Math.floor(Math.random() * 100)}`,
            zone: `Zone_${Math.floor(Math.random() * 100)}`,
            personName: 'person name',
            personRoom: `Room_${Math.floor(Math.random() * 100)}`,
            pageSize: 10,
            cancelTimestamp: {seconds: timestamp, nanos: 467737400},
            cancelPerson: 'person name',
            cancelLocation: `Room_${Math.floor(Math.random() * 100)}`,
            responseTime: {seconds: Math.floor(Math.random() * 1000000), nanos: 321549100},
        }

        pushTest(messageTest, 'Call')
    }

Collectives™ on Stack Overflow

How to generate random records for elasticsearch

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related