Put multiple items into DynamoDB by Java code

Question

I would like use batchWriteItem method of SDK Amazon to put a lot of items into table. I retrive the items from Kinesis, ad it has a lot of shard. I used this method for one item:

public static void addSingleRecord(Item thingRecord) {


    // Add an item
    try
    {

        DynamoDB dynamo = new DynamoDB(dynamoDB); 
        Table table = dynamo.getTable(dataTable);
        table.putItem(thingRecord);

    } catch (AmazonServiceException ase) {
        System.out.println("addThingsData request  "
                + "to AWS was rejected with an error response for some reason.");
        System.out.println("Error Message:    " + ase.getMessage());
        System.out.println("HTTP Status Code: " + ase.getStatusCode());
        System.out.println("AWS Error Code:   " + ase.getErrorCode());
        System.out.println("Error Type:       " + ase.getErrorType());
        System.out.println("Request ID:       " + ase.getRequestId());
    } catch (AmazonClientException ace) {
        System.out.println("addThingsData - Caught an AmazonClientException, which means the client encountered "
                + "a serious internal problem while trying to communicate with AWS, "
                + "such as not being able to access the network.");
        System.out.println("Error Message: " + ace.getMessage());
    }
}

public static void addThings(String thingDatum) {
    Item itemJ2;
    itemJ2 = Item.fromJSON(thingDatum);
    addSingleRecord(itemJ2);

}

The item is passed from:

private void processSingleRecord(Record record) {
    // TODO Add your own record processing logic here


    String data = null;
    try {


        // For this app, we interpret the payload as UTF-8 chars.
        data = decoder.decode(record.getData()).toString();
        System.out.println("**processSingleRecord - data " + data);
        AmazonDynamoDBSample.addThings(data);

    } catch (NumberFormatException e) {
        LOG.info("Record does not match sample record format. Ignoring record with data; " + data);
    } catch (CharacterCodingException e) {
        LOG.error("Malformed data: " + data, e);
    }
}

Now if i want to put a lot of record, I will use:

    public static void writeMultipleItemsBatchWrite(Item thingRecord) {
    try {                    


            dataTableWriteItems.addItemToPut(thingRecord);
        System.out.println("Making the request.");
        BatchWriteItemOutcome outcome = dynamo.batchWriteItem(dataTableWriteItems);
        do {

                // Check for unprocessed keys which could happen if you exceed provisioned throughput

                Map<String, List<WriteRequest>> unprocessedItems = outcome.getUnprocessedItems();

                if (outcome.getUnprocessedItems().size() == 0) {
                    System.out.println("No unprocessed items found");
                } else {
                    System.out.println("Retrieving the unprocessed items");
                    outcome = dynamo.batchWriteItemUnprocessed(unprocessedItems);
                }

            } while (outcome.getUnprocessedItems().size() > 0);

    }  catch (Exception e) {
        System.err.println("Failed to retrieve items: ");
        e.printStackTrace(System.err);
    }  

}

but how can I send the last group? because I send only when I have 25 items, but at the end the number is lower.

In stream there is no last batch... You can also add a criterion of max_wait_time = 10 minutes, and put events into DynamoDB even if you have fewer than 25. — Guy
– Guy, Commented Jun 2, 2015 at 21:54

Alexander Patrikalakis · Accepted Answer · 2015-06-04 09:56:53Z

1

You can write items to your DynamoDB table one at a time using the Document SDK in a Lambda function attached to your Kinesis Stream using PutItem or UpdateItem. This way, you can react to Stream Records as they appear in the Stream without worrying about whether there are any more records to process. Behind the scenes, BatchWriteItem consumes the same amount of write capacity units as the corresponding PutItem calls. A BatchWriteItem will be as latent as the PUT in the batch that takes the longest. Therefore, using BatchWriteItem, you may experience higher average latency than with parallel PutItem/UpdateItem calls.

edited Jun 4, 2015 at 9:56

answered Jun 4, 2015 at 2:39

Alexander Patrikalakis

5,2251 gold badge35 silver badges49 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

luca Over a year ago

Do PutItem and BatchWriteItem have the same performance? Because if it's so, then i use PutItem as I have already done.

Collectives™ on Stack Overflow

Put multiple items into DynamoDB by Java code

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related