AWS DynamoDB multiple "NE" string filters?

Question

I'm trying to scan/query an AWS DynamoDB table for a list of items where id (a single string) is not equal to any of strings A, B, C, D, etc.

I've tried something like this:

for (String itemString : items) {
  scanExpression.addFilterCondition("id", 
      new Condition().withComparisonOperator(ComparisonOperator.NE)
      .withAttributeValueList(new AttributeValue().withS(itemString)));
}

PaginatedScanList<Items> result = mapper.scan(Item.class, scanExpression);

What appears to happen is that each time I add filter, it overwrites the previous values, so that the scan only checks against one itemString, not several. Am I missing something, or is this a limitation of DynamoDB?

Note: if it matters, in my example, I listed four values (A, B, C, D) but in production, this may be a list of hundreds of values.

mkobit · Accepted Answer · 2015-07-14 13:04:01Z

You are correct in saying that the way you are doing it "overwrites the previous values". Here is the relevant code from the 1.9.23 SDK on DynamoDBScanExpression:

public void addFilterCondition(String attributeName, Condition condition) {
    if ( scanFilter == null )
        scanFilter = new HashMap<String, Condition>();

    scanFilter.put(attributeName, condition);
}

Since your attributeName will be id for both puts, the last value will win in this case.

In my opinion, this is poor behavior from DynamoDBScanExpression, and I would even lean more towards saying it is a bug that should be reported. The documentation does not state anything about when a duplicate attributeName is added and the method name makes it seem like this is unexpected behavior.

I don't see a good way to work around this other than building out the entire filter expression.

Another Note: On the documentation, I don't see a length constraint for how long a filter expression can be as well as how many ExpressionAttributeNames and ExpressionAttributeValues are allowed on a request. That may come into account if you are trying to filter out a ton of attribute values, but I haven't found any documentation of what that limit might be or what behavior you should expect.

StringJoiner filterExpression = new StringJoiner(" AND ");
Map<String, AttributeValue> expressionAttributeValues = new HashMap<>();
int itemAttributeSuffix = 0;
for (String itemString : items) {
    StringBuilder expression = new StringBuilder("#idname")
            .append(" ")
            .append("<>")
            .append(" ")
            .append(":idval")
            .append(itemAttributeSuffix);
    filterExpression.add(expression);
    expressionAttributeValues.put(":idval" + itemAttributeSuffix++,
        new AttributeValue().withS(itemString));
}
Map<String, String> expressionAttributeNames = Collections.singletonMap("#idname", "id");
scanExpression.setFilterExpression(filterExpression.toString());
scanExpression.setExpressionAttributeNames(expressionAttributeNames);
scanExpression.setExpressionAttributeValues(expressionAttributeValues);

PaginatedScanList<Items> result = mapper.scan(Item.class, scanExpression);

This approach worked perfectly, at least on a small-scale test, but it will be interesting to see how it works at a larger scale. Thanks for the thoughtful answer!

Collectives™ on Stack Overflow

AWS DynamoDB multiple "NE" string filters?

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related