Merge list of maps using Java 8 Stream API

Question

I'm trying to learn Java 8 Stream and when I try to convert some function to java8 to practice. I meet a problem.

I'm curious that how can I convert follow code to java stream format.

/*
 * input example:
 * [
    {
        "k1": { "kk1": 1, "kk2": 2},
        "k2": {"kk1": 3, "kk2": 4}
    }
    {
        "k1": { "kk1": 10, "kk2": 20},
        "k2": {"kk1": 30, "kk2": 40}
    }
  ]
 * output:
 * {
        "k1": { "kk1": 11, "kk2": 22},
        "k2": {"kk1": 33, "kk2": 44}
   }
 *
 *
 */
private static Map<String, Map<String, Long>> mergeMapsValue(List<Map<String, Map<String, Long>>> valueList) {
    Set<String> keys_1 = valueList.get(0).keySet();
    Set<String> keys_2 = valueList.get(0).entrySet().iterator().next().getValue().keySet();
    Map<String, Map<String, Long>> result = new HashMap<>();
    for (String k1: keys_1) {
        result.put(k1, new HashMap<>());
        for (String k2: keys_2) {
            long total = 0;
            for (Map<String, Map<String, Long>> mmap: valueList) {
                Map<String, Long> m = mmap.get(k1);
                if (m != null && m.get(k2) != null) {
                    total += m.get(k2);
                }
            }
            result.get(k1).put(k2, total);
        }
    }
    return result;
}

So all maps are the same - i.e. have the same keys at both levels? — Boris the Spider
– Boris the Spider, Commented Jun 28, 2016 at 9:19
This is not a code translation service. You need to show us what you have already tried so we can tell you what you are doing wrong. — explv
– explv, Commented Jun 28, 2016 at 9:32
You should rethink your original approach first, i.e. what to iterate over in the outer loop and what in the inner loop. — Holger
– Holger, Commented Jun 28, 2016 at 9:46
@Holger I know I should think first, but I totally can't get a way to finish it. — yunfan
– yunfan, Commented Jun 28, 2016 at 10:06

Tunaki · Accepted Answer · 2016-06-28 09:55:27Z

7

The trick here is to collect correctly the inner maps. The workflow would be:

Flat map the list of map List<Map<String, Map<String, Long>>> into a stream of map entries Stream<Map.Entry<String, Map<String, Long>>>.
Group by the key of each of those entry, and for the values mapped to same key, merge the two maps together.

Collecting maps by merging them would ideally warrant a flatMapping collector, which unfortunately doesn't exist in Java 8, although it will exist in Java 9 (see JDK-8071600). For Java 8, it is possible to use the one provided by the StreamEx library (and use MoreCollectors.flatMapping in the following code).

private static Map<String, Map<String, Long>> mergeMapsValue(List<Map<String, Map<String, Long>>> valueList) {
    return valueList.stream()
                    .flatMap(e -> e.entrySet().stream())
                    .collect(Collectors.groupingBy(
                        Map.Entry::getKey,
                        Collectors.flatMapping(
                            e -> e.getValue().entrySet().stream(),
                            Collectors.<Map.Entry<String,Long>,String,Long>toMap(Map.Entry::getKey, Map.Entry::getValue, Long::sum)
                        )
                    ));
}

Without using this convenient collector, we can still build our own with equivalent semantics:

private static Map<String, Map<String, Long>> mergeMapsValue2(List<Map<String, Map<String, Long>>> valueList) {
    return valueList.stream()
                    .flatMap(e -> e.entrySet().stream())
                    .collect(Collectors.groupingBy(
                        Map.Entry::getKey,
                        Collector.of(
                            HashMap::new,
                            (r, t) -> t.getValue().forEach((k, v) -> r.merge(k, v, Long::sum)),
                            (r1, r2) -> { r2.forEach((k, v) -> r1.merge(k, v, Long::sum)); return r1; }
                        )
                    ));
}

answered Jun 28, 2016 at 9:55

Tunaki

138k46 gold badges370 silver badges443 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Boris the Spider Over a year ago

I've stolen your Long::sum - I keep forgetting that exists. I think your Stream approach is better overall as it doesn't produce intermediate List results - although both yours and mine are completely illegible. I would advocate for the foreach with Java 8 Map methods approach...

Tunaki Over a year ago

@BoristheSpider Yes, using good old for loops would perhaps be generally better here.

yunfan Over a year ago

I'm sorry I don't understand your method, So I add printf in the function. It seems the lambda (r1, r2) -> { r2.forEach((k, v) -> r1.merge(k, v, Long::sum)); return r1; } don't be execute at all. Can you explain how the Collector.of works? I can't get much information about it by searching.

Boris the Spider Over a year ago

@yunfan the final argument to the Collector is the "combiner" - this will only be used if the stream is run in parallel for a sufficiently large datasets and will combine the results from different threads.

Boris the Spider · Accepted Answer · 2016-06-28 13:43:48Z

4

As a starting point, converting to use computeIfAbsent and merge gives us the following:

private static <K1, K2> Map<K1, Map<K2, Long>> mergeMapsValue(List<Map<K1, Map<K2, Long>>> valueList) {
    final Map<K1, Map<K2, Long>> result = new HashMap<>();
    for (final Map<K1, Map<K2, Long>> map : valueList) {
        for (final Map.Entry<K1, Map<K2, Long>> sub : map.entrySet()) {
            for (final Map.Entry<K2, Long> subsub : sub.getValue().entrySet()) {
                result.computeIfAbsent(sub.getKey(), k1 -> new HashMap<>())
                        .merge(subsub.getKey(), subsub.getValue(), Long::sum);
            }
        }
    }
    return result;
}

This removes much of the logic from your inner loop.

This code below is wrong, I leave it here for reference.

Converting to the Stream API is not going to make it neater, but lets give it a go.

import static java.util.stream.Collectors.collectingAndThen;
import static java.util.stream.Collectors.groupingBy;
import static java.util.stream.Collectors.mapping;
import static java.util.stream.Collectors.toList;

private static <K1, K2> Map<K1, Map<K2, Long>> mergeMapsValue(List<Map<K1, Map<K2, Long>>> valueList) {
    return valueList.stream()
            .flatMap(v -> v.entrySet().stream())
            .collect(groupingBy(Entry::getKey, collectingAndThen(mapping(Entry::getValue, toList()), l -> l.stream()
                    .reduce(new HashMap<>(), (l2, r2) -> {
                        r2.forEach((k, v) -> l2.merge(k, v, Long::sum);
                        return l2;
                    }))));
}

This is what I've managed to come up with - it's horrible. The problem is that with the foreach approach, you have a reference to each level of the iteration - this makes the logic simple. With the functional approach, you need to consider each folding operation separately.

How does it work?

We first stream() our List<Map<K1, Map<K2, Long>>>, giving a Stream<Map<K1, Map<K2, Long>>>. Next we flatMap each element, giving a Stream<Entry<K1, Map<K2, Long>>> - so we flatten the first dimension. But we cannot flatten further as we need to K1 value.

So we then use collect(groupingBy) on the K1 value giving us a Map<K1, SOMETHING> - what is something?

Well, first we use a mapping(Entry::getValue, toList()) to give us a Map<K1, List<Map<K2, Long>>>. We then use collectingAndThen to take that List<Map<K2, Long>> and reduce it. Note that this means we produce an intermediate List, which is wasteful - you could get around this by using a custom Collector.

For this we use List.stream().reduce(a, b) where a is the initial value and b is the "fold" operation. a is set to new HashMap<>() and b takes two values: either the initial value or the result of the previous application of the function and the current item in the List. So we, for each item in the List use Map.merge to combine the values.

I would say that this approach is more or less illegible - you won't be able to decipher it in a few hours time, let alone a few days.

edited Jun 28, 2016 at 13:43

answered Jun 28, 2016 at 9:55

Boris the Spider

61.4k6 gold badges111 silver badges177 bronze badges

6 Comments

Holger Over a year ago

DON’T use reduce when you are modifying the arguments of the function.

yunfan Over a year ago

@Holger . I don't get your mean. It seems his code don't modifying the arguments of the function.

Holger Over a year ago

@yunfan: the function passed to reduce modifies the the map l2 by invoking merge on it for each mapping of r2.

Boris the Spider Over a year ago

@yunfan, Holger is correct - reduce is meant for immutable items and returning a result combining the two inputs. I mutate the the LHS Map given as an input to the combiner - this breaks the contract.

yunfan Over a year ago

@BoristheSpider Thank your so much. So the best implement of your method is create new hashmap merge both l2 and r2? I'm also wonder how can I get the count of groupingby results to calculate the average results?

|

Chilloutman · Accepted Answer · 2017-07-27 16:19:57Z

I took the flatMap(e -> e.entrySet().stream()) part from Tunaki, but used a shorter variant for the collector:

Map<String, Integer> merged = maps.stream()
  .flatMap(map -> map.entrySet().stream())
  .collect(Collectors.toMap(
    Map.Entry::getKey, Map.Entry::getValue, Integer::sum));

More elaborate example:

Map<String, Integer> a = new HashMap<String, Integer>() {{
  put("a", 2);
  put("b", 5);
}};
Map<String, Integer> b = new HashMap<String, Integer>() {{
  put("a", 7);
}};
List<Map<String, Integer>> maps = Arrays.asList(a, b);

Map<String, Integer> merged = maps.stream()
  .flatMap(map -> map.entrySet().stream())
  .collect(Collectors.toMap(
    Map.Entry::getKey, Map.Entry::getValue, Integer::sum));

assert merged.get("a") == 9;
assert merged.get("b") == 5;

Collectives™ on Stack Overflow

Merge list of maps using Java 8 Stream API

3 Answers 3

4 Comments

6 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

4 Comments

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related