How to remove duplicate xml-nodes using xslt?

Question

I want to remove duplicates when all variables are exact matches using xslt.

In this xml node 3 should be removed because it is a perfect copy of node 1.

<root> 
    <trips> 
      <trip> 
        <got_car>0</got_car> 
        <from>Stockholm, Sweden</from> 
        <to>Gothenburg, Sweden</to> 
        <when_iso>2010-12-06 00:00</when_iso> 
      </trip>
      <trip> 
        <got_car>0</got_car> 
        <from>Stockholm, Sweden</from> 
        <to>New york, USA</to> 
        <when_iso>2010-12-06 00:00</when_iso> 
      </trip>
      <trip> 
        <got_car>0</got_car> 
        <from>Stockholm, Sweden</from> 
        <to>Gothenburg, Sweden</to> 
        <when_iso>2010-12-06 00:00</when_iso> 
      </trip>
      <trip> 
        <got_car>1</got_car> 
        <from>Test, Duncan, NM 85534, USA</from> 
        <to>Test, Duncan, NM 85534, USA</to> 
        <when_iso>2010-12-06 00:00</when_iso> 
      </trip> 
    <trips> 
<root>

possible duplicate of How to remove duplicate XML nodes using XSLT — Robert Harvey
– Robert Harvey, Commented Dec 5, 2010 at 22:33

user357812 · Accepted Answer · 2010-12-06 14:02:21Z

With a better desing, this stylesheet:

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
    <xsl:key name="kTripByContent" match="trip"
             use="concat(got_car,'+',from,'+',to,'+',when_iso)"/>
    <xsl:template match="node()|@*">
        <xsl:copy>
            <xsl:apply-templates select="node()|@*"/>
        </xsl:copy>
    </xsl:template>
    <xsl:template match="trip[generate-id() !=
                              generate-id(key('kTripByContent',
                                              concat(got_car,'+',
                                                     from,'+',
                                                     to,'+',
                                                     when_iso))[1])]"/>
</xsl:stylesheet>

Output:

<root>
    <trips>
        <trip>
            <got_car>0</got_car>
            <from>Stockholm, Sweden</from>
            <to>Gothenburg, Sweden</to>
            <when_iso>2010-12-06 00:00</when_iso>
        </trip>
        <trip>
            <got_car>0</got_car>
            <from>Stockholm, Sweden</from>
            <to>New york, USA</to>
            <when_iso>2010-12-06 00:00</when_iso>
        </trip>
        <trip>
            <got_car>1</got_car>
            <from>Test, Duncan, NM 85534, USA</from>
            <to>Test, Duncan, NM 85534, USA</to>
            <when_iso>2010-12-06 00:00</when_iso>
        </trip>
    </trips>
</root>

Flack · Accepted Answer · 2010-12-05 23:25:50Z

1

This code:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<xsl:output method="xml" indent="yes"/>

<xsl:key name="trip-tth" match="/root/trips/trip" use="concat(got_car, '+', from, '+', to, '+', when_iso)"/>

<xsl:template match="root/trips">   
    <xsl:copy>
        <xsl:apply-templates select="trip[generate-id(.) = generate-id( key ('trip-tth', concat(got_car, '+', from, '+', to, '+', when_iso) ) )]"/>
    </xsl:copy>
</xsl:template>

<xsl:template match="trip">
    <xsl:copy-of select="."/>
</xsl:template>

</xsl:stylesheet>

Will do the trick.

It utilizes the fact that generate-id() applied to a key will take the id of the first node, that matches a given criteria. And in our case criteria is concatenated value of each trip child element.

edited Dec 5, 2010 at 23:25

answered Dec 5, 2010 at 22:32

Flack

5,8572 gold badges26 silver badges27 bronze badges

6 Comments

Dimitre Novatchev Over a year ago

@Flack: Generally your answer isn't entirely correct. Whenever using concatenation as key, it is safe and necessary to include in this concatenation separating strings, in order to avoid considering concat('A','BC') and concat('AB','C') to be the "same".

Himmators Over a year ago

Actually, could you explain how I put the new data in a for-each-array?

Flack Over a year ago

@Dimitre Novatchev, thanks for useful remark. I've edited my response for future readers.

Flack Over a year ago

@Kristoffer Nolgren, if I got your point right, you want to use for-each instead of apply-templates? Well, xpath expression will be the same.

user357812 Over a year ago

@Flack: This answer has value because of the key concatenation (once it follows @Dimitre remarks...), but the rest of the stylesheet has many faults: absolute path for key/@match pattern, loosing the root element, push style template applying for later explicit xsl:copy...

|

Community · Accepted Answer · 2017-05-23 12:18:39Z

1

If you are using XSLT 1.0, this answer may help: How to remove duplicate XML nodes using XSLT. It is easier with XSLT 2.0 but that is not universally deployed

edited May 23, 2017 at 12:18

CommunityBot

11 silver badge

answered Dec 5, 2010 at 22:14

peter.murray.rust

38.2k46 gold badges161 silver badges226 bronze badges

Collectives™ on Stack Overflow

How to remove duplicate xml-nodes using xslt?

3 Answers 3

Comments

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related