How to remove duplicates of given value in list of JSON objects?

Question

I have the following data I am parsing:

master = [
 {'Title': 'Jordan MA2 "Future Beginnings"', 'Price': 150, 'Currency': 'USD', 'Picture': 'https://static.nike.com/a/images/t_default/a960ad50-4a05-4066-9916-e334e68f1dfd/jordan-ma2-future-beginnings-shoes-bK3TsG.png', 'Link': 'nike.com/t/jordan-ma2-future-beginnings-shoes-bK3TsG/DA2552-100', 'Brand': 'nike'},
 {'Title': 'Jordan MA2 "Future Beginnings"', 'Price': 150, 'Currency': 'USD', 'Picture': 'https://static.nike.com/a/images/t_default/a960ad50-4a05-4066-9916-e334e68f1dfd/jordan-ma2-future-beginnings-shoes-bK3TsG.png', 'Link': 'nike.com/t/jordan-ma2-future-beginnings-shoes-bK3TsG/DA2552-100', 'Brand': 'jordan'},
 {'Title': 'lace-up leather boots', 'Price': 1904, 'Currency': ' USD', 'Picture': 'https://cdn-images.farfetch-contents.com/16/49/73/72/16497372_32445982_480.jpg', 'Link': 'farfetch.com/shopping/men/rick-owens-lace-up-leather-boots-item-16497372.aspx?storeid=11893', 'Brand': 'yeezy'},
 {'Title': 'Air Jordan XXXV Low', 'Price': 175, 'Currency': 'USD', 'Picture': 'https://static.nike.com/a/images/t_default/a4778a7d-3678-4f7f-a289-773b1e9faf01/air-jordan-xxxv-low-basketball-shoes-6q5Z2t.png', 'Link': 'nike.com/t/air-jordan-xxxv-low-basketball-shoes-6q5Z2t/DJ9805-190', 'Brand': 'nike'},
 {'Title': 'Air Jordan XXXV Low', 'Price': 175, 'Currency': 'USD', 'Picture': 'https://static.nike.com/a/images/t_default/a4778a7d-3678-4f7f-a289-773b1e9faf01/air-jordan-xxxv-low-basketball-shoes-6q5Z2t.png', 'Link': 'nike.com/t/air-jordan-xxxv-low-basketball-shoes-6q5Z2t/DJ9805-190', 'Brand': 'jordan'},
 {'Title': 'The Lug slip-on boots', 'Price': 1250, 'Currency': ' USD', 'Picture': 'https://cdn-images.farfetch-contents.com/17/08/52/70/17085270_34331632_480.jpg', 'Link': 'farfetch.com/shopping/men/bottega-veneta-the-lug-slip-on-boots-item-17085270.aspx?storeid=9671', 'Brand': 'yeezy'},
 {'Title': 'Jordan Series .01', 'Price': 80, 'Currency': 'USD', 'Picture': 'https://static.nike.com/a/images/t_default/2527995d-01cd-43ea-b66d-0f1248035bb3/jordan-series-1-shoes-TxmFhR.png', 'Link': 'nike.com/t/jordan-series-1-shoes-TxmFhR/CV8129-100', 'Brand': 'nike'}
]

The first two json objects in the list have the same picture link, but different brands. How do I remove one of the two with the same picture link and same title?

I have tried this, but it doesn't seem to mutate the data.

master = { each['Picture'] : each for each in master }.values()

Regards

Andrej Kesely · Accepted Answer · 2021-08-01 19:19:21Z

You can use set to check for duplicities. For example:

out, seen = [], set()
for d in master:
    if (d["Title"], d["Picture"]) not in seen:
        seen.add((d["Title"], d["Picture"]))
        out.append(d)

print(out)

Prints:

[
    {
        "Title": 'Jordan MA2 "Future Beginnings"',
        "Price": 150,
        "Currency": "USD",
        "Picture": "https://static.nike.com/a/images/t_default/a960ad50-4a05-4066-9916-e334e68f1dfd/jordan-ma2-future-beginnings-shoes-bK3TsG.png",
        "Link": "nike.com/t/jordan-ma2-future-beginnings-shoes-bK3TsG/DA2552-100",
        "Brand": "nike",
    },
    {
        "Title": "lace-up leather boots",
        "Price": 1904,
        "Currency": " USD",
        "Picture": "https://cdn-images.farfetch-contents.com/16/49/73/72/16497372_32445982_480.jpg",
        "Link": "farfetch.com/shopping/men/rick-owens-lace-up-leather-boots-item-16497372.aspx?storeid=11893",
        "Brand": "yeezy",
    },
    {
        "Title": "Air Jordan XXXV Low",
        "Price": 175,
        "Currency": "USD",
        "Picture": "https://static.nike.com/a/images/t_default/a4778a7d-3678-4f7f-a289-773b1e9faf01/air-jordan-xxxv-low-basketball-shoes-6q5Z2t.png",
        "Link": "nike.com/t/air-jordan-xxxv-low-basketball-shoes-6q5Z2t/DJ9805-190",
        "Brand": "nike",
    },
    {
        "Title": "The Lug slip-on boots",
        "Price": 1250,
        "Currency": " USD",
        "Picture": "https://cdn-images.farfetch-contents.com/17/08/52/70/17085270_34331632_480.jpg",
        "Link": "farfetch.com/shopping/men/bottega-veneta-the-lug-slip-on-boots-item-17085270.aspx?storeid=9671",
        "Brand": "yeezy",
    },
    {
        "Title": "Jordan Series .01",
        "Price": 80,
        "Currency": "USD",
        "Picture": "https://static.nike.com/a/images/t_default/2527995d-01cd-43ea-b66d-0f1248035bb3/jordan-series-1-shoes-TxmFhR.png",
        "Link": "nike.com/t/jordan-series-1-shoes-TxmFhR/CV8129-100",
        "Brand": "nike",
    },
]

user8060120 · Accepted Answer · 2021-08-01 19:25:32Z

1

you very closely to the right solution just make the composite key x['Title']+x['Picture'] + x['Link'], then take values:

list({x['Title']+x['Picture'] + x['Link']:x for x in master}.values())

answered Aug 1, 2021 at 19:25

user8060120

Collectives™ on Stack Overflow

How to remove duplicates of given value in list of JSON objects?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related