Clojure: parse json and extract values

Question

I'm making an API call and using Cheshire to parse the JSON:

(defn fetch_headlines [source]
  (let [articlesUrl (str "https://newsapi.org/v2/top-headlines?sources=" 
                         source
                         "&apiKey=a688e6494c444902b1fc9cb93c61d6987")]
    (-> articlesUrl
        client/get
        generate-string 
        parse-string)))

The JSON payload:

{"status" 200, "headers" {"access-control-allow-headers" "x-api-key, authorization", "content-type" "application/json; charset=utf-8", "access-control-allow-origin" "*", "content-length" "7434", "connection" "close", "pragma" "no-cache", "expires" "-1", "access-control-allow-methods" "GET", "date" "Thu, 28 Mar 2019 20:22:16 GMT", "x-cached-result" "false", "cache-control" "no-cache"}, "body" "{\"status\":\"ok\",\"totalResults\":10,\"articles\":[{\"source\":{\"id\":\"cnn\",\"name\":\"CNN\"},\"author\":null,\"title\":\"Trump: Mueller probe was 'attempted takeover' of government - CNN Video\",\"description\":\"In a Fox News interview with Sean Hannity, President Trump called special counsel Robert Mueller's probe an \\"attempted takeover of our government.\\"\",\"url\":\"http://us.cnn.com/videos/politics/2019/03/28/trump-mueller-probe-attempted-takeover-hannity-cpt-sot-vpx.cnn\",\"urlToImage\":\"https://cdn.cnn.com/cnnnext/dam/assets/190324191527-06-trump-mueller-reaction-0324-super-tease.jpg\",\"publishedAt\":\"2019-03-28T20:09:04.1891948Z\",\"content\":\"Chat with us in Facebook Messenger. Find out what's happening in the world as it unfolds.\"},{\"source\":{\"id\":\"cnn\",\"name\":\"CNN\"},\"author\":null,\"title\":\"James Clapper reacts to call he should be investigated - CNN Video\",\"description\":\"Former Director of National Intelligence James Clapper reacts to White House press secretary Sarah Sanders saying he and other former intelligence officials should be investigated after special counsel Robert Mueller did not establish collusion between the Tr…\",\"url\":\"http://us.cnn.com/videos/politics/2019/03/26/james-clapper-reponse-mueller-report-sarah-sanders-criticism-bts-ac360-vpx.cnn\",\"urlToImage\":\"https://cdn.cnn.com/cnnnext/dam/assets/190325211210-james-clapper-ac360-03252019-super-tease.jpg\",\"publishedAt\":\"2019-03-28T20:08:43.1736236Z\",\"content\":\"Chat with us in Facebook Messenger. Find out what's happening in the world as it unfolds.\"},{\"source\":{\"id\":\"cnn\",\"name\":\"CNN\"},\"author\":\"Maegan Vazquez, CNN\",\"title\":\"Trump set for first rally since Mueller investigation ended\",\"description\":\"President Donald Trump, making his first appearance before supporters since Robert Mueller ended his investigation, is set to speak during a rally in Grand Rapids, Michigan Thursday night.\",\"url\":\"http://us.cnn.com/2019/03/28/politics/donald-trump-grand-rapids-rally/index.html\",\"urlToImage\":\"https://cdn.cnn.com/cnnnext/dam/assets/190321115403-07-donald-trump-lead-image-super-tease.jpg\",\"publishedAt\":\"2019-03-28T19:49:26Z\",\"content\":\"Washington (CNN)President Donald Trump, making his first appearance before supporters since Robert Mueller ended his investigation, is set to speak during a rally in Grand Rapids, Michigan Thursday night.\r\nThe rally follows a chaotic week in Washington, preci… [+2099 chars]\"},{\"source\":{\"id\":\"cnn\",\"name\":\"CNN\"},\"author\":\"Katelyn Polantz, CNN\",\"title\":\"Judge orders Justice Dept. to turn over Comey memos\",\"description\":\"A federal judge has ordered that the James Comey memos are turned over, in a court case brought by CNN and other media organizations for access to the documents memorializing former FBI Director's interactions with President Donald Trump.\",\"url\":\"http://us.cnn.com/2019/03/28/politics/james-comey-memo-lawsuit/index.html\",\"urlToImage\":\"https://cdn.cnn.com/cnnnext/dam/assets/181209143047-comey-1207-super-tease.jpg\",\"publishedAt\":\"2019-03-28T19:14:45Z\",\"content\":\"Washington (CNN)A federal judge has ordered that the Justice Department and FBI submit James Comey's memos in full to the court under seal, in a court case brought by CNN and other media organizations for access to the documents memorializing the former FBI d… [+1043 chars]\"},{\"source\":{\"id\":\"cnn\",\"name\":\"CNN\"},\"author\":\"Clare Foran and Manu Raju, CNN\",\"title\":\"Pelosi calls AG's summary of Mueller report 'arrogant'\",\"description\":\"House Speaker Nancy Pelosi on Thursday criticized Attorney General William Barr's summary of special counsel Robert Mueller's report, calling it \\"condescending\\" and \\"arrogant\\" and saying \\"it wasn't the right thing to do.\\"\",\"url\":\"http://us.cnn.com/2019/03/28/politics/pelosi-mueller-report-congress-barr-summary/index.html\",\"urlToImage\":\"https://cdn.cnn.com/cnnnext/dam/assets/190328130240-02-nancy-pelosi-03282019-super-tease.jpg\",\"publishedAt\":\"2019-03-28T18:48:25Z\",\"content\":null},{\"source\":{\"id\":\"cnn\",\"name\":\"CNN\"},\"author\":\"Analysis by Chris Cillizza, CNN Editor-at-large\",\"title\":\"The 43 most outrageous lines from Donald Trump's phone interview with Sean Hannity\",\"description\":\"There's no \\"reporter\\" that President Donald Trump likes more than Fox News' Sean Hannity -- largely due to Hannity's unwavering, puppy dog-like support for the President. Trump likes to reward people who play nice with him, which brings us to the 45-minute ph…\",\"url\":\"http://us.cnn.com/2019/03/28/politics/sean-hannity-donald-trump-mueller/index.html\",\"urlToImage\":\"https://cdn.cnn.com/cnnnext/dam/assets/190328140149-01-hannity-trump-file-super-tease.jpg\",\"publishedAt\":\"2019-03-28T18:44:21Z\",\"content\":\"(CNN)There's no \\"reporter\\" that President Donald Trump likes more than Fox News' Sean Hannity -- largely due to Hannity's unwavering, puppy dog-like support for the President. Trump likes to reward people who play nice with him, which brings us to the 45-minu… [+14785 chars]\"},{\"source\":{\"id\":\"cnn\",\"name\":\"CNN\"},\"author\":null,\"title\":\"Puerto Rico Gov.: I'll punch the bully in the mouth - CNN Video\",\"description\":\"In an exclusive interview with CNN, Puerto Rico Governor Ricardo Rosselló said he would not sit back and allow his officials to be bullied by the White House.\",\"url\":\"http://us.cnn.com/videos/politics/2019/03/28/ricardo-rossello-trump-bully-puerto-rico-sot-vpx.cnn\",\"urlToImage\":\"https://cdn.cnn.com/cnnnext/dam/assets/190328123504-puerto-rico-gov-ricardo-rosello-super-tease.jpg\",\"publishedAt\":\"2019-03-28T18:08:33.7312458Z\",\"content\":\"Chat with us in Facebook Messenger. Find out what's happening in the world as it unfolds.\"},{\"source\":{\"id\":\"cnn\",\"name\":\"CNN\"},\"author\":\"Jeremy Herb, Manu Raju and Ted Barrett, CNN\",\"title\":\"Jared Kushner interviewed by Senate Intelligence Committee\",\"description\":\"President Donald Trump's son-in-law Jared Kushner returned to the Senate Intelligence Committee for a closed door interview Thursday as part of the committee's Russia investigation.\",\"url\":\"http://us.cnn.com/2019/03/28/politics/jared-kushner-senate-intelligence/index.html\",\"urlToImage\":\"https://cdn.cnn.com/cnnnext/dam/assets/180302124221-30-jared-kushner-super-tease.jpg\",\"publishedAt\":\"2019-03-28T16:21:29Z\",\"content\":null},{\"source\":{\"id\":\"cnn\",\"name\":\"CNN\"},\"author\":\"Jeremy Herb and Laura Jarrett, CNN\",\"title\":\"Mueller report more than 300 pages, sources say\",\"description\":\"Special counsel Robert Mueller's confidential report on the Russia investigation is more than 300 pages, according to a Justice Department official and a second source with knowledge of the matter.\",\"url\":\"http://us.cnn.com/2019/03/28/politics/mueller-report-pages/index.html\",\"urlToImage\":\"https://cdn.cnn.com/cnnnext/dam/assets/190324130054-05-russia-investigation-0324-super-tease.jpg\",\"publishedAt\":\"2019-03-28T15:52:01Z\",\"content\":null},{\"source\":{\"id\":\"cnn\",\"name\":\"CNN\"},\"author\":\"Jim Acosta and Kevin Liptak, CNN\",\"title\":\"Exclusive: Puerto Rico governor warns White House over funding\",\"description\":\"Tensions are escalating between President Donald Trump and Puerto Rico's governor over disaster relief efforts that have been slow in coming for the still-battered island after Hurricane Maria.\",\"url\":\"http://us.cnn.com/2019/03/28/politics/ricardo-rossell-donald-trump-puerto-rico-funding/index.html\",\"urlToImage\":\"https://cdn.cnn.com/cnnnext/dam/assets/180920230539-pr-storm-of-controversy-rossello-trump-super-tease.jpg\",\"publishedAt\":\"2019-03-28T15:19:39Z\",\"content\":null}]}", "trace-redirects" ["https://newsapi.org/v2/top-headlines?sources=cnn&apiKey=a688e6494c444902b1fc9cb93c61d687"]}

I'd like to extract to extract the URLs from the returned JSON payload, I've tried this:

(defn fetch_headlines [source]
  (let [articlesUrl (str "https://newsapi.org/v2/top-headlines?sources="
                         source
                         "&apiKey=a688e6494c444902b1fc9cb93c61d697")]
    (-> articlesUrl
        client/get
        generate-string 
        parse-string
        (get-in ["source" "url"]))))

But I get a nil result, any ideas?

SOLUTION based on user feedback:

(defn fetch-headlines [source]
  (let [articlesUrl (str "https://newsapi.org/v2/top-headlines?sources=" 
                         source
                         "&apiKey=a688e6494c444902b1fc9cb93c61d697")]
    (-> articlesUrl
        client/get
        :body
        parse-string
        (get-in ["articles" 0 "url"]))))

stop calling generate-string before parse-string? generate string just creates JSON from a clojure object, so you’re wrapping it and unwrapping it with the generate/parse pair. — pete23
– pete23, Commented Mar 29, 2019 at 6:56
best way of debugging this stuff is to build yhr expression up one clause at a time at the REPL, and male sure you understand the return value at each stage. — pete23
– pete23, Commented Mar 29, 2019 at 6:57
@pete23 calling "parse-string" directly on the JSON payload without "generate-string" returns an error: classCastException clojure.lang.PersistentArrayMap cannot be cast to java.base/java.lang.String cheshire.core/parse-string (core.clj:81) — jwesonga
– jwesonga, Commented Mar 29, 2019 at 16:49
look at what you have in your hand by executing this in the REPL. i suspect you’ll discover you now need to take the body as per the answer — pete23
– pete23, Commented Mar 29, 2019 at 18:04

cfrick · Accepted Answer · 2019-03-29 06:59:41Z

1

What you need is inside the body key, but the value corresponding to that key is still a string and not yet a clojure map. When you look for source, you're getting nil back because that key doesn't exist (it should be inside body, after correctly parsing the string into json).

Once you've properly parsed the body value, it should be something like:

(let [index-of-article 0]
  (get-in response ["body" "articles" index-of-article "url"]))

where index-of-article is the positional index of the article you want, since articles contains a vector of articles.

edited Mar 29, 2019 at 6:59

cfrick

37.3k7 gold badges61 silver badges76 bronze badges

answered Mar 28, 2019 at 21:09

Solaxun

2,7922 gold badges28 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

jwesonga Over a year ago

i tried this: (defn fetch-headlines [source] (let [articlesUrl (str "newsapi.org/v2/top-headlines?sources=" source "&apiKey=a688e6494c444902b1fc9cb93c61d697")] (-> articlesUrl client/get generate-string parse-string (get "body") (get-in ["articles" [0] "url"]) )))

cfrick Over a year ago

@jwesonga it's just 0 - not [0]. That was used to indicate a "placeholder".

Solaxun Over a year ago

@cfrick Thanks for fixing that - it was confusing in retrospect, but yes, I meant just to show that you need a positional index there.

Collectives™ on Stack Overflow

Clojure: parse json and extract values

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related