Curl callback not returning the same number of bytes

Question

I'm getting an inconsistent failure when trying to grab http data, using curl.

libcurl: (23) Failed writing body (23048858 != 16058)

It was continuously failing, then it started working. Now it's failing again. I've hunted around and most of the posts either blame the callback or the order of setting curl options. I've rearranged the options, and I don't see any problems with the callback. In fact when I debug the callback the input parameters match the output value that I am actually getting. So I'm not sure where this value of 23048858 comes from.

Source

SwReceiver* SwReceiver::instance;

SwReceiver *SwReceiver::getInstance(
  std::string &address,
  std::string &key,
  std::string &options)
{
  if(!instance)
  {
    instance = new SwReceiver(address, key, options);
    return instance;
  }
}

SwReceiver::SwReceiver(
  std::string &address,
  std::string &key,
  std::string &options)
{
  curl_global_init(CURL_GLOBAL_ALL);
  data = curl_easy_init();
  if(data)
  {
    std::string addrstr = address + '/' + key + '/' + options;
    // std::cout << "Address = " << addrstr << std::endl;
    curl_easy_setopt(data, CURLOPT_URL, addrstr.c_str());
    curl_easy_setopt(data, CURLOPT_WRITEDATA, (void *)&chunk);
    curl_easy_setopt(data, CURLOPT_USERAGENT, "libcurl-agent/1.0");
    curl_easy_setopt(data, CURLOPT_ERRORBUFFER, errorBuffer);
    curl_easy_setopt(data, CURLOPT_WRITEFUNCTION, writeMemoryCallback);
    memset(errorBuffer, 0, sizeof(CURL_ERROR_SIZE));
    /* DEBUG */
    curl_easy_setopt(data, CURLOPT_VERBOSE, 1);

    runQuery();
  }
  else
  {
    //TODO add to error buffer
  }
}

SwReceiver::~SwReceiver()
{
  delete(instance);
  curl_easy_cleanup(data);
}

void SwReceiver::runQuery()
{
  result = curl_easy_perform(data);
  if(result != CURLE_OK)
  {
    size_t len = strlen(errorBuffer);
    fprintf(stderr, "\nlibcurl: (%d) ", result);
    if(len)
    {
      fprintf(stderr, "%s%s", errorBuffer,
        ((errorBuffer[len - 1] != '\n') ? "\n" : ""));
    }
    else
    {
      fprintf(stderr, "%s\n", curl_easy_strerror(result));
    }
  }
}

MemoryStruct *SwReceiver::getData()
{
  return &chunk;
}

size_t SwReceiver::writeMemoryCallback(
  void *contents, size_t size, size_t nmemb, void *userp)
{
  size_t realSize = size * nmemb;
  struct MemoryStruct *mem = (struct MemoryStruct *)userp;

  char *ptr = (char*)realloc(mem->memory, mem->size + realSize + 1);
  if(ptr == NULL)
  {
    //TODO This should write to the error buffer
    std::cout << "Not enought memory!\n";
    return 0;
  }

  mem->memory = ptr;
  memcpy(&(mem->memory[mem->size]), contents, realSize);
  mem->size += realSize;
  mem->memory[mem->size] = 0;
}

the memory struct source

 struct MemoryStruct
{
  char * memory;
  size_t size;

  MemoryStruct()
  : memory((char*)malloc(1)),
  size(0)
  {

  }
};

I didn't post the header for size reasons. If you feel I should post let me know.

DEBUG

(gdb)
(gdb) break 79
Breakpoint 1 at 0x40240c: file swreceiver.cpp, line 79.
(gdb) run
Starting program: /home/ubuntu/src/stickweather/stickweather
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
[New Thread 0x7ffff1de9700 (LWP 26989)]
[Thread 0x7ffff1de9700 (LWP 26989) exited]
*   Trying 34.193.12.42...
* Connected to api.darksky.net (34.193.12.42) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
*        subject: CN=darksky.net
*        start date: Apr 26 00:00:00 2019 GMT
*        expire date: May 26 12:00:00 2020 GMT
*        subjectAltName: api.darksky.net matched
*        issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*        SSL certificate verify ok.
> GET /forecast/<hidden>/37.8267,-122.4233 HTTP/1.1
Host: api.darksky.net
User-Agent: libcurl-agent/1.0
Accept: */*

< HTTP/1.1 200 OK
< Date: Tue, 22 Oct 2019 14:18:40 GMT
< Content-Type: application/json; charset=utf-8
< Content-Length: 28035
< Connection: keep-alive
< X-Authentication-Time: 687ms
< X-Forecast-API-Calls: 13
< Cache-Control: max-age=60
< Expires: Tue, 22 Oct 2019 14:19:39 +0000
< X-Response-Time: 333.671ms
< Vary: Accept-Encoding
<

Thread 1 "stickweather" hit Breakpoint 1, SwReceiver::writeMemoryCallback (contents=0x650be6, size=1, nmemb=16058,
    userp=0x635848) at swreceiver.cpp:82
82        size_t realSize = size * nmemb;

delete(instance); in the destructor is not a good idea, as that would double-destroy the only instance. — molbdnilo
– molbdnilo, Commented Oct 22, 2019 at 14:43
Take a few minutes and think about what value your callback returns. (Enable more warnings in your compiler.) — molbdnilo
– molbdnilo, Commented Oct 22, 2019 at 14:47
the callback was copied from curl's webpage. they didn't use the return value. Do you think that could have an effect? Which compiler warning are you referring to? — mreff555
– mreff555, Commented Oct 22, 2019 at 14:53
If you're referring to this sample, your code looks very much like it except that the sample does return a value. If you're referring to something else, I'm sure the curl project would appreciate a bug report. — molbdnilo
– molbdnilo, Commented Oct 22, 2019 at 15:03

Pete Kirkham · Accepted Answer · 2019-10-22 14:50:55Z

2

Your writeMemoryCallback does not return the number of bytes written, at the bottom control reaches the end of the function without a return, so on x86 it will probably return garbage. I'd suspect that is where the other value is coming from.

https://curl.haxx.se/libcurl/c/CURLOPT_WRITEFUNCTION.html

answered Oct 22, 2019 at 14:50

Pete Kirkham

49.5k5 gold badges96 silver badges176 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

mreff555 Over a year ago

Right that makes sense. I didn't realize the callback was actually using the return value. I double checked curl's website and it was correct. I must have transcribed it incorrectly. Thanks.

Collectives™ on Stack Overflow

Curl callback not returning the same number of bytes

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related