Elasticsearch - PHP Bulk

Question

I am trying to bulk - import data into elasticsearch via cURL from PHP.

For start, I would like to add that I copied the import data format generated by the PHP and pasted it into Sense and the bulk import works just fine. But by sending the same data via cURL to the same link, with the same method that I used in Sense, I am receiving the following error message:

{"_index":"product","_type":"pid","_id":"_bulk","found":false}

OR, if I do not specify the _index and _type via link and I specify it via the json I send, I get the following error

{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"No endpoint or operation is available at [_bulk]"}],"type":"illegal_argument_exception","reason":"No endpoint or operation is available at [_bulk]"},"status":400}

The way I am creating the cURL request is the following

protected $curl_opts = array(
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_CUSTOMREQUEST => "GET",
    CURLOPT_TIMEOUT => 10
);

......

public function sendcURL() {
    $this->curl->create('http://localhost:9200/_bulk';

    foreach($this->curl_opts as $option => $option_value)
        $this->curl->option($option, $option_value);

    $this->curl->http_header('Content-Type', 'application/json');
    $this->curl->option(CURLOPT_BINARYTRANSFER, true);
    $this->curl->option(CURLOPT_HEADER, true);
    $this->curl->post($json_data);
    $this->execute();
}

Consider that the $json_data is correctly formatted, as well, consider that I am using the correct link / method.

As well, I know of the elasticsearch-php github repo (even searched how they do bulk in there, and it is similar to my method), but I would prefer writing my own methods & libraries for the moment as what I need at the moment won't require a complete elastic-php library.

What am I doing wrong?

Gustavo Díaz · Accepted Answer · 2017-10-18 19:17:57Z

1

Have you considered that as is written in the ES docs, When sending requests to the _bulk endpoint the Content-Type header should be set to application/x-ndjson

ES bulk Docs

Other thing that could be is that you are using CURLOPT_CUSTOMREQUEST => "GET" but the real request to ES is a POST.

According to DOCS it could cause libcurl to send invalid requests and it could possibly confuse the remote server badly

CURL_OPT DOCS

if one of this approach works, let me know it, and i will edit the answer to just the correct one

edited Oct 18, 2017 at 19:17

answered Oct 18, 2017 at 19:06

Gustavo Díaz

517 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

AndreyP · Accepted Answer · 2020-04-07 13:45:58Z

0

CURLOPT_CUSTOMREQUEST => 'GET' can be used with php curl:

$requests = '';
foreach ($queries as $query) {
    list ($data, $headers) = $query;
    $requests .= json_encode($headers) . "\n";
    $requests .= json_encode($data) . "\n";
}

$ch = $this->_ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, $requests);
curl_setopt($ch, CURLOPT_HTTPHEADER, ['Content-Type: application/x-ndjson']);
curl_setopt($ch, CURLOPT_CUSTOMREQUEST, 'GET');
$response = @curl_exec($ch);

answered Apr 7, 2020 at 13:45

AndreyP

2,7071 gold badge32 silver badges19 bronze badges

Collectives™ on Stack Overflow

Elasticsearch - PHP Bulk

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related