Unable to Load data from CSV to Elasticsearch using Logstash

Question

I'm using Logstash 2.3.3 to load data from csv file to Elasticsearch 2.3.3.

Here is my Logstash configuration file:

input {
  #stdin {
   # type => "stdin-type"
  #}

  file {
    path => "/Users/hieupm/export.txt"
    start_position => "beginning"
    codec => plain {
      charset => "CP1252"
    }
  }
}

filter {
  mutate {
    gsub => [
      "message", "\"", " ",
      "message", " +", " ",
      "message", " ;", ";",
      "message", "; ", ";"
    ]
    lowercase => ["message"]
  }
  csv {
    columns => ["siren","raison_sociale","enseigne","cp_ville","naf","nature_juridique","code_tranche"] 
    separator => ","
  }
}

output {
  #stdout {
  #}

  elasticsearch {
    hosts => "127.0.0.1:9200"
    index => "ereputation"
    document_id => "%{siren}"
  }
}

The execution log (in terminal) didn't give any error but it just displayed as below and never finished

Settings: Default pipeline workers: 8
Pipeline main started

My sample data:

siren,raison_sociale,enseigne,cp_ville,naf,nature_juridique,code_tranche 
5420021,ETABLISSEMENTS BIQUEZ,,80100 ABBEVILLE,4669B,5710,11
5420120,STE DES SUCRERIES DU MARQUENTERRE,,62140 MARCONNELLE,1081Z,5599,11
5480082,STE BRETONNE DE TRAVAUX PUBLICS,,44600 ST NAZAIRE,4312A,5599,21
5480546,SOCIETE DE DISTRIBUTION DE PAPIER,,44350 GUERANDE,1712Z,5710,21
5520176,CARTONNAGE HERNAS,,80210 FEUQUIERES EN VIMEU,1721A,5710,21
5520242,PROSPA,,80510 LONGPRE LES CORPS SAINTS,2030Z,5710,12
5520325,ETABLISSEMENTS ADRIEN RIQUIER,,80570 DARGNIES,4674B,5710,12
5540380,SNC COMTE ET CIE,PHARMACIE COMTE,04000 DIGNE LES BAINS,4773Z,5202,11
5541552,ETABLISSEMENTS BONNET FRERES,,04370 BEAUVEZER,5610C,5499,03

Thanks for any suggestions!

It's easier to debug this without ES in the mix by using the commented out stdout output. This is most likely a parsing issue. — pickypg
– pickypg, Commented Jun 29, 2016 at 18:46
I tried with your filter and output (input via stdin) and the data was successfully parsed and stored in ES. So I don't understand what is your problem. — baudsp
– baudsp, Commented Jun 30, 2016 at 7:51
Thanks for your responses, I've tried with MAC but it's not working? It's running but never finished! — minh-hieu.pham
– minh-hieu.pham, Commented Jun 30, 2016 at 8:31

MosheZada · Accepted Answer · 2016-10-07 14:07:50Z

0

I'm the author of elasticsearch_loader
If you just want to load CSV data into elasticsearch you can use elasticsearch_loader
I wrote ESL for this exact problem

You can download it with pip:

pip install elasticsearch-loader

And then you will be able to load csv/json/parquet files into elasticsearch by issuing:

elasticsearch_loader --index ereputation --type incident csv input1.csv input2.csv

answered Oct 7, 2016 at 14:07

MosheZada

2,4291 gold badge17 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

minh-hieu.pham Over a year ago

Thanks so much, your module works well for my case :)

Collectives™ on Stack Overflow

Unable to Load data from CSV to Elasticsearch using Logstash

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related