Efficiently getting XML into Elasticsearch

Question

Currently I am using scrapy to parse a large XML file from an ftp server into elasticsearch. It works but seems quite a heavy weight solution and it uses a lot of memory too.

I am wondering if I am better off writing a plugin for ES instead. I know logstash can do it but I can't do inline language detection etc with that.

A) if I write an actual plugin for ES I think it has to be in Java to pull in the data. Is there any advantage in this approach or could I write a separate Python script to push the data in instead. Is there any clear reason for selecting one method over the other (assuming I don't know Java or Python)

This comes down to:

Would the memory management be better with an actual ES plugin
Is Java better suited to processing XML than say, Python?

Slam · Accepted Answer · 2015-04-03 20:00:34Z

2

Converting XML to JSON is rather question about understanding actual data in XML, as it can be not so easy to transform to JSON and usually needs additional logic. For this reason, there's no error-proof XML>JSON translators.

If you'll decide to use python to do that, take a look at eTree, lxml and xmltodict. JSON support is in python's stdlib natively.

If you'll decide to try some luck from ES side, look at elasticsearch-xml. It may fit your needs in case of consistent XML.

Talking about python vs java performance for parsing - if performance is a key for you, you can leverage some libraries, that is already optimized at low-level, but generally, good java code should perform better.

edited Apr 3, 2015 at 20:00

answered Apr 3, 2015 at 15:55

Slam

8,6621 gold badge40 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Efficiently getting XML into Elasticsearch

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related