I have a .csv file which I would like to convert into a .jsonl file.
I found the Pandas to_json method:
df = pd.read_csv('DIRECTORY/texts1.csv', sep=';')
df.to_json ('DIRECTORY/texts1.json')
However, I am not aware of a function to turn it into a .jsonl format. How can I do this?
This is probably a bit late, but I wrote a silly module called csv-jsonl that may help with this sort of thing.
>>> from csv_jsonl import JSONLinesDictWriter
>>> l = [{"foo": "bar", "bat": 1}, {"foo": "bar", "bat": 2}]
>>> with open("foo.jsonl", "w", encoding="utf-8") as _fh:
... writer = JSONLinesDictWriter(_fh)
... writer.writerows(l)
...
It extends the native csv module, so it's mostly familiar. Hope it helps.
I'm not sure if this result is compliant with "jsonl" syntax, but it's a hack that might get towards a relevant outcome.
The primary trick is to treat each line of the input file as a separate JSON file upon export, then read that JSON back in from disk and treat as distinct jsonl lines.
I'm starting from a CSV that contains
hello, from, this, file
another, amazing, line, csv
last, line, of, file
The snippet below builds on another post.
import pandas
df = pandas.read_csv("myfile.csv", header=None)
file_to_write = ""
for index in df.index:
df.loc[index].to_json("row{}.json".format(index))
with open("row{}.json".format(index)) as file_handle:
file_content = file_handle.read()
file_to_write += file_content + "\n"
with open("result.jsonl","w") as file_handle:
file_handle.write(file_to_write)
The resulting .jsonl file contains
{"0":"hello","1":" from","2":" this","3":" file"}
{"0":"another","1":" amazing","2":" line","3":" csv"}
{"0":"last","1":" line","2":" of","3":" file"}
If the row indices are not desired, those could be removed from the .to_json() line of the Python snippet above.
Thought this should be added. This version builds upon Ben's answer but avoids using temporary files and optimizes string handling, thus addressing the potential inefficiencies and issues in the original script.
import pandas as pd
# Reading the CSV file into a DataFrame without headers
df = pd.read_csv("myfile.csv", header=None)
# Prepare an empty list to collect JSON strings
json_lines = []
# Convert each row to a JSON string and append it to the list
for index, row in df.iterrows():
json_str = row.to_json(orient='records')
json_lines.append(json_str)
# Join all JSON strings with newline characters and write to a single file
with open("result.jsonl", "w") as file_handle:
file_handle.write("\n".join(json_lines))
a lot of attempts to hijack a common practice. Just append the JSON strings at the end of the file you want. That;s the whole point. You only need to read to the next newline to read a JSON document instead of reading the entire file.ndjson.orgappeared beforejsonlines.organd contained the same text as the historicaljson.orgsite, without having any relation to either Douglas Crockford or ECMAto_jsoncan write each row in a separate row if you useorient='records', lines=True. From to_json docs:If ‘orient’ is ‘records’ write out line delimited json format. Will throw ValueError if incorrect ‘orient’ since others are not list like.