1

I have thousands of *.db files containing sensor data. All *.db files have the same structure like, ID,MACID,RSSI,ENTRY,EXIT,DEVICEID,UNIQUEID. I have seen a similar question like this one but the solution does not work for me.

What I have tried so far:

  1. Initially tried DB Browser for SQLite to individually convert each *.db to .csv format. It's a lot of copy-paste and I hated it!

  2. Using online forums, created a makeshift script as follows;

import sqlite3
from sqlite3 import Error

import pandas as pd
dataPath = "path_to_a_single_db_file"
# create database connection
conn = sqlite3.connect(dataPath)
# a custom function to create database connection
# Reference: http://www.sqlitetutorial.net/sqlite-python/sqlite-python-select/
def create_connection(db_file):
              try:
                   conn = sqlite3.connect(db_file)
                    return conn
              except Error as e:
                    print(e)

              return None
# create a function for reading & writing the Wifi table

def read_write_wifi_tbl_to_csv(conn):
            cur = conn.cursor()
            query='select * from WIFI'
            data=pd.read_sql(query,conn)
            data.to_csv('export.csv')

def main():
    # create a database connection
    conn = create_connection(dataPath)
    with conn:
      print("1. Query and writing to Wifi table")
      read_write_wifi_tbl_to_csv(conn)

if __name__ == '__main__':
                main()

The above code snippet works well for a single db file. How to make this work for multiple db files.

3
  • I dont understand why the code is not formatted properly. I'm not new to SO and in past i've edited many posts on code formatting, but today, I can't seem to properly format the code for my own post!!!! What an irony!. Commented Jan 17, 2019 at 7:48
  • I think I've fixed the formatting - you can use triple-back-quotes now to bracket code, so you don't have to put extra spaces in each line. Please check I've not messed anything critical up. Commented Jan 17, 2019 at 8:22
  • @Spacedman thanks a lot for your help in this regard. Also on the advice on using triple-back-quotes, greatly appreciate it. Commented Jan 17, 2019 at 8:25

1 Answer 1

1

If you use python,

Use sqlite and csv packages to convert each sqlite .db file to csv file.

Then use pandas to read all csv files and write to single csv file using pandas.

EDIT: well, pandas is not required you can directly write to single csv file.

EDIT 2: Here is a quick notebook for that: https://github.com/darshanz/CombineMultipleSqliteToCsv

Sign up to request clarification or add additional context in comments.

6 Comments

I'm not quite good with python. I'm more well-versed with R. What i see from online forums 1, 2 is that I need to establish a database connection first, like, db = sqlite3.connect('data/mydb'). There is no database. All i have are individual *.db files. So how to solve this issue?
you can iterate over the files in a directory in R perhaps using lapply.
Please check my notebook in the my last EDIT
thanks for the inputs and your time. But it still does not answer my question. I've edited the question now.
Well, is the problem using multiple files. you can iterate through all the files using os package. I have added new python script you can use it directly. I tested with 2 files in the input folder.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.