I have a huge number of text files, data in each file differs from one file to another.
I would like to loop into the list of files and create sqlite database which includes one table corresponding to each file.
Data is provided in the text file in the format of "column name":"value"|"column name":"value"| ....etc.
Here are the challenges faced:
- Managed to pass the table name as parameter in the CREATE table SQL query
- How to pass the columns as a variable taking into consideration that the data differs from one file to another? I cannot explicitly define column names as in "INSERT INTO TBL_NAME VALUES (Id INTEGER, Parameter TEXT)" ...etc
Example for the text files:
WBTS.txt:
WBTS|version:xxxxxxx|DistName:"xxxx-xxxx/xxx-xxx/WBTS-xxx|Id:2011659|name:VT_AO_U_xxxx_xxxxxxxxxx|BTSIPAddress:0.0.0.0|BTSRACHCapaIncCapability:2|BTSSupportForHSPACM:2|BlindHOIntraBTSQCheck:0|BroadcastSIB15:0|BroadcastSIB15_2:0|BroadcastSIB15_3:0|DCNLinkStatus:0|DCNSecurityStatus:255|DLORLAveragingWindowSize:0|DSCPHigh:46|DSCPLow:0|DSCPMedDCH:34|DSCPMedHSPA:26|DediMeasRepPeriodCSdata:3|DediMeasRepPeriodPSdata:1|DedicatedMeasReportPeriod:5|DelayThresholdMax:10000|DelayThresholdMax2msTTI:10000|DelayThresholdMid:7000|DelayThresholdMid2msTTI:7000|DelayThresholdMin:5000|DelayThresholdMin2msTTI:5000|EnhOLPCRTWPEnabled:0|FreqChangeCapability:2|HARQRVConfiguration:1|HSDPA14MbpsPerUser:1|HSDPACCEnabled:1|HSDPAULCToDSCP:34|HSUPACCEnabled:1|HSUPADLCToDSCP:34|HSUPAXUsersEnabled:60|IPBasedRouteIdIub:254|IPBasedRouteIdIub2:0|IPBasedRouteIdIub3:0|IPBasedRouteIdIub4:0|IPNBId:104|InactCACThresholdATM:170|InactCACThresholdIP:80|InactUsersCIDThreshold:6|IntelligentSDPrioHO:0|IubTransportSharing:0|LoadControlPeriodPS:20|MDCBufferingTime:50|MaxBTSOMFrameSize:1460|MaxFPDLFrameSizeIub:1428|MaxNumberEDCHLCG:0|MeasFiltCoeff:5|MinUDPPortIub:1026|NEType:24|NodeBRABReconfigSupport:0|OverbookingSwitch:0|PDUSize656WithHSDSCH:2|PSAveragingWindowSize:4|PSRLAveragingWindowSize:0|PWSMAVTrafficVERLogic:0|PWSMEnableWakeUpTime:0|PWSMInUse:0|PWSMRemCellSDBeginHour:22|PWSMRemCellSDBeginMin:0|PWSMRemCellSDEndHour:6|PWSMRemCellSDEndMin:0|PWSMShutdownBeginHour:22|PWSMShutdownBeginMin:0|PWSMShutdownEndHour:6|PWSMShutdownEndMin:0|PWSMWeekday:0|ProbabilityFactorMax:1000|ProbabilityFactorMax2msTTI:1000|PrxAlpha:0|PrxMeasAveWindow:20|PrxTargetPSAdjustPeriod:5|PtxAlpha:0|PtxDPCHmax:-30|PtxDPCHmin:-28|PtxMeasAveWindow:10|RACHloadIndicationPeriod:1|RFSharingState:0|RRIndPeriod:40|RRMULDCHActivityFactorCSAMR:50|RRMULDCHActivityFactorCSNTData:95|RRMULDCHActivityFactorCSTData:100|RRMULDCHActivityFactorPSBackgr:50|RRMULDCHActivityFactorPSStream:50|RRMULDCHActivityFactorPSTHP1:50|RRMULDCHActivityFactorPSTHP2:50|RRMULDCHActivityFactorPSTHP3:50|RRMULDCHActivityFactorSRB:1|SBTSId:xxxxx|SatelliteIubUsage:0|SchedulingPeriod:100|TQMId:0|TQMId2:0|TQMId3:0|TQMId4:0|ToAWEOffsetNRTDCHIP:0|ToAWEOffsetRTDCHIP:0|ToAWSOffsetNRTDCHIP:0|ToAWSOffsetRTDCHIP:0|WBTSChangeOrigin:2|WBTSName:VT_AO_U_xxxx_xxxxxxxxxx|WBTSSWBuildId:No information available|WinACRABsetupDL:5|WinACRABsetupUL:5|WinLCHSDPA:5|WinLCHSUPA:5|
WBTS|version:xxxxxxx|DistName:"xxxx-xxxx/xxx-xxx/WBTS-xxx|Id:2110802|name:ET_AO_U_xxxx_xxxxxxxxxx|BTSIPAddress:0.0.0.0|BTSRACHCapaIncCapability:2|BTSSupportForHSPACM:2|BlindHOIntraBTSQCheck:0|BroadcastSIB15:0|BroadcastSIB15_2:0|BroadcastSIB15_3:0|DCNLinkStatus:0|DCNSecurityStatus:255|DLORLAveragingWindowSize:0|DSCPHigh:46|DSCPLow:0|DSCPMedDCH:34|DSCPMedHSPA:26|DediMeasRepPeriodCSdata:3|DediMeasRepPeriodPSdata:1|DedicatedMeasReportPeriod:5|DelayThresholdMax:10000|DelayThresholdMax2msTTI:10000|DelayThresholdMid:7000|DelayThresholdMid2msTTI:7000|DelayThresholdMin:5000|DelayThresholdMin2msTTI:5000|EnhOLPCRTWPEnabled:0|FreqChangeCapability:2|HARQRVConfiguration:1|HSDPA14MbpsPerUser:1|HSDPACCEnabled:1|HSDPAULCToDSCP:34|HSUPACCEnabled:1|HSUPADLCToDSCP:34|HSUPAXUsersEnabled:60|IPBasedRouteIdIub:272|IPBasedRouteIdIub2:0|IPBasedRouteIdIub3:0|IPBasedRouteIdIub4:0|IPNBId:110|InactCACThresholdATM:170|InactCACThresholdIP:80|InactUsersCIDThreshold:6|IntelligentSDPrioHO:0|IubTransportSharing:0|LoadControlPeriodPS:20|MDCBufferingTime:50|MaxBTSOMFrameSize:1460|MaxFPDLFrameSizeIub:1428|MaxNumberEDCHLCG:0|MeasFiltCoeff:5|MinUDPPortIub:1026|NEType:24|NodeBRABReconfigSupport:0|OverbookingSwitch:0|PDUSize656WithHSDSCH:2|PSAveragingWindowSize:4|PSRLAveragingWindowSize:0|PWSMAVTrafficVERLogic:0|PWSMEnableWakeUpTime:0|PWSMInUse:0|PWSMRemCellSDBeginHour:22|PWSMRemCellSDBeginMin:0|PWSMRemCellSDEndHour:6|PWSMRemCellSDEndMin:0|PWSMShutdownBeginHour:22|PWSMShutdownBeginMin:0|PWSMShutdownEndHour:6|PWSMShutdownEndMin:0|PWSMWeekday:0|ProbabilityFactorMax:1000|ProbabilityFactorMax2msTTI:1000|PrxAlpha:0|PrxMeasAveWindow:20|PrxTargetPSAdjustPeriod:5|PtxAlpha:0|PtxDPCHmax:-30|PtxDPCHmin:-28|PtxMeasAveWindow:10|RACHloadIndicationPeriod:1|RFSharingState:0|RRIndPeriod:40|RRMULDCHActivityFactorCSAMR:50|RRMULDCHActivityFactorCSNTData:95|RRMULDCHActivityFactorCSTData:100|RRMULDCHActivityFactorPSBackgr:50|RRMULDCHActivityFactorPSStream:50|RRMULDCHActivityFactorPSTHP1:50|RRMULDCHActivityFactorPSTHP2:50|RRMULDCHActivityFactorPSTHP3:50|RRMULDCHActivityFactorSRB:1|SBTSId:xxxxx|SatelliteIubUsage:0|SchedulingPeriod:100|TQMId:0|TQMId2:0|TQMId3:0|TQMId4:0|ToAWEOffsetNRTDCHIP:0|ToAWEOffsetRTDCHIP:0|ToAWSOffsetNRTDCHIP:0|ToAWSOffsetRTDCHIP:0|WBTSChangeOrigin:2|WBTSName:ET_AO_U_xxxx_xxxxxxxxxx|WBTSSWBuildId:No information available|WinACRABsetupDL:5|WinACRABsetupUL:5|WinLCHSDPA:5|WinLCHSUPA:5|
WBTS|version:xxxxxxx|DistName:"xxxx-xxxx/xxx-xxx/WBTS-xxxx|Id:2770911|name:EU_AO_U_xxxx_xxxxxxxxxx|BTSIPAddress:0.0.0.0|BTSRACHCapaIncCapability:2|BTSSupportForHSPACM:2|BlindHOIntraBTSQCheck:0|BroadcastSIB15:0|BroadcastSIB15_2:0|BroadcastSIB15_3:0|DCNLinkStatus:0|DCNSecurityStatus:255|DLORLAveragingWindowSize:0|DSCPHigh:46|DSCPLow:0|DSCPMedDCH:34|DSCPMedHSPA:26|DediMeasRepPeriodCSdata:3|DediMeasRepPeriodPSdata:1|DedicatedMeasReportPeriod:5|DelayThresholdMax:10000|DelayThresholdMax2msTTI:10000|DelayThresholdMid:7000|DelayThresholdMid2msTTI:7000|DelayThresholdMin:5000|DelayThresholdMin2msTTI:5000|EnhOLPCRTWPEnabled:0|FreqChangeCapability:2|HARQRVConfiguration:1|HSDPA14MbpsPerUser:1|HSDPACCEnabled:1|HSDPAULCToDSCP:34|HSUPACCEnabled:1|HSUPADLCToDSCP:34|HSUPAXUsersEnabled:60|IPBasedRouteIdIub:341|IPBasedRouteIdIub2:0|IPBasedRouteIdIub3:0|IPBasedRouteIdIub4:0|IPNBId:1117|InactCACThresholdATM:170|InactCACThresholdIP:80|InactUsersCIDThreshold:6|IntelligentSDPrioHO:0|IubTransportSharing:0|LoadControlPeriodPS:20|MDCBufferingTime:50|MaxBTSOMFrameSize:1460|MaxFPDLFrameSizeIub:1428|MaxNumberEDCHLCG:0|MeasFiltCoeff:5|MinUDPPortIub:1026|NEType:24|NodeBRABReconfigSupport:0|OverbookingSwitch:0|PDUSize656WithHSDSCH:2|PSAveragingWindowSize:4|PSRLAveragingWindowSize:0|PWSMAVTrafficVERLogic:0|PWSMEnableWakeUpTime:0|PWSMInUse:0|PWSMRemCellSDBeginHour:22|PWSMRemCellSDBeginMin:0|PWSMRemCellSDEndHour:6|PWSMRemCellSDEndMin:0|PWSMShutdownBeginHour:22|PWSMShutdownBeginMin:0|PWSMShutdownEndHour:6|PWSMShutdownEndMin:0|PWSMWeekday:0|ProbabilityFactorMax:1000|ProbabilityFactorMax2msTTI:1000|PrxAlpha:0|PrxMeasAveWindow:20|PrxTargetPSAdjustPeriod:5|PtxAlpha:0|PtxDPCHmax:-30|PtxDPCHmin:-28|PtxMeasAveWindow:10|RACHloadIndicationPeriod:1|RFSharingState:0|RRIndPeriod:40|RRMULDCHActivityFactorCSAMR:50|RRMULDCHActivityFactorCSNTData:95|RRMULDCHActivityFactorCSTData:100|RRMULDCHActivityFactorPSBackgr:50|RRMULDCHActivityFactorPSStream:50|RRMULDCHActivityFactorPSTHP1:50|RRMULDCHActivityFactorPSTHP2:50|RRMULDCHActivityFactorPSTHP3:50|RRMULDCHActivityFactorSRB:1|SBTSId:xxxxx|SatelliteIubUsage:0|SchedulingPeriod:100|TQMId:0|TQMId2:0|TQMId3:0|TQMId4:0|ToAWEOffsetNRTDCHIP:0|ToAWEOffsetRTDCHIP:0|ToAWSOffsetNRTDCHIP:0|ToAWSOffsetRTDCHIP:0|WBTSChangeOrigin:2|WBTSName:EU_AO_U_xxxx_xxxxxxxxxx|WBTSSWBuildId:No information available|WinACRABsetupDL:5|WinACRABsetupUL:5|WinLCHSDPA:5|WinLCHSUPA:5|
RA.txt:
RA|version:xxxxx|DistName:"xxxx-xxxx/xxx-404490/RA-xxxxxxx|Id:2633805|routingAreaIdLac:xxxxx|routingAreaIdMcc:xxx|routingAreaIdMnc:xx|routingAreaIdRac:x|
RA|version:xxxxx|DistName:"xxxx-xxxx/xxx-404500/RA-xxxxxxx|Id:3015559|routingAreaIdLac:xxxxx|routingAreaIdMcc:xxx|routingAreaIdMnc:xx|routingAreaIdRac:x|
Sample code for one text file:
import sqlite3
tableName = "WBTS"
fileName = "WBTS.txt"
conn = sqlite3.connect("topo.db")
cur = conn.cursor()
#dropQuery = 'DROP TABLE IF EXISTS ' + tableName
dropQuery = 'DROP TABLE IF EXISTS {}' .format(tableName)
cur.execute(dropQuery)
print('table dropped successfully')
#createQuery = 'CREATE TABLE IF NOT EXISTS {} ({})' .format(tableName, )
#cur.execute(createQuery)
#print('table created successfully')
with open(fileName, 'r'):
for line in fileName:
data = line.split('|')
createQuery = 'CREATE TABLE IF NOT EXISTS {} ({})'.format(tableName, data)
cur.execute(createQuery)
print('table created successfully')
insertQuery = 'INSERT INTO {} VALUES {}'.format(tableName, data)
cur.execute(insertQuery)
print('data inserted successfully')
conn.commit()
conn.close()
I checked several posts related to sqlite in python but couldn't find the concept to be followed in such case.
Appreciate your support
datain'CREATE TABLE IF NOT EXISTS {} ({})'.format(tableName, data)looks like?