6

I have a .dat file which I don't have any idea about how it was created and what delimiter was used and any details about it. I just have its corresponding mdf and csv file. Thats all. Is there any way in python to read this .dat file?

Few ways that I have tried:

file = "736_2_PerformanceCurve_(23_0C)_(13_5V).dat"
datContent = [i.strip().split() for i in open(file, encoding='latin1').readlines()]
datContent

which gives output

[['|CF,2,1,1;|CK,1,3,1,1;'],
 ['|NO,1,7,1,0,,0,;'],
 ['|NL,1,10,1252,0x407;'],
 ['|CT,1,41,0,6,Bench#,24,Korrosionstest', '15A046-01,0,;'],
 ['|CT,1,30,0,11,StartOfTest,8,06/30/17,0,;'],
 ['|CT,1,58,0,10,ResultPath,36,c:\\korrosionstest\\daten\\#170161-OR02,0,;'],
 ['|CT,1,59,0,11,GraphicPath,36,c:\\korrosionstest\\daten\\#170161-OR02,0,;'],
 ['|CT,1,31,0,15,GraphicBaseName,5,736_2,0,;'],
 ['|CT,1,26,0,10,PartNumber,5,736_2,0,;'],
 ['|CT,1,31,0,9,VA-Nr.', 'GS,11,170161-OR02,0,;'],
 ['|CT,1,62,0,9,VA-Nr.',
  'CC,42,TO_ENV_2017_G2_C1_Platform_CC-122164-03-08,0,;'],
 ['|CT,1,24,0,6,Tester,8,Behrendt,0,;'],
 ['|CT,1,32,0,15,Test', 'Department,6,GS/ETR,0,;'],
 ['|CG,1,5,1,1,1;'],
 ['|CD,1,16,1E-2,1,1,s,0,0,0;'],
 ['|NT,1,27,30,', '6,2017,14,25,15.8050001;'],
 ['|CC,1,3,1,1;'],
 ['|CP,1,16,1,2,4,16,0,0,1,0;'],
 ['|Cb,1,33,1,0,1,1,0,11718,0,11718,1,5E-3,0,;'],
 ['|CR,1,30,1,6.103888176768602E-3,0,1,1,A;'],
 ['|CN,1,28,0,0,0,16,ai_iB1_Strom_ECU,0,;'],
 ['|CG,1,5,1,1,1;'],
 ['|CD,1,16,1E-2,1,1,s,0,0,0;'],
 ['|NT,1,27,30,', '6,2017,14,25,15.8050001;'],
 ['|CC,1,3,1,1;'],
 ['|CP,1,16,2,2,4,16,0,0,1,0;'],
 ['|Cb,1,37,1,0,2,1,11718,11718,0,11718,1,5E-3,0,;'],
 ['|CR,1,30,1,3.662332906061161E-3,0,1,1,V;'],
 ['|CN,1,31,0,0,0,19,ai_iB1_Spannung_UBB,0,;'],

The corresponding csv file for the same

[![enter image description here][1]][1]

EDIT : 

from asammdf import MDF
dat_file = r"C:\Users\HPO2KOR\Desktop\Work\data1.dat"
mdf_file = r"C:\Users\HPO2KOR\Desktop\Work\data1.mdf"
df = mdf.to_dataframe()
mdf = MDF(mdf_file)
df.head()

which gives me [![enter image description here][2]][2]

How do I read the same data from the dat file, is there any library or code for the same?

Improve this question
6
  • I am not sure what the data is behind the DAT file, but see if you can find that out ? lifewire.com/what-is-a-dat-file-2620550 . That way you may have a better idea how to open the file Commented Jan 17, 2020 at 7:20
  • Can you share sample dat file? Commented Jan 17, 2020 at 7:27
  • The ".dat" file most likely comes from iMC Famos imcdataworks.com/products/measurement-software/imc-famos . I don't think that there is a parser for those files in the open. Commented Feb 5, 2020 at 7:33
  • yes iMC famos was able to read this dat file successfully but it did not create it, actually it was created by some other system Commented Feb 5, 2020 at 7:57
  • or do you know if they have any API for the same? Commented Feb 5, 2020 at 7:59

2 Answers 2

Reset to default
4

If I look at the file it looks for me like a specific format.

One data block starts with a | and ends with a ;. In the data block the data are splitted with ,. Basically it's like a CSV but the newline is ;.

Now with the help of regex you can read this data like this:

import re

with open("resources/input.dat") as f:
    lines = f.readlines()
    text = "".join(lines)

regex = r"\|(.*?);"
matches = re.finditer(regex, text, re.MULTILINE | re.DOTALL)


data = []

for matchNum, match in enumerate(matches, start=1):
    for group in match.groups():
        data.append(group.split(","))

for d in data:
    print(d)

Input

|CF,2,1,1;|CK,1,3,1,1;
|NO,1,7,1,0,,0,;
|CT,1,41,0,6,Bench,24,Korrosionstest', '15A046-01,0,
otherline_data;

Output

['CF', '2', '1', '1']
['CK', '1', '3', '1', '1']
['NO', '1', '7', '1', '0', '', '0', '']
['CT', '1', '41', '0', '6', 'Bench', '24', "Korrosionstest'", " '15A046-01", '0', '\notherline_data']

As you can see even if the data block doesn't end at a new line, you still get the data until the defined end mark ;.

Edit

I downloaded your .dat file. As you can see after line 1133 there are strange characters that doesn't make sense at all. This characters or rather bytes are probably the information you need to process the data in the beginning. Basically it looks like some compressed data with the needed background information I informed you in the comment.

FAMOS has the knowledge to interpret that byte string and therefore can present you with the data as it is intented. How to interpret this? Ask the source where you get the data or find it in the FAMOS code.

I don't think somebody here can answer you this. And I don't know how. This is too specific and therefore it is better to go where you get the data.

A snippet from the .dat file: (In total 32404 lines and only 1133 with data) enter image description here

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

This is what I am getting for the entire .dat file '\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03\x00\x03
the first few lines were successfully parsed but below which I m getting this
that are control characters in hex. If you look here: donsnotes.com/tech/charsets/ascii.html you see what it means. Basically you have an end of text character, a null character in repeat. I don't know what information you want to extract from this chunk of data. But it's probably some garbage?
But famos software can extract data from any dat file even if it doesn't have any idea about how the data was created. How did the developers do this, do you hve any idea that how can it be possible if a user defines data format in a dat file
@PulkitBhatnagar it may be a standard in your field. As I see FAMOS is for measurement and stuff. Maybe that is a standard that is widely used for this kind of software. But I don't know if it's even public available.
|
0

You could use the following code:

infile = open('.input.dat', 'r')
for line in infile:
    # Typical line: variable = value
    variable, value = line.split('=')
    variable = variable.strip()  # remove leading/traling blanks
infile.close()

There is more information here: file reading

Improve this answer

1 Comment

UnpicklingError: invalid load key, '|'.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.