Using Python Lists and arrays

Question

After years of working with multidimensional arrays, the Python array concept of 'List' seems quite complex to me (although it is said to be superior).
I have long (thousands of lines) 2-dimensional array files with hexadecimal values with the following format (can be with or without line separation):

[0xF26,0x70,0x66],[0x1158,0x72,0xA6],[0x1388,0x72,0xB6],
[0x15BA,0x4E,0x08],[0x17E9,0x70,0x92],[0x1A1D,0x72,0x94],
[0x1C4F,0x72,0xB4],[0x4409,0x4A,0x14], etc. etc.

I wish to use the file in Python, extracting and manipulating any random element.
I realize that I will have to convert the file into a list, and use the list.
The length of the file (number records) is dynamic, the width (elements in each record) is fixed. Which is the most efficient Pythonian way for this? I can change the file format if required (separation chars, etc).
New edit:
Based on some clues in the few replies I recieved, I managed to make progress, but the question is still there.
Here is what I have done, but at the end, I can not make it function like a 2-dim array, as can be seen in the attached code:

>>> test1 = open("C:/testdata1.txt", 'r') #This opens automatically as a 
#list, but with line breaks and no start and end brackets.
>>> test2 = test1.read()                # Convert to string
>>> test2 = test2.replace("\n","")      # Remove line breaks
>>> test2 = "[" + test2 + "]"           # Add brackets
>>> print(test2)

# The result looks like pure 2-dim list, but does not behave like one:
[[0x0,0x42,0x2A],[0x229,0x44,0x7C],[0x452,0x40,0x03],[0xCF9,0x4E,0x08], 
[0xF26,0x70,0x66],[0x1158,0x72,0xA6],[0x1388,0x72,0xB6],]

#This gives an error
>>> print(test2[1][2])
Traceback (most recent call last):
  File "<pyshell#79>", line 1, in <module>
    print(test2[1][2])
IndexError: string index out of range

#But it runs like one-dim array of chars
>>> print(test2[8])
4
>>>
# If I copy and paste the above list as a new list, it works nicely!

Can better use:
>>> with open("C:/testdata1.txt", 'r') as file:
     for line in file:
     file.read()
# But, again, reading result with line breaks, no brackets.
 '[0x229,0x44,0x7C],\n[0x452,0x40,0x03],\n[0xCF9,0x4E,0x08],
 \n[0xF26,0x70,0x66],\ n[0x1158,0x72,0xA6],\n[0x1388,0x72,0xB6],'

The array format is n rows by 3 cols. It is hexadecimal (string) that any program can convert to values. As the format is Pythonian, I tried to call elements as with array, but it does not work. I have tried all kinds of code, but for no avail. — samtal
– samtal, Commented Jul 27, 2018 at 12:53

javidcf · Accepted Answer · 2018-07-27 12:51:39Z

2

If you really can format the file as you please, just make it so it is a Python module:

# bigarray.py

bigarray = [
[0xF26,0x70,0x66],[0x1158,0x72,0xA6],[0x1388,0x72,0xB6],
[0x15BA,0x4E,0x08],[0x17E9,0x70,0x92],[0x1A1D,0x72,0x94],
[0x1C4F,0x72,0xB4],[0x4409,0x4A,0x14], # etc. etc.
]

The from some other module:

# mymodule.py

from bigarray import bigarray

print(bigarray[1][2])

answered Jul 27, 2018 at 12:51

javidcf

59.9k7 gold badges87 silver badges134 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

samtal Over a year ago

This solution does not handle the imported file which is the source of the problem. When building the data manually as in the example, everything is just fine, as expected, but I have thousands of records.

whackamadoodle3000 · Accepted Answer · 2018-07-27 12:54:52Z

0

Try this:

with open("filename.txt","r") as f:
    [[hex(r) for r in e.split(",")] for e in f.read().replace("\n","")[1:-2].split("],[")]

answered Jul 27, 2018 at 12:54

whackamadoodle3000

6,7604 gold badges29 silver badges45 bronze badges

1 Comment

samtal Over a year ago

ASAIK, this is intended to clean the file and make it a list. Have tried it, but it gives many errors, and, as can be seen in my added text in the main question, but the list does not work for me, probably there are some hidden chars in the 'clean' list ??

samtal · Accepted Answer · 2018-07-30 19:14:03Z

Well, after lots of tests and tries, I found the (simple) solution as follows:

I had an excel csv output file in the following format: (The commas and brackets were added in Excel):

[0x0,0x42,0x2A],
[0x229,0x44,0x7C],
[0x452,0x40,0x03],
[0xCF9,0x4E,0x08],

I manipulated the data and 'cleaned' it to make it Python 2-dim List format (had to remove line breaks, remove trailing comma, add start and end brackets, see my original question edit)):

[[0x0,0x42,0x2A],[0x229,0x44,0x7C],[0x452,0x40,0x03],[0xCF9,0x4E,0x08]]

The problem was that although it looks like 2-dim list, it was interpreted by Python as a string.

Eventually, with the help of an old thread 1, I found the solution on how to convert a 'legal' Python string into a Python List: myList = eval(string).

After that, the structure behaved like a nice 2-dim list (array). It was simple but took me hours to find.

The full code:

>>> test = eval("[" + (open("C:/testdata1.txt" , 'r').read().replace("\n","").rstrip(",") + "]"))
>>> print(hex(test[1][2]))
0x7c

Thanks to all who tried to help.

Collectives™ on Stack Overflow

Using Python Lists and arrays

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related