0

I am porting some Objective C code.

I need to create a list (I will use any type of list, whatever makes sense) of objects. The class looks like this:

class Foo{
    double fooValue1;
    double fooValue2;
    double fooValue3;
    double fooValue4;
}

I am reading a binary file into a byte array which works fine. The byte array is a list of 4 doubles. So if there are 100 instances of Foo in the file, the byte array has 3200 bytes. i.e. 4 * 8 * 100.

in Objective C, a variable is declared as a pointer to an array of type Foo. It is very fast to load that array by simply copying the bytes. The statement to do this is:

[byteArray getBytes:fooData range:range]

where range is an NSRange instance with location=0 and length=length of the byte array, byteArray is the raw byte[] read from the file and fooData is the pointer to the target array.

At the moment, I'm looping through the byte array, creating a new instance of Foo for each 32 bytes, then assigning the fields by converting each 8 bytes into doubles. For thousands of objects, this is slow.

I'm targeting API 8 so Arrays.copyOf is not available to me although if it offers a good solution, I would consider changing the target.

The question is; Is there a way to create a list in a similar "one statement" fashion to Objective C?

Thanks

[EDIT] Here's the final code I used based on Peter's answer. I should add that the file actually contains a list of lists, separated with headers. The result of parsing the file is an array of byte arrays.

ArrayList<SCGisPointData> pointData = new ArrayList<SCGisPointData>();
SCGisPointData thisPointdata;

for (byte[] ring : linearRings) {

    DoubleBuffer buffer = ByteBuffer.wrap(ring).order(ByteOrder.nativeOrder()).asDoubleBuffer().asReadOnlyBuffer();

    thisPointdata= new SCGisPointData ();

    while (buffer.hasRemaining()) {
        thisPointdata = new SCGisPointData();
        thisPointdata.longitude = buffer.get();
        thisPointdata.latitude = buffer.get();
        thisPointdata.sinLatitude = buffer.get();
        thisPointdata.cosLatitude = buffer.get();
        pointData.add(thisPointdata);  
    }

}
Improve this question
3
  • Java doesn't work on direct memory Commented Jan 23, 2013 at 17:49
  • @CodeChimp. I know, but the JVM can - for example using System.arrayCopy. I'm not asking about pointers, just looking for a neat way to initialise a list from a byte array. Commented Jan 23, 2013 at 17:50
  • Sorry, I was mid answer when someone walked in and I must have hit enter by mistake...what I meant to say is that Java doesn't work on direct memory access like C/C++/ObjectiveC does, or the concept of a "union" where a value can be either this or that, so it can't do direct memory copies. You could possible create a class that uses custom serialization, but ultimately you will still be looping through bytes in an array. Its one of the downsides of Java, really. Commented Jan 23, 2013 at 18:24

2 Answers 2

Reset to default
2

There is no way in pure Java to magically assume one type can be turning into a different type. BTW You can do this in HotSpot using Unsafe but I don't believe you can use that in Android which is perhaps for the best.

You can speed up translation by using a direct ByteBuffer with native byte ordering. This minimises the overhead, but is not as fast in Android as it is in HotSpot (as the later can turn these into intrinsic methods)

I would time how long this takes as an estimate

// or use a MappedByteBuffer from a file
ByteBuffer bb = ByteBuffer.allocateDirect(3200).order(ByteOrder.nativeOrder());
while(bb.remaining() > 0) {
   float f= bb.getFloat();
}

Note: this won't run long enough to even be JITted to native code.

ByteBuffer bb = ByteBuffer.allocateDirect(3200).order(ByteOrder.nativeOrder());
for (int i = 0; i < 12; i++) {
    long start = System.nanoTime();
    bb.clear(); // simulate we have 3200 bytes to read.
    int count = 0;
    while (bb.remaining() > 0) {
        float f = bb.getFloat();
        count++;
    }
    long time = System.nanoTime() - start;
    System.out.printf("Took %,d micro-seconds to read %,d float values%n", time / 1000, count);
}

on HotSpot Java 7 prints

Took 960 micro-seconds to read 800 float values
Took 141 micro-seconds to read 800 float values
Took 99 micro-seconds to read 800 float values
Took 101 micro-seconds to read 800 float values
Took 100 micro-seconds to read 800 float values
Took 102 micro-seconds to read 800 float values
Took 115 micro-seconds to read 800 float values
Took 127 micro-seconds to read 800 float values
Took 108 micro-seconds to read 800 float values
Took 79 micro-seconds to read 800 float values
Took 78 micro-seconds to read 800 float values
Took 79 micro-seconds to read 800 float values
Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

Thanks Peter. This looks interesting. I'll try this and report back.
Hi Peter. I used a derivation of your solution which works nicely. I've posted the final code in case anyone else has a similar need. Thank you!
If you can avoid creating and wrapping the byte[] it will be much faster. You can read straight into the direct buffer.
Hmmmm. Very tasty, thank you for the tip! I changed the file parser to create an array of buffers, rather than an array of byte[] and as you say, another big increase in performance. Thanks again!
Those buffers are reusable. Recycling them can reduce your overhead as well. Generally you don't need more than one.
1

Take a look at MappedByteBuffer. Available on Android and lets you:

  • Map a file as a buffer of bytes
  • Read double values directly from said buffer.

You will have to be careful to select the right endianess. Here is some sample code (note that you the API will do the heavy lifting of 'converting' the byte buffer into a double buffer, and that you can retrieve multiple doubles from the buffer in one go):

final String fileName = "/some/path/data.txt";
final MappedByteBuffer byteBuffer = new FileInputStream(fileName)
    .getChannel()
    .map(MapMode.READ_ONLY, 0, 3200);
final DoubleBuffer doubleBuffer = byteBuffer.asDoubleBuffer();

double[] swap = new double[4];
doubleBuffer.get(swap);
Improve this answer

5 Comments

Thank you. Happily, the doubles are stored as big endian. I'm not sure that MappedByteBuffer gains anything since I would need the same loop to convert the bytes[] to double but I'll give it a go.
@Simon - well the conversion has to happen from bytes to double, at some level. But with the Buffer classes the API does most of the heavy lifting. I included a code sample.
Definitely also pay attention to Peter Lawrey's answer. He has alot of experience in this area.
I tried this method, and it works fine except I had left out some info which didn't seem relevant to the OP. The file contains multiple lists which are separated with headers so I had to use some pointers into the byte array to skip over those. My actual implementation reads the file into an array or byte arrays, one for each list. In this case, Peter's solution is a more natural fit so I've accepted that one but upvoted yours for effort and a nice answer should anyone have a file which is "raw". Thanks again.
PS. There is no difference in speed since they both use a wrapped buffer to do the heavy lifting.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.