It seems this file is stored in a different Endianness than the one java uses. The bytes probably need to be reversed before converting to double, you could try the following code to read the value:
long l = in.readLong();
double d = Double.longBitsToDouble(Long.reverseBytes(l));
Here is an example that illustrates the problem:
double d = 500.0;
long l1 = Double.doubleToLongBits(d);
long l2 = Long.reverseBytes(l1);
System.out.println(Double.longBitsToDouble(l1));
System.out.println(Double.longBitsToDouble(l2));
Output:
500.0
2.088356E-317
RandomAccessFile, I guess you are not using serialization. So how are you writing and then reading the data? How about a code snippet for each?