Is it possible to create an in-memory File object in Java?

Question

One of the optimization projects I'm working on right now makes extensive use of EPANet. We make repeated calls to two simulation methods in EPANet to understand how water flows through a water distribution network.

HydraulicSim is one of the classes we make use of. See the overloaded simulate methods:

public void simulate(File hyd) throws ENException {
    ...
}

public void simulate(OutputStream out) throws ENException, IOException {
    ...
}

public void simulate(DataOutput out) throws ENException, IOException {
    ...
}

The other class we use is QualitySim. Here, we also use the overloaded simulate methods:

public void simulate(File hydFile, File qualFile) throws IOException, ENException {
    ...
}

void simulate(File hydFile, OutputStream out) throws IOException, ENException {
    ...
}

Here's what we're currently doing:

Create two File objects, hydFile and qualFile.
Call HydraulicSim.simulate on hydFile.
Call QualitySim.simulate on hydFile and qualFile.
Delete the files.

The problem is that we have to do this a lot of times. For a large problem, we can do this hundreds of thousands or even millions of times. You can imagine the slowdown repeatedly creating/writing/deleting these files causes.

So my question is this: Is it possible for me to create these files so that they reside only in memory and never touch the disk? Each file is very small (I'm talking a few hundred bytes), so throwing them into memory won't be a problem; I just need to figure out how. I searched around and didn't really find much, save for MappedByteBuffer, but I'm not sure how, or if it's even possible, to create a File from that class.

Any advice is welcome!

It might be worth it to submit an issue to the EPANet team. The simulate methods all simply use the File objects to obtain input and output streams. If they provided a QualitySim.simulate method that took the streams as params (which they do for the output stream but not the input stream) you could bypass the FS. Or just fork it and do it yourself. — Mike B
– Mike B, Commented Mar 10, 2014 at 18:04
You can use ByteArrayOutputStream and ByteArrayInputStream, but you must pre-allocate the byte array to be large enough. — Hot Licks
– Hot Licks, Commented Mar 10, 2014 at 18:04
Do you have to use files? Since you already have some functions that use Streams you could use ByteArrayOutputStream and ByteArrayInputStream instead. — Leonard Brünings
– Leonard Brünings, Commented Mar 10, 2014 at 18:04
@LeonardBrünings The QualitySim class doesn't provide a method that accepts a stream for the input parameter (hydFile). — Mike B
– Mike B, Commented Mar 10, 2014 at 18:05
@MikeB yeah I saw that, but the change would be trivial to implement as you already mentioned. So I would just add the method an make a pull request for them to include it. — Leonard Brünings
– Leonard Brünings, Commented Mar 10, 2014 at 18:10

Leos Literak · Accepted Answer · 2014-03-10 18:57:14Z

7

The most straightforward solution is to mount ramdisk and store file here. Then you do not need to touch your code and file access will be lightning fast :-)

Linux

# mkfs -q /dev/ram1 8192
# mkdir -p /ramcache
# mount /dev/ram1 /ramcache

MacOS

http://osxdaily.com/2007/03/23/create-a-ram-disk-in-mac-os-x/

Windows

edited Mar 10, 2014 at 18:57

answered Mar 10, 2014 at 17:51

Leos Literak

9,46022 gold badges97 silver badges174 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Geoff Over a year ago

I love this option, but two of the devs use Windows, and my main dev machine is a Mac (OSX 10.8.x). Is there an equivalent option for MS and Apple devs?

Leos Literak Over a year ago

I googled osxdaily.com/2007/03/23/create-a-ram-disk-in-mac-os-x and support.microsoft.com/kb/257405

Geoff Over a year ago

It looks like the OSX option could be viable, but the Windows option is for Windows 2000. Ideally, I'd love a solution that is platform-agnostic.

Leos Literak Over a year ago

Another: softperfect.com/products/ramdisk. It is just setup script that differs on each platform.

Trying · Accepted Answer · 2014-03-10 18:09:39Z

4

you can use Memory mapped file. Java provides APIs in FileChannel Class e.g.

channel.map(FileChannel.MapMode.READ_WRITE, start, PAGE_SIZE)

public abstract MappedByteBuffer map(FileChannel.MapMode mode,
                   long position,
                   long size)
                              throws IOException

Java provides only 32 bit pointer to manipulate the MappedByteBuffer but you can use slicing to map more than 2GB of file e.g.

private void memoryMapFile(File file) throws FileNotFoundException, IOException {
        RandomAccessFile randomAccessFile = new RandomAccessFile(file, "rw");
        FileChannel channel = randomAccessFile.getChannel();
        List<MappedByteBuffer> buffers = new ArrayList<MappedByteBuffer>();
        long size = channel.size();
        long start = 0;
        int index = 0;
        for (index = 0; start + PAGE_SIZE <= size; index++) {       
            buffers.add(index, channel.map(FileChannel.MapMode.READ_WRITE, start, PAGE_SIZE));
            start = (index+1)*PAGE_SIZE;
        }
        if(start+PAGE_SIZE > size){
            buffers.add(index, channel.map(FileChannel.MapMode.READ_WRITE, start, PAGE_SIZE));
        }
    }

answered Mar 10, 2014 at 18:09

Trying

14.3k11 gold badges72 silver badges113 bronze badges

2 Comments

Leonard Brünings Over a year ago

And how do you get the methods that accept a File object and construct the streams by themselves to use this file? Does it suffice to just call memoryMapFile(File file) before invoking the other function?

Geoff Over a year ago

Thanks for this. Your example more clearly shows how to use MappedByteBuffer. I guess the problem is still that I've gotta pass a File object into these methods, though. I think modifying the EPANet code is ultimately what I need to do.

Mike B · Accepted Answer · 2014-03-11 16:28:30Z

2

It might be worth it to submit an issue to the EPANet team. The simulate methods all simply use the File objects to obtain input and output streams. If they provided a QualitySim.simulate method that took the streams as params (which they do for the output stream but not the input stream) you could bypass the FS. Or just fork it and do it yourself.

answered Mar 11, 2014 at 16:28

Mike B

5,4612 gold badges26 silver badges45 bronze badges

1 Comment

Geoff Over a year ago

Thanks for your suggestion! The issue and pull request are now live at github.com/Baseform/Baseform-Epanet-Java-Library/issues/2

dkatzel · Accepted Answer · 2014-03-10 18:00:45Z

2

It might be overkill but Java 7 provides a way to create custom file systems:

http://docs.oracle.com/javase/7/docs/technotes/guides/io/fsp/filesystemprovider.html

You can use this to create your own in memory file system that is backed by a Map or byte[]s. However, it appears that the FileSystem uses the new Path object, not a File (although there are methods to convert to/from file)

Might be overkill for what you are trying to do though.

answered Mar 10, 2014 at 18:00

dkatzel

31.8k3 gold badges67 silver badges68 bronze badges

Collectives™ on Stack Overflow

Is it possible to create an in-memory File object in Java?

4 Answers 4

4 Comments

2 Comments

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

4 Comments

2 Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related