com.bigdata.io
Class FileChannelUtility

java.lang.Object
  com.bigdata.io.FileChannelUtility

public class FileChannelUtility
extends Object

A helper class for operations on FileChannels.

Version:

$Id: FileChannelUtility.java 2547 2010-03-24 20:44:07Z thompsonbry $

Author:

Bryan Thompson

Field Summary

protected static boolean	DEBUG
protected static boolean	INFO
protected static org.apache.log4j.Logger	log Log for btree opeations.

Constructor Summary

FileChannelUtility()

Method Summary

static int	readAll(FileChannel channel, ByteBuffer src, long pos) Deprecated. by readAll(IReopenChannel, ByteBuffer, long) which handles transparent re-opening of the store in order to complete the read operation.
static int	readAll(IReopenChannel<FileChannel> opener, ByteBuffer src, long pos) Reads Buffer.remaining() bytes into the caller's ByteBuffer from the channel starting at offset pos.
static int	transferAll(FileChannel sourceChannel, long fromPosition, long count, RandomAccessFile out, long toPosition) FileChannel to FileChannel transfer of count bytes from the sourceChannel starting at the fromPosition onto the out file starting at its current position.
static int	writeAll(FileChannel channel, ByteBuffer data, long pos) Deprecated. by writeAll(IReopenChannel, ByteBuffer, long)
static int	writeAll(IReopenChannel<FileChannel> opener, ByteBuffer data, long pos) Write bytes in data from the position to the limit on the channel starting at pos.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

log

protected static final org.apache.log4j.Logger log

Log for btree opeations.

INFO

protected static final boolean INFO

DEBUG

protected static final boolean DEBUG

Constructor Detail

FileChannelUtility

public FileChannelUtility()

Method Detail

readAll

public static int readAll(FileChannel channel,
                          ByteBuffer src,
                          long pos)
                   throws IOException

Deprecated. by readAll(IReopenChannel, ByteBuffer, long) which handles transparent re-opening of the store in order to complete the read operation.

Reads Buffer.remaining() bytes into the caller's ByteBuffer from the channel starting at offset pos. The position of the ByteBuffer is advanced to the limit. The offset of the channel is NOT modified as a side-effect.

This handles partial reads by reading until nbytes bytes have been transferred.

Parameters:

channel - The source FileChannel.

src - A ByteBuffer into which the data will be read.

pos - The offset from which the data will be read.

Returns:

The #of disk read operations that were required.

Throws:

IOException

IllegalArgumentException - if any parameter is null.

IllegalArgumentException - if pos is negative.

IllegalArgumentException - if dst does not have any bytes remaining.

TODO:

This might need to use a static direct buffer pool to avoid leaking temporary direct buffers since Java may attempt to allocate a "temporary" direct buffer if [dst] is not already a direct buffer.

readAll

public static int readAll(IReopenChannel<FileChannel> opener,
                          ByteBuffer src,
                          long pos)
                   throws IOException

Reads Buffer.remaining() bytes into the caller's ByteBuffer from the channel starting at offset pos. The position of the ByteBuffer is advanced to the limit. The offset of the channel is NOT modified as a side-effect.

This handles partial reads by reading until nbytes bytes have been transferred.

Parameters:

opener - An object which knows how to re-open the FileChannel and knows when the resource is no longer available.

src - A ByteBuffer into which the data will be read.

pos - The offset from which the data will be read.

Returns:

The #of disk read operations that were required.

Throws:

IOException

IllegalArgumentException - if any parameter is null.

IllegalArgumentException - if pos is negative.

IllegalArgumentException - if dst does not have any bytes remaining.

TODO:

This might need to use a static direct buffer pool to avoid leaking temporary direct buffers since Java may attempt to allocate a "temporary" direct buffer if [dst] is not already a direct buffer.

writeAll

public static int writeAll(FileChannel channel,
                           ByteBuffer data,
                           long pos)
                    throws IOException

Deprecated. by writeAll(IReopenChannel, ByteBuffer, long)

Write bytes in data from the position to the limit on the channel starting at pos. The position of the buffer will be advanced to the limit. The position of the channel is not changed by this method.

Note: I have seen count != remaining() for a single invocation of FileChannel#write(). This occurred 5 hours into a run with the write cache disabled (so lots of small record writes). All of a sudden, several writes wound up reporting too few bytes written - this persisted until the end of the run (Fedora core 6 with Sun JDK 1.6.0_03). I have since modified this code to use a loop to ensure that all bytes get written.

Returns:

The #of disk write operations that were required.

Throws:

IOException

TODO:

This might need to use a static direct buffer pool to avoid leaking temporary direct buffers since Java probably will attempt to allocate a "temporary" direct buffer if [data] is not already a direct buffer. There is logic in TemporaryRawStore for handling the overflow from a heap buffer onto the disk which already does this.

writeAll

public static int writeAll(IReopenChannel<FileChannel> opener,
                           ByteBuffer data,
                           long pos)
                    throws IOException

Write bytes in data from the position to the limit on the channel starting at pos. The position of the buffer will be advanced to the limit. The position of the channel is not changed by this method. If the backing channel is asynchronously in another thread closed then it will be re-opened and the write will continue.

Note: I have seen count != remaining() for a single invocation of FileChannel#write(). This occurred 5 hours into a run with the write cache disabled (so lots of small record writes). All of a sudden, several writes wound up reporting too few bytes written - this persisted until the end of the run (Fedora core 6 with Sun JDK 1.6.0_03). I have since modified this code to use a loop to ensure that all bytes get written.

Parameters:

opener -

data -

pos -

Returns:

The #of disk write operations that were required.

Throws:

IOException

transferAll

public static int transferAll(FileChannel sourceChannel,
                              long fromPosition,
                              long count,
                              RandomAccessFile out,
                              long toPosition)
                       throws IOException

FileChannel to FileChannel transfer of count bytes from the sourceChannel starting at the fromPosition onto the out file starting at its current position. The position on the sourceChannel is updated by this method and will be the positioned after the last byte transferred (this is done by FileChannel.transferFrom(java.nio.channels.ReadableByteChannel, long, long) so we have no choice about that). The position on the out file is updated by this method and will be located after the last byte transferred (this is done since we may have to extend the output file in which case its position will have been changed anyway so this makes the API more uniform).

Parameters:

sourceChannel - The source FileChannel.

fromPosition - The first byte to transfer from the sourceChannel.

count - The #of bytes to transfer.

out - The output file.

toPosition - The data will be transferred onto the output channel starting at this position.

Returns:

#of IO operations required to effect the transfer.

Throws:

IOException

TODO:

as a workaround, detect the Windows platform and do the IO ourselves when the #of bytes to transfer exceeds 500M., does not handle asynchronous closes due to interrupts in other threads during a concurrent NIO operation on the same source/dest file channel (I have never seen this problem and perhaps the NIO API excludes it)., I have seen this exception when the JVM was allocated to much of the total RAM on the machine (in this case 2600m out of 3G available with 4G installed). The workaround is to reduce the -Xmx parameter value. This could also be corrected by failing over to a copy based on normal file IO.

 Caused by: java.io.IOException: Map failed
         at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:758)
         at sun.nio.ch.FileChannelImpl.transferFromFileChannel(FileChannelImpl.java:537)
         at sun.nio.ch.FileChannelImpl.transferFrom(FileChannelImpl.java:600)
         at com.bigdata.io.FileChannelUtility.transferAll(FileChannelUtility.java:553)
         at com.bigdata.journal.AbstractBufferStrategy.transferFromDiskTo(AbstractBufferStrategy.java:420)
         at com.bigdata.journal.DiskOnlyStrategy.transferTo(DiskOnlyStrategy.java:2442)
         at com.bigdata.btree.IndexSegmentBuilder.writeIndexSegment(IndexSegmentBuilder.java:1865)
         at com.bigdata.btree.IndexSegmentBuilder.call(IndexSegmentBuilder.java:1148)
         at com.bigdata.resources.IndexManager.buildIndexSegment(IndexManager.java:1755)
 

ISSUE:

There is a known bug with large transfers under Windows. See FileChannel.transferTo() doesn't work if address space runs out