Java Project Panama result from reading from MemorySegment differs from ByteBuffer

Question

I'm currently investigating switching from memory mapped files using the ByteBuffer/DoubleBuffer API to the newly introduced project Panama API.

Since MemorySegment has the .asByteBuffer method, I first tried to use it as a drop-in replacement and port the rest of the software from there.

Unfortunately, i'm unable to get a simple test to work, because when reading from the generated byte buffer, I'm getting different results than from the memory segment and the original data.

I first tried to use an Arena instead of a mapped file. Minimal failing example is down below (Java 21 with preview features):

import java.lang.foreign.Arena;
import java.lang.foreign.MemorySegment;
import java.lang.foreign.SegmentAllocator;
import java.lang.foreign.ValueLayout;
import java.lang.foreign.ValueLayout.OfDouble;
import java.nio.ByteBuffer;
import java.nio.DoubleBuffer;
import org.junit.jupiter.api.Assertions;
import org.junit.jupiter.api.Test;

public class MemorySegmentTest {

  @Test
  void testLoad() {

    try (final Arena arena = Arena.ofAuto()) {
      final MemorySegment segment = arena.allocate(Double.BYTES * 10,
          OfDouble.JAVA_DOUBLE.byteAlignment());

      SegmentAllocator alloc = SegmentAllocator.slicingAllocator(segment);

      final double[] stored = new double[]{15.345, 7.231, 123.34234};
      final MemorySegment allocated = alloc.allocateArray(ValueLayout.JAVA_DOUBLE, stored);

      // ok
      Assertions.assertEquals(stored[0], allocated.getAtIndex(ValueLayout.JAVA_DOUBLE, 0));

      final ByteBuffer byteBuffer = allocated.asByteBuffer();
      final double fromByteBuffer = byteBuffer.getDouble(0);
      Assertions.assertEquals(stored[0], fromByteBuffer); // fails

      final DoubleBuffer doubleBuffer = allocated.asByteBuffer().asDoubleBuffer();
      final double fromDoubleBuffer = doubleBuffer.get(0);
      Assertions.assertEquals(stored[0], fromDoubleBuffer); // fails
    }
  }
}

Replacing the arena with a mapped file like this also does not work.

FileChannel channel = FileChannel.open(file.toPath(), StandardOpenOption.READ,
    StandardOpenOption.WRITE, StandardOpenOption.CREATE);

final MemorySegment currentSegment = channel.map(MapMode.READ_WRITE, 0, capacity,
    Arena.ofAuto());

Due to the error message: org.opentest4j.AssertionFailedError: expected: <15.345> but was: <2.954938175610712E237> i think it's something to do with the byte aligment or so, but i'm not an expert in that.

Any ideas what might be wrong?

Answer 1

It looks like you may have a machine with ByteOrder.nativeOrder() == LITTLE_ENDIAN. However the documentation of MemorySegment.asByteBuffer reports that byteBuffer will have byte order BIG_ENDIAN.

In order for your memory access when to work you need to use consistent byte order to your chosen layout, in this case:

var layout = ValueLayout.JAVA_DOUBLE;
...
// final ByteBuffer byteBuffer = allocated.asByteBuffer();
final ByteBuffer byteBuffer = allocated.asByteBuffer().order(ByteOrder.nativeOrder());
// or rather: .order(layout.order()) 

Also, you must get DoubleBuffer with same byte order, so create doubleBuffer via byteBuffer:

// final DoubleBuffer doubleBuffer = allocated.asByteBuffer().asDoubleBuffer();
final DoubleBuffer doubleBuffer = byteBuffer.asDoubleBuffer();

Whatever layout/byte order you need to choose will of course need to be consistent with the apps that generate the file format you are mapping in your Java app.

This JDK22 based example demonstrates:

public static void main(String... args) {

    final double[] stored = new double[] { 15.345, 7.231, 123.34234 };

    try (final Arena arena = Arena.ofConfined()) {

        for (ByteOrder byteOrder : List.of(ByteOrder.nativeOrder())) {

            OfDouble layout = ValueLayout.JAVA_DOUBLE;
            final MemorySegment allocated = arena.allocateFrom(layout, stored);

            final ByteBuffer byteBuffer = allocated.asByteBuffer().order(layout.order());
            final DoubleBuffer doubleBuffer1 = byteBuffer.asDoubleBuffer();
            final DoubleBuffer doubleBuffer2 = allocated.asByteBuffer().asDoubleBuffer();

            final double fromByteBuffer = byteBuffer.getDouble(0);
            final double fromDoubleBuffer1 = doubleBuffer1.get(0);
            final double fromDoubleBuffer2 = doubleBuffer2.get(0);

            System.out.println();
            System.out.println("byteOrder                       =" + byteOrder + " layout.order=" + layout.order());
            System.out.println("allocated.asByteBuffer().order()=" + allocated.asByteBuffer().order());
            System.out.println("byteBuffer.order()              =" + byteBuffer.order());
            System.out.println("byte[]                          =" + Arrays.toString(allocated.toArray(ValueLayout.JAVA_BYTE)));
            System.out.println("fromByteBuffer="+ fromByteBuffer+ " fromDoubleBuffer1="+ fromDoubleBuffer1+ " fromDoubleBuffer2="+ fromDoubleBuffer2);
        }
    }
}

On my LITTLE_ENDIAN machine prints:

byteOrder                       =LITTLE_ENDIAN layout.order=LITTLE_ENDIAN
allocated.asByteBuffer().order()=BIG_ENDIAN
byteBuffer.order()              =LITTLE_ENDIAN
byte[]                          =[113, 61, 10, -41, -93, -80, 46, 64, 6, -127, -107, 67, -117, -20, 28, 64, 53, 7, 8, -26, -24, -43, 94, 64]
fromByteBuffer=15.345 fromDoubleBuffer1=15.345 fromDoubleBuffer2=2.954938175610712E237