Why is StandardOpenOption.DELETE_ON_CLOSE not deleting the source file of the FileChannel?

Question

We have underneath method in Java which should delete the source file when its close method is called.

private void appendFile(Path destination, Path source) {

    try (FileChannel sourceChannel = FileChannel.open(source, StandardOpenOption.READ, StandardOpenOption.DELETE_ON_CLOSE);
         FileChannel destinationChannel = FileChannel.open(destination, StandardOpenOption.WRITE, StandardOpenOption.APPEND)) {
        destinationChannel.transferFrom(sourceChannel, destinationChannel.size(), sourceChannel.size());
    } catch (IOException ex) {
       // Do something with this exception
    }
}

Now we run a functional integration test on this and see that the source file is not deleted.

Can someone help us with this ?

Answer 1

From StandardOpenOption.DELETE_ON_CLOSE option documentation:

When this option is present then the implementation makes a best effort attempt to delete the file when closed by the appropriate close method. If the close method is not invoked then a best effort attempt is made to delete the file when the Java virtual machine terminates (either normally, as defined by the Java Language Specification, or where possible, abnormally).

This option is primarily intended for use with work files that are used solely by a single instance of the Java virtual machine. This option is not recommended for use when opening files that are open concurrently by other entities. Many of the details as to when and how the file is deleted are implementation specific and therefore not specified. In particular, an implementation may be unable to guarantee that it deletes the expected file when replaced by an attacker while the file is open. Consequently, security sensitive applications should take care when using this option.

So it's only the best effort, not 100% guarantee that it will be deleted. Maybe it's still open by some other writer?

Answer 2

We had a similar issue around 2015, after a user on Win7/x64 regularly found leftover files that hadn't been removed after the program terminated. After some research and trial-and-error, we found it only happened with files that had recently been memory mapped, and fixed it by avoiding memory mapping on files we wanted to delete soon/later.

FileChannelImpl.transferFromFileChannel memory maps the source for transfer. (Depends on your JVM — I'm basing this on OpenJDK.) While the JVM cleans up after the copy by unmapping, thus invalidating the created view, the OS may delay the actual clean-up to another point in time. Until that happens, the file has a live (but inaccessible) memory map that may prevent unlinking.

This question appears related: How to properly close MappedByteBuffer?

For reference: jdk11/sun.nio.ch.FileChannelImpl#transferFromFileChannel

private long transferFromFileChannel(FileChannelImpl src,
                                     long position, long count)
    throws IOException
{
    if (!src.readable)
        throw new NonReadableChannelException();
    synchronized (src.positionLock) {
        long pos = src.position();
        long max = Math.min(count, src.size() - pos);

        long remaining = max;
        long p = pos;
        while (remaining > 0L) {
            long size = Math.min(remaining, MAPPED_TRANSFER_SIZE);
            // ## Bug: Closing this channel will not terminate the write
            MappedByteBuffer bb = src.map(MapMode.READ_ONLY, p, size);
            try {
                long n = write(bb, position);
                assert n > 0;
                p += n;
                position += n;
                remaining -= n;
            } catch (IOException ioe) {
                // Only throw exception if no bytes have been written
                if (remaining == max)
                    throw ioe;
                break;
            } finally {
                unmap(bb);
            }
        }
        long nwritten = max - remaining;
        src.position(pos + nwritten);
        return nwritten;
    }
}