QDataStream reads and writes more bytes than QFile::length() reports to have

Question

I have a utility that should copy files from one location to another.

The problem I have is when reading X bytes using the QDataStream and writing it, the number of bytes being read/written exceeds the number of bytes the file has. I see this problem happen with a number of files.

I am using a QDataStream::readRawData() and QDataStream::writeRawData() to facilitate reading/writing to and from files as shown below

 QDataStream in(&sourceFile);
 QDataStream out(&newFile);

 // Read/Write byte containers
 qint64 fileBytesRead = 0;
 quint64 fileBytesWritten = 0;
 qint64 bytesWrittenNow = 0;

 quint8* buffer = new quint8[bufSize];
 while ((fileBytesRead = in.readRawData((char*)buffer, bufSize)) != 0) {

      // Check if we have a read/write mismatch
      if (fileBytesRead == -1) {
           printCritical(TAG, QString("Mismatch read/write: [R:%1/W:%2], total file write/max [W:%3/M:%4]. File may be corrupted, skipping...").arg(QString::number(fileBytesRead), QString::number(bytesWrittenNow), QString::number(fileBytesWritten), QString::number(storageFile.size)));

           // close source file handle
           sourceFile.close();

           // Close file handle
           newFile.close();

           return BackupResult::IOError;
      }

      // Write buffer to file stream
      bytesWrittenNow = out.writeRawData((const char*)buffer, fileBytesRead);

      // Check if we have a read/write mismatch
      if (bytesWrittenNow == -1) {
           printCritical(TAG, QString("Mismatch read/write: [R:%1/W:%2], total file write/max [W:%3/M:%4]. File may be corrupted, skipping...").arg(QString::number(fileBytesRead), QString::number(bytesWrittenNow), QString::number(fileBytesWritten), QString::number(storageFile.size)));

           // close source file handle
           sourceFile.close();

           // Close file handle
           newFile.close();

           return BackupResult::IOError;
      }

      // Add current buffer size to written bytes
      fileBytesWritten += bytesWrittenNow;

      if(fileBytesWritten > storageFile.size) {
          qWarning() << "Extra bytes read/written exceeding file length";    <================= this line is hit every now and then
      }

      //...

This problem isn't consistent, but it happens every now and then, I have no idea why. Anyone have thoughts on a possible cause?

Answer 1

The name of the function QDataStream::writeRawData() sounds like ideal for writing binary data. Unfortunately, that's only half of the story.

The open-mode of the file is relevant as well under certain conditions – e.g. if the QFile is opened on Windows with QIODevice::Text:

QIODevice::Text

When reading, the end-of-line terminators are translated to '\n'. When writing, the end-of-line terminators are translated to the local encoding, for example '\r\n' for Win32.

I prepared an MCVE to demonstrate that:

// Qt header:
#include <QtCore>

void write(const QString &fileName, const char *data, size_t size, QIODevice::OpenMode mode)
{
  qDebug() << "Open file" << fileName;
  QFile qFile(fileName);
  qFile.open(mode | QIODevice::WriteOnly);
  QDataStream out(&qFile);
  const int ret = out.writeRawData(data, size);
  qDebug() << ret << "bytes written.";
}

// main application
int main(int argc, char **argv)
{
  const char data[] = {
    '\x00', '\x01', '\x02', '\x03', '\x04', '\x05', '\x06', '\x07',
    '\x08', '\x09', '\x0a', '\x0b', '\x0c', '\x0d', '\x0e', '\x0f'
  };
  const size_t size = sizeof data / sizeof *data;
  write("data.txt", data, size, 0);
  write("test.txt", data, size, QIODevice::Text);
}

Built and tested in VS2017 on Windows 10:

Open file "data.txt"
16 bytes written.
Open file "test.txt"
16 bytes written.

Result inspected with the help of cygwin:

$ ls -l *.txt
-rwxrwx---+ 1 scheff Domänen-Benutzer 427 Jun 23 08:24 CMakeLists.txt
-rwxrwx---+ 1 scheff Domänen-Benutzer  16 Jun 23 08:37 data.txt
-rwxrwx---+ 1 scheff Domänen-Benutzer  17 Jun 23 08:37 test.txt

$

data.txt has 16 bytes as expected but test.txt has 17 bytes. Oops!

$ hexdump -C data.txt
00000000  00 01 02 03 04 05 06 07  08 09 0a 0b 0c 0d 0e 0f  |................|
00000010

$ hexdump -C test.txt
00000000  00 01 02 03 04 05 06 07  08 09 0d 0a 0b 0c 0d 0e  |................|
00000010  0f                                                |.|
00000011

$

Obviously, the underlying Windows file function “corrected” the \n to \r\n – 09 0a 0b became 09 0d 0a 0b. Hence, there occurs one additional byte which was not part of the originally written data.

Similar effects may happen when the QFile is opened for reading with QIODevice::Text involved.