Binary Files

Learning Objectives

• Understand what is meant by the term stream.
• Be able to open a binary file for reading.
• Be able to read several bytes from a binary file into a buffer.
• Be able to seek to a given offset in a binary file.
• Be able to write several bytes to a binary file from a buffer.
• Be able to get the size of a file by seeking to the end of the file stream.
• Be able to tell where you are anywhere in the file stream.
• Understand how values are stored in big-endian and little-endian architectures.
• Understand what data types can be used to ensure exact data sizes.

---

What is a Binary File?

All files are technically binary files. However, what we mean here is that there is no "text processing". When you used the insertion and extraction operators on an ifstream or ofstream, the function you called automatically converted the text you gave it into the sequence of 0s and 1s required for that text to be represented. Text files are easy to read, every 8-bits (1-byte) is a character. So, we just read byte-by-byte and we have the structure of the file. Binary files, on the other hand, have a structure given by its format. For example, JPEG pictures have a specific format, and so do MP3 files. These are binary files because C++ sees them as just a sequence of 0s and 1s. WE as the programmer put the structure behind these files.

---

The <cstdio> Header

The <cstdio> header contains a data type called FILE, which is opaque--we don't know or care what's in it. We just use it to pass between C-style functions to read and write binary files. However, the function's we'll need are summarized below.

Function	Description
fopen(path, mode)	Opens a file given by path and returns a FILE pointer. The mode is a string with one or more of the following: 'w' - open for writing, 'r' - open for reading, 'b' - open as binary.
fclose(FILE)	Closes a file given a file pointer.
fseek(FILE, offset, whence)	Seeks to a location within the file based on offset. Whence can be SEEK_SET (from the start of the file), SEEK_CUR (from the current location), or SEEK_END (from the end of the file).
ftell(FILE)	Returns the byte offset of where you are in the file.
fread(buffer, size, nmemb, FILE)	Reads size x nmemb number of bytes into a pointer given by buffer from the file given by FILE. Generally used for binary files.
fwrite(buffer, size, nmemb, FILE)	Writes size x nmemb number of bytes from a pointer given by buffer to the file given by FILE. Generally used for binary files.
fprintf(FILE, string, ...)	Just like printf, except redirects the output to a file. Used for text files.
fscanf(FILE, format, ...)	Just like scanf, except reads from a file instead of console. Used for text files.
fgets(buffer, max_size, FILE);	Gets a line from the file up to and including the newline character or max_size, whichever occurs first. Returns a pointer to the start of the string or nullptr if it could not read a string. Used for text files.

---

Opening and Closing Files

So, we open a file using fopen, which takes two parameters, a path and a mode. The mode is a string that tells fopen how you want to open the file--for reading, for writing, and as binary? The fopen() function returns a FILE pointer. The FILE structure isn't important, and we really don't want to know what's in it because it is probably different for different architectures.

#include <cstdio> // for fopen and fclose
int main()
{
   FILE *fl;
   fl = fopen("/path/to/my/file.bin", "rb");  // open for rb (read/binary) 
   if (nullptr == fl) {
      printf("Could not open file.\n");
      return -1;
   }
   fclose(fl);
   return 0;
}

In the code above, we open a file called file.bin for reading and as a binary file. Again, the mode tells fopen this by specifying r for reading and b for binary. When I open the file, I ALWAYS check to see if it actually opened. If it did not, you will get nullptr.

Notice that with all of the file functions, except fopen(), we pass a FILE pointer. This is how each function knows what file you're referring to. Just like an ifstream or ofstream, we can open as many streams as we want.

---

Reading and Writing

The fread() and fwrite() functions transfer bytes directly from memory to the file or from the file to memory. There is no translation in between, unlike with fprintf() and fscanf(). The print/scan versions expect a text document, and they function with text. You can still use them with a binary file, but it might not produce the file you expect!

The fread() and fwrite() functions take four parameters each, and they both return the number of elements that were read or written. This is why I always tell you to use a size of 1. The parameters for both fread and fwrite are as follows (from left-to-right).

Parameter	Description
void *buffer	A pointer to a memory location where the bytes can be read from (for fwrite) or written to (for fread). For fwrite, this is a const void *buffer since you only READ from memory to a file.
size_t size	size_t is usually an unsigned int or unsigned long. This should always be 1 for this class! This parameter is the size of the structure you're reading or writing.
size_t nmemb	nmemb stands for number of members. For example, if I wanted to write an array, say int myarray[10], I would put 10 here for 10 elements.
FILE *stream	This is the FILE pointer to refer to whatever file you have opened. The file stream must already be opened, otherwise this could cause a segmentation fault!

Recall that I said that fread and fwrite return the number of elements. Say we had the following code:

int main() {
    FILE *fl;
    fl = fopen("my.bin", "rb");
    if (nullptr == fl)
        return -1;
    int my_values[100];
    size_t this_many = fwrite(my_values, 4, 100, fl);
    printf("I wrote %lu bytes.\n", this_many);
    fclose(fl);
    return 0;
}

The fread and fwrite functions will multiply the size parameter by the nmemb parameter for the total number of bytes to read from or write to the file, respectively. Therefore, in the code above, if everything worked OK, we would get "I wrote 100 bytes.", even though we wrote a total of 400 bytes. The reason is because we only get back how many members were written and NOT the number of bytes. Since we really deal in bytes only, I recommend doing the following for fwrite:

size_t this_many = fwrite(my_values, 1, 4*100, fl);

Now, since we say the size of each member is only 1 byte, we will get back the number of members, which incidentally is the number of bytes written to the file. This is why I recommend setting the size parameter to 1.

---

Using sizeof()

Notice how I hardcoded 4 as the size of an individual integer in the code above. This is not very good practice since the size of an integer might change from one architecture to another (not likely, but it could? happen). So, there is a compile-time function called sizeof(). This function will return the number of bytes the parameter you give it requires in memory. Here are some examples:

struct MyStruct {
   int a;
   int b;
   int c;
};
int main() {
   MyStruct ms;
   MyStruct *msp = new MyStruct;
   int my_array[100];
   printf("%u\n", sizeof(MyStruct));
   printf("%u\n", sizeof(ms));
   printf("%u\n", sizeof(msp));
   printf("%u\n", sizeof(*msp));
   printf("%u\n", sizeof(int));
   printf("%u\n", sizeof(my_array));
   delete [] my_array;
   return 0;
}

The code above produces the following output.

12
12
8
12
4
400

I can give sizeof() the name of a structure, the name of an instance of a structure, a data type (such as int), the name of an array, a pointer, or a dereferenced pointer. C++ will try to figure out what you want at compile time.

Notice that sizeof(msp) is 8. This is because we're requesting the size of msp itself, which is a pointer. Recall that pointers are always 8 bytes because they store an 8-byte memory address and not the actual structure itself. BE CAREFUL about this. Notice that we can dereference the pointer to get the actual data type. We don't need the pointer itself to be a valid memory location since this is done at compile time. Instead, we're just telling C++ we don't want the size of the memory address, but we want the size of what's AT the memory address, which is the MyStruct structure.

So, we can now write the fwrite() function much better by doing the following.

size_t this_many = fwrite(my_values, 1, sizeof(my_array), fl);
if (sizeof(my_array) != this_many) {
    printf("Could not write my_array.\n");
    return -1;
}

You need to also be careful with pointers to arrays. Remember with the new[] keyword, we don't know how many bytes we actually have. The operating system keeps track, but we can easily break out of our assigned memory spaced by simply indexing into the pointer more than we've got. The same can happen with sizeof.

int main() {
    int *p = new int [ 100 ];
    printf("%u\n", sizeof(p));
    printf("%u\n", sizeof(*p));
    return 0;
}

The code above will produce the following output.

8
4

Notice that when we deference the pointer, we only get the size of a SINGLE integer. This is because of how pointers work. The pointer is just a memory address. When we dereference that memory address, we get the size of the single integer. The same would be true if we had an array of structures. We'd only get the size of the first structure, not the entire array.

---

File Positions

Just like ifstream and ofstream, we are dealing with file streams. When we read 4 bytes from the beginning of the file, we are now positioned 4 bytes within the file. Therefore, we must have some way to tell where we are and some way to move around.

We can use the ftell() and fseek() functions to tell us where we are in the file stream and to go anywhere we want within the file, respectively. The ftell() function only takes a FILE pointer as a parameter, and it will return where you are in the file. Just like arrays, this is a 0-based index. When you first open a file, you most likely will be at index 0, unless you opened the file with the 'a' (append) keyword, where you'll be at the end of the file instead.

The fseek() function is a little bit more involved, but it's quite straightforward. We have three parameters: the file stream, the offset, and the whence. There are three ways we can seek: (1) SEEK_SET, (2) SEEK_CUR, and (3) SEEK_END. These use the offset from the beginning of the file, the current index you're looking at, or from the end of the file, respectively. Generally, we use a positive offset for SEEK_SET, a positive or negative offset for SEEK_CUR, and a negative offset for SEEK_END.

int main() {
    FILE *fl;
    fl = fopen("somefile.bin", "rb");
    printf("We are at %u\n", ftell(fl));
    fseek(fl, 0, SEEK_END);
    printf("The file is %u bytes long.\n", ftell(fl));
    fseek(fl, 0, SEEK_SET);
    printf("We are now at the beginning of the file.");
    fseek(fl, 4, SEEK_CUR); // seeks from index 0 to 4
    fseek(fl, 4, SEEK_CUR); // seeks from index 4 to 8
    printf("We are now at byte index %u\n", ftell(fl));
    return 0;
}

I've demonstrated all the ways to seek and tell where we are within a file. You must be careful, however, it IS possible to seek past the end of a file if we give an offset that's too large. Therefore, it's always advisable to seek to the end, do an ftell(), to get the total number of bytes in a file. Furthermore, when seeking 0 with a whence of SEEK_END, we are PAST the last byte in the file. So, if we did a read, even for one byte, it would fail if we did this. This is why ftell() in this case would return the size of the file and NOT the index of the last byte.

---

Example

Let's write to a file a certain structure that contains an integer and a double:

// writer.cpp
#include <cstdio>
struct FileStruct {
    int integer;
    double real;
};
int main() {
    FileStruct fs = { 1234, 7.654 };
    FILE *fl = fopen("test.bin", "wb");
    if (nullptr == fl) {
        printf("Unable to open file test.bin\n");
        return -1;
    }
    size_t bytes_written = fwrite(&fs, 1, sizeof(fs), fl);
    if (sizeof(fs) != bytes_written) {
        printf("Error writing to file\n");
    }
    else {
        printf("Wrote %u bytes!\n", bytes_written);
    }
    fclose(fl);
    return 0;
}

Now, let's make the reader read from the file written to by the writer and then output what we got:

// reader.cpp
#include <cstdio>
struct FileStruct {
    int integer;
    double real;
};
int main() {
    FileStruct fs;
    FILE *fl = fopen("test.bin", "rb");
    if (nullptr == fl) {
        printf("Unable to open file test.bin\n");
        return -1;
    }
    size_t bytes_read = fread(&fs, 1, sizeof(fs), fl);
    if (sizeof(fs) != bytes_read) {
        printf("Error reading from file\n");
    }
    else {
        printf("Read %u bytes!\n", bytes_read);
        printf("The integer is %d, and the real is %lf\n", fs.integer, fs.real);
    }
    fclose(fl);
    return 0;
}

Just like an ofstream, if we open a file for writing, and it does not exist, it will create it--if it can. Therefore, our fopen() should succeed even if test.bin doesn't exist because fopen() will create it. However, if we ran the reader.cpp program first without creating test.bin, the fopen() would fail since test.bin MUST exist if it is to be opened for reading.

---

Endianness

Endianness refers to which end of a value gets stored first. We can store things in memory storing the leftmost byte into the lowest memory value. This is called big-endian, since the "big" end (the most significant byte) gets stored first. This is called byte-order.

Big-Endian Byte Order

Big-endian byte order refers to when the most significant byte is stored first, as shown below.

Little-Endian Byte Order

The opposite of big-endian byte order would be little-endian, where the least significant byte is stored first, as shown below.

Example

How would the 32-bit value, 0xdeadbeef be stored in a little endian architecture?

The first byte to be stored will start at the little end, so the first byte stored is 0xef, then 0xbe, then 0xad, then 0xde, so 0xefbeadde.

---

Standard Data Sizes in <cstdint>

In C++, the data types char, short, int, and long have different meanings or non-exact meanings. For example, on Windows a long is 32-bits whereas a long on Unix is 64-bits for a 64-bit machine. This leads to ambiguity which causes issues especially when we need an exact number of bytes when reading/writing binary files.

Luckily we have a header at our disposal called <cstdint> (C Standard Integer). This library gives us new type definitions that are specific. They all follow the same format xintyy_t. In this case, x can be a "u" which stands for "unsigned". The yy is replaced with 8, 16, 32, or 64 to denote the number of bits this integral data type will take.

The following table shows the most common data types you will use in cstdint. The nice thing about these data sizes is they don't change depending on the architecture you're on!

Data Type	Size (in bytes)	Signed/Unsigned
int8_t	1	signed
uint8_t	1	unsigned
int16_t	2	signed
uint16_t	2	unsigned
int32_t	4	signed
uint32_t	4	unsigned
int64_t	8	signed
uint64_t	8	unsigned

---

Video