Endianness matters when it comes to game consoles. The Wii, the PS3, and the XBox 360 all run big-endian, while all major desktop machines run little-endian. If there's a chance you'll want to compile your code for one of those game consoles someday, then you need to think about endianness. And if you don't think about it now, then from personal experience I can tell you that it'll be extremely painful when you have to add big-endian support to a whole completed codebase which was written without it.

Similarly, endianness can matter when you're communicating with internet servers. If you want cross-platform server software, then all your network code should really be transmitting/receiving using the standard network byte ordering: big-endian.

A bigger issue which you don't mention is that of struct padding. A single struct or class can have different sizes on different platforms or compilers, and its fields can exist at different offsets within the struct. Put simply: Even if you decide not to care about endianness, you still can't simply treat a struct or class as a binary buffer and naively write it to a file or a network packet. You need to individually read or write each field. And honestly, if you're doing that (because you have to in order to support cross-platform), you might as well convert the values to/from network byte ordering at the same time.

Endianness matters when it comes to game consoles. The Wii, the PS3, and the XBox 360 all run big-endian, while all major desktop computers (as of the date I'm writing this answer) run little-endian. If there's a chance you'll want to compile your code for one of those game consoles someday, or if someone releases another popular big-endian desktop machine someday -- it wasn't so many years ago that all Macintoshes were big-endian, after all -- then you need to think about endianness. And if you don't think about it now, then from personal experience I can tell you that it'll be extremely painful when you have to add big-endian support to a whole completed codebase which was written without it.

Similarly, endianness can matter when you're communicating with internet servers. If you want cross-platform server software, then all your network code should really be transmitting/receiving using the standard network byte ordering: big-endian.

A related issue which you don't mention is that of struct padding. A single struct or class can have different sizes on different platforms or even just between different compilers, and its fields can exist at different offsets within the struct. Put simply: Even if you decide not to care about endianness, you still can't simply treat a struct or class as a binary buffer and naively write it to a file or a network packet. In order to transmit a struct of data from one machine to another, you need to individually read or write each field, rather than treating the struct as a single blob of memory. And honestly, if you're doing that (because you have to in order to support cross-platform), then you might as well convert the values to/from network byte ordering at the same time.