Difference between openssl rc4 command line encrypt and cpp file implementing rc4

Question

I am working on trying to encrypt a text file via RC4 with a cpp file that I wrote with openssl/rc4 headers, and then decrypt via the command line to show that my implementation is correct.

My terminal command for the file is below, and the cpp file is below it, along with the terminal compile command I used for it.

There barely seems to be any information about this anywhere online, outside of some vague youtube videos that explain how the RC4 cypher works(which I already know). I can't find anything in the man pages to explain the details of the openssl implementation.

Any pointers on why my cpp file isn't decrypting to the original content would be much appreciated. I am tearing my hair out over here trying to figure this out. Thanks in advance.

(and yes, I understand there are vulnerabilities that make RC4 less of a good option, but right now, I just want to understand how these work)

command line encrypt:

openssl rc4-40 -k PASSword123 -in /home/chris/Desktop/test.txt -out /home/chris/Desktop/ssloutput.txt -p -nosalt

cpp file compilation:

g++ rc4.cpp -o rc4 -lssl -lcrypto

cpp file:

#include <openssl/rc4.h>
#include <string>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>

int main()
{
    int fd = open("/home/chris/Desktop/ssloutput.txt", O_RDWR);
    unsigned char keygen[12] = "PASSword123";
    RC4_KEY key;

    struct stat st;
    fstat(fd, &st);
    int size = st.st_size;

    unsigned char* fileIn;
    fileIn = (unsigned char*) calloc(size, sizeof(char));
    pread(fd, fileIn, size, 0);
    unsigned char *fileOut = (unsigned char*)malloc(size);

    RC4_set_key(&key, 16, keygen);
    RC4(&key, size, fileIn, fileOut);
    close(fd);

    int fd2 = open("/home/chris/Desktop/rc4output.txt", O_RDWR | O_CREAT);
    pwrite(fd2, fileOut, size, 0);
    close(fd2);

    free(fileIn);
    free(fileOut);

    return 0;
}

Yes, they are both 116 bytes. Shouldn't matter even if they weren't though(Although they would technically always be with RC4). The algorithm is supposed to be symmetrical. — Chris
– Chris, Commented Apr 11, 2019 at 1:17
If you’re using rc4-40 on the CLI, surely you should be passing 5 to RC4_set_key() as the key size? Otherwise if you want to use 16 byte keys, use rc4 mode on the CLI. — Alastair McCormack
– Alastair McCormack, Commented Apr 11, 2019 at 1:18
This is true, I should have caught that. But regardless, when I run just rc4 in the CLI, instead of rc4-40, it should match up with the key size = 16, but the files still don't match up. — Chris
– Chris, Commented Apr 11, 2019 at 1:35
That's a 15 byte key... If you're still telling the function to use 16, yeah, still going to be a problem. — Shawn
– Shawn, Commented Apr 11, 2019 at 2:17

Shawn · Accepted Answer · 2019-04-11 03:19:03Z

1

So, here's a version of your code with a lot of error checking added, bugs fixed, odd stuff (Using O_RDWR with open() when you're only reading or writing? pread()? pwrite()?) cleaned up, and using EVP_BytesToKey() like the -k option to openssl rc4 uses (That was the key (heh) factor):

#include <fcntl.h>
#include <openssl/evp.h>
#include <openssl/rc4.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/stat.h>
#include <sys/types.h>
#include <unistd.h>

int main() {
  int fd = open("ssloutput.txt", O_RDONLY);
  if (fd < 0) {
    perror("open ssloutput.txt");
    return 1;
  }

  struct stat st;
  if (fstat(fd, &st) < 0) {
    perror("fstat");
    return 1;
  }
  size_t size = st.st_size;

  unsigned char *fileIn = calloc(size, 1);
  if (!fileIn) {
    perror("calloc");
    return 1;
  }
  if (read(fd, fileIn, size) != (ssize_t)size) {
    perror("read");
    return 1;
  }
  close(fd);

  unsigned char *fileOut = malloc(size);
  if (!fileOut) {
    perror("malloc");
    return 1;
  }

  // Prepare the key according to the same rules as openssl rc4 -k foo
  char keygen[] = "PASSword123";
  RC4_KEY key;
  unsigned char computed_key[16];
  if (EVP_BytesToKey(EVP_rc4(), EVP_sha256(), NULL,
                     (const unsigned char *)keygen, strlen(keygen), 1,
                     computed_key, NULL) != 16) {

    fputs("Error calculating rc4 key!\n", stderr);
    return 1;
  }
  // Should match the one printed out by openssl rc4 -p
  fputs("key=", stdout);
  for (size_t n = 0; n < sizeof computed_key; n += 1) {
    printf("%02hhx", computed_key[n]);
  }
  putchar('\n');

  RC4_set_key(&key, sizeof computed_key, computed_key);
  RC4(&key, size, fileIn, fileOut);

  int fd2 = open("rc4output.txt", O_WRONLY | O_TRUNC | O_CREAT, 0644);
  if (fd2 < 0) {
    perror("open rc4output.txt");
    return 1;
  }
  if (write(fd2, fileOut, size) != (ssize_t)size) {
    perror("write");
    return 1;
  }
  close(fd2);

  free(fileIn);
  free(fileOut);

  return 0;
}

Demonstration:

$ cat input.txt
the quick brown dog jumped over the lazy red fox.
$ gcc -o myrc4 -O -Wall -Wextra myrc4.c -lcrypto
$ openssl rc4 -k PASSword123 -md sha256 -p -nosalt -in input.txt -out ssloutput.txt
key=B554C1D224D8EF1738ED4EE238317463
$ ./myrc4
key=B554C1D224D8EF1738ED4EE238317463
$ cat rc4output.txt
the quick brown dog jumped over the lazy red fox.

edited Apr 11, 2019 at 3:19

answered Apr 11, 2019 at 3:05

Shawn

53.9k3 gold badges29 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Chris Over a year ago

Thank you so much, that worked perfectly (This is a C++ file, though, so I had to cast the malloc and calloc calls to (Unisgned char*). So the issue, you're saying was not including the EVP_Bytes to key? Where did you find the information that it was needed? I'm happy for the code fix, but I'd like to understand how you got there. I would've thought that functionality should have been built in to the lssl and libcrypto libraries?

Shawn Over a year ago

If that was supposed to be C++, why was it using malloc(), C casts, C headers, etc at all? Looks like C, I assume C. Didn't notice the tag. Anyways, I got the need for EVP_BytesToKey() from that SO question I linked to in a comment. Then it was just a matter of reading that function's documentation. That question demonstrated how to use -K to give openssl the exact bytes to use as a key, so I took the other route, demonstrating how to work with -k..

Chris Over a year ago

Ah wow, I can't believe I glossed over that sentence. He specifically mentioned the function lol. Anyways, thanks for the help. Much appreciated.

Shawn Over a year ago

The OpenSSL documentation definitely leaves things to be desired, though.

Collectives™ on Stack Overflow

Difference between openssl rc4 command line encrypt and cpp file implementing rc4

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related