Using RSA KeyPair in C# server and PHP Client

According to 3v4l, openssl_get_publickey() is returning false. You need to give it a PEM formatted public key.

If you want a simpler interface that you're less likely to make a mistake that undermines the security of your application, check out libsodium.

Libsodium is a fork of NaCl, an easy-to-use high-speed software library for network communication, encryption, decryption, signatures, etc. written by Daniel J. Bernstein, Tanja Lange, and Peter Schwabe (three world-class cryptography experts known for their research into elliptic curve cryptography and side-channel cryptanalysis).

Libsodium is a portable fork of NaCl that is easier to use and has a lot of valuable features (e.g. password hashing with scrypt).

Generating a Libsodium Keypair In C#

using Sodium;

// snip

var keypair = PublicKeyBox.GenerateKeyPair();
string secretKey = Convert.ToBase64String(keypair.PrivateKey);
string publicKey = Convert.ToBase64String(keypair.PublicKey);

Be sure to read the relevant libsodium .NET documentation for how to use it.

Encrypting a Message from PHP to C#

<?php
$decoded = base64_decode($encoded_publickey);
define('YOUR_RAW_BINARY_CSHARP_PUBLIC_KEY', $decoded);

$php_keypair = \Sodium\crypto_box_keypair();
$php_public = \Sodium\crypto_box_publickey($php_keypair);
$php_secret = \Sodium\crypto_box_secretkey($php_keypair);

$nonce = \Sodium\randombytes_buf(\Sodium\CRYPTO_BOX_NONCEBYTES);

$message_keypair = \Sodium\crypto_box_keypair_from_secretkey_and_publickey(
    $php_secret,
    YOUR_RAW_BINARY_CSHARP_PUBLIC_KEY
);
$encrypted = \Sodium\crypto_box(
    $message,
    $nonce,
    $message_keypair
);

$encrypted will be raw binary; the C# app needs $nonce, $encrypted, and $php_public (in addition to its own secret key) to decrypt $encrypted to see what $message contains.

Be sure to read the relevant PHP libsodium documentation for how to use it.

Encrypting an Anonymous Message to your C# App from a PHP Script

<?php
$anon_msg = \Sodium\crypto_box_seal($message, YOUR_RAW_BINARY_CSHARP_PUBLIC_KEY);

The crypto_box_seal documentation is a must-read.

Looks like you're using a 512-bit key for $pubkey in PHP?

Anyway, I was able to convert the key to one OpenSSH can use using the following (uses phpseclib):

<?php
include('Crypt/RSA.php');

$a = 'BgIAAACkAABSU0ExAAIAAAEAAQBdZ3klDbVjH8oiBtGzHIMixo/TKPlv492kuau9chnARvkpxaRd8Qa82kIF2AvrEllhzjD07UHkVxoVZA2aYN+t';
$a = base64_decode($a);

echo parseCSBBlob($a);

// https://msdn.microsoft.com/en-us/library/windows/desktop/aa375601(v=vs.85).aspx
function parseCSBBlob($str) {
    // from https://msdn.microsoft.com/en-us/library/windows/desktop/aa387453(v=vs.85).aspx
    extract(unpack('atype/aversion/vreserved/Valgo', $str));
    if (ord($type) != 6) { // 6 == PUBLICKEYBLOB
        return false;
    }
    //https://msdn.microsoft.com/en-us/library/windows/desktop/aa375549(v=vs.85).aspx
    if ($algo != 0x0000a400) { // 0x0000a400 == CALG_RSA_KEYX
        return false;
    }
    $str = substr($str, 8); // aavV
    extract(unpack('Vmagic/Vbitlen/Vpubexp', $str));
    if ($magic != 0x31415352) { // RSA1
        return false;
    }
    $str = substr($str, 12); // VVV
    if (strlen($str) != $bitlen / 8) {
        return false;
    }
    $str = strrev($str);

    $rsa = new Crypt_RSA();
    $rsa->loadKey(array(
        'e' => new Math_BigInteger($pubexp, 256),
        'n' => new Math_BigInteger($str, 256)
    ));
    return $rsa;
}

You could also do this with the $rsa object that parseCSBBlob returns.

$rsa->setEncryptionMode(CRYPT_RSA_ENCRYPTION_PKCS1);
define('CRYPT_RSA_PKCS15_COMPAT', true);
$rsa->encrypt($data);