How to convert a string in cpp struct to python struct

Question

I wrote a tcp server and tcp client, and I want to send a structure from client to server, and then the server send the structure back. Server is written in cpp, and client is written in python. Then I met a problem, that my structure has a member which type is string, then I cannot find a proper format string to describe this member in python(https://docs.python.org/2/library/struct.html). So I want to know how can I describe the string attribute in python?

below is the minimal code: ------server---------

#include <stdlib.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <cstring>
#include <iostream>

struct Message
{
    int id;
    std::string text;
};

int main()  
{
    int sockfd = socket(AF_INET, SOCK_STREAM, 0);
    struct sockaddr_in my_addr;  
    bzero(&my_addr, sizeof(my_addr));
    my_addr.sin_family = AF_INET;
    my_addr.sin_port = htons(11111);
    my_addr.sin_addr.s_addr = htonl(INADDR_ANY);

    unsigned retry_times = 0;
    while(retry_times < 3)
    {
        if(bind(sockfd, (struct sockaddr*)&my_addr, sizeof(my_addr)) == 0)
        {
            break;
        }
        sleep(1000);
        ++retry_times;
    }
    listen(sockfd, 1);

    struct sockaddr_in client_addr;              
    socklen_t addr_len = sizeof(client_addr);        
    int connfd = accept(sockfd, (struct sockaddr*)&client_addr, &addr_len);

    while(true)  
    {
        Message msg;
        unsigned int len = recv(connfd, &msg, sizeof(msg), 0);
        if(len == 0)
        {
            break;
        }
        std::cout << msg.id << "--" << msg.text << std::endl;
        send(connfd, &msg, sizeof(msg), 0);
    }

    close(connfd);
    close(sockfd);
}

-----------client-----------

import socket
import struct

class TCPClient:
    def __init__(self):
        self.server_address = ('127.0.0.1', 11111)  
        self.sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        self.sock.connect_ex(self.server_address)

    def send(self, msg):
        self.sock.send(msg)

    def recv(self):
        return self.sock.recv(512)

    def close(self):
        self.sock.close()

if __name__ == '__main__':
    client = TCPClient()
    fmt = 'is'         #### what format string will be better? this fmt is useless 
    msg = bytearray(512)
    struct.pack_into(fmt, msg, 0, 1, "Hello")
    client.send(msg)
    data = client.recv()
    print struct.unpack_from(fmt, data)
    client.close()

You can't just ship the binary format of a struct. You need to define a protocol and have both sides implement the protocol. Your string is likely not even stored within the struct -- it's in dynamically allocated memory outside the struct, so you're probably not even sending the string in any format, much less a defined, stable protocol — xaxxon
– xaxxon, Commented Sep 26, 2017 at 9:24
en, you suggestion is right, I didn't allocate enough memory for my structure. Thank you sincerely — Chen Dong
– Chen Dong, Commented Sep 26, 2017 at 10:00
I'm concerned you completely misunderstood my answer. You're approaching this completely wrong. it has nothing to do with how much space you allocate for the structure, you cannot just ship a data structure like this across the wire -- it's meaningless on the other side. You need to define a protocol for how you intend to represent the things you want to ship and then make both sides match that specification. Shipping the binary representation of a complex struct is not going to work for you in any meaningful way. — xaxxon
– xaxxon, Commented Sep 26, 2017 at 10:03
Here this code is not for an industrial project, I just want to test cpp structure convert to python structure — Chen Dong
– Chen Dong, Commented Sep 27, 2017 at 2:31
You need to read my responses and the answer below carefully. You are doing this VERY WRONG — xaxxon
– xaxxon, Commented Sep 27, 2017 at 2:50

John Zwinck · Accepted Answer · 2017-09-26 12:13:17Z

1

There is no Python format string which will fix this problem. The root cause is that you are sending the bytes of a std::string on the network. But a std::string often contains a pointer (and size) to the actual content, which means you are not sending the actual content at all!

To fix it, you have two main choices--either modify your struct:

struct Message
{
    int32_t id; // use a fixed-size integer type
    char text [50]; // assumes you know the max length
};

Or more conventionally, write a serialization routine:

void send_msg(int fd, const Message& msg)
{
    // XXX: you must check the return value of send()
    send(fd, &msg.id, sizeof(msg.id), 0);
    uint32_t text_size = msg.text.size();
    send(fd, &text_size, sizeof(text_size), 0);
    send(fd, &msg.text.data(), msg.text.size(), 0);
}

Now we are sending the ID (4 bytes, Python format i), the text_size (4 bytes, unsigned, Python format I), then the text (text_size bytes, Python format '{}s'.format(text_size)). You can unpack it in two steps on the receiving end: first unpack iI to get the ID and text_size, then unpack '{}s'.format(text_size) to get the text.

Note that since you are using TCP, partial writes and reads are possible. So you need to deal with the fact that your sender might send only half a message, and your receiver might receive only half a message. Your current logic does not handle this (but will likely appear to work most of the time if text_size is less than 500).

answered Sep 26, 2017 at 12:13

John Zwinck

252k44 gold badges346 silver badges459 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Chen Dong Over a year ago

your second method seems break my original idea(used 3 send command), but you showed me another way to solve this question

Collectives™ on Stack Overflow

How to convert a string in cpp struct to python struct

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related