Convert Dataframe Dataset<Row> to JSON Format of String Data Type for particular Columns and convert the JSON String back to Dataframe

For the above, I assume that the data set has been partitioned across the number of Spark workers and it is a generic data set of Row (data frame), then the below mechanism can be employed.

1. Define a class with the required attributes as a data container
2. Take the data set content as a List (takeAsList method if data set, refer)
3. Create and populate the objects of your data container (and store in such a way to identify them later, you shall have to repopulate them with decrypted data)
4. Serialize the list into a JSON array with Jackson (refer) Step 4 & 5 can be combined with Jackson custom serializer refer example
5. Make the REST call and repopulate the data container objects (after deserializing the response with Jackson)
6. Create a data frame (an example)
7. Process the data frame (dataset of rows)

NOTE: The JSON structure you have provided seems not to be correct, JSON array is [{},{},{}]

In your case, given the format of the request JSON, direct conversion of rows will not work, as mentioned in point 1, make a set of model classes, you could consider the below model classes.

package org.test.json;

import java.util.List;

public class RequestModel {

protected ApplicationInfo applicationInfo;
protected List<RequestData> requestData;

public ApplicationInfo getApplicationInfo() {return applicationInfo;}
public void setApplicationInfo(ApplicationInfo applicationInfo) {this.applicationInfo = applicationInfo;}

public List<RequestData> getRequestData() {return requestData;}
public void setRequestData(List<RequestData> requestData) {this.requestData = requestData;}

}//class closing




package org.test.json;

public class ApplicationInfo {

protected String appId;

public String getAppId() {return appId;}
public void setAppId(String appId) {this.appId = appId;}

}//class closing




package org.test.json;

public class RequestData {

protected String secureData;
protected String secureDataType;
protected int index;

public String getSecureData() {return secureData;}
public void setSecureData(String secureData) {this.secureData = secureData;}

public String getSecureDataType() {return secureDataType;}
public void setSecureDataType(String secureDataType) {this.secureDataType = secureDataType;}

public int getIndex() {return index;}
public void setIndex(int index) {this.index = index;}

}//class closing

Process the list as obtained from the data frame and populate the model classes and then convert with Jackson to get the request JSON.

The below should do what you are looking for, don't directly run this, the data set is null

        //Do not run this, will generate NullPointer, for example only
    Dataset<Row> ds=null;
    List<Row> rows=ds.collectAsList();

    RequestModel request=new RequestModel();

    //Set application id
    ApplicationInfo appInfo=new ApplicationInfo();
    appInfo.setAppId("some id");
    request.setApplicationInfo(appInfo);

    List<RequestData> reqData=new ArrayList<>();
    for(int i=0;i<rows.size();i++) {

        //Incrementally generated for each row
        int index=i;

        Row r=rows.get(i);
        int rowLength=r.size();

        for(int j=0;j<rowLength;j++) {

            RequestData dataElement=new RequestData();
            dataElement.setIndex(index);

            switch(j) {

                case 1:{dataElement.setSecureData(r.getString(j));dataElement.setSecureDataType("PID");break;}
                case 3:{dataElement.setSecureDataType(r.getString(j));dataElement.setSecureDataType("FIRSTNAME");break;}
                default:{break;}

            }//switch closing

            reqData.add(dataElement);

        }//for closing

    }//for closing