How to split these strings into an array

Here's a way to do it that will work in C and C++:

//String to tokenize:
char str[] = "let's get some tokens!";

//A set of delimiters:
char delims[] = " ";

//List of tokens:
char *tok1 = NULL,
     *tok2 = NULL,
     *tok3 = NULL;

//Tokenize the string:
tok1 = strtok(str, delims);
tok2 = strtok(NULL, delims); //after you get the first token
tok3 = strtok(NULL, delims); //supply "NULL" as first strtok parameter

You can modify this is various ways. You can put all "strtok(NULL, delims)" calls in a loop to make it more flexible, you can interface with C++ string using .c_str(), etc.

getline has an optional delimiter, so you can use stringstream to do it with very little effort on your part. The downside is that (I believe) it only works with one delimiter at a time.

Bissect the problem as follows:

1. I have a pointer to substring. How to find the next substring?
2. I have a pointer to substring. How do I add it as the next element to array?

Now, solve 1 and 2. If any problem, ask again.

You can use string::find_first_of and string::substr. Just be careful to check for "empty" strings; find_first_of will find chars, so \r and \n will both be split off by the resulting algorithm.

Alternatively, iterate over the whole string, and copy the previous part when you come across another delimiter.

A really simple way is to just use flex:
You can build a really simpler lexer for a C++ application in a few lines that is very readable.

Note:

I would note that you should be careful with '\r\n'. If you open a file in text mode (the default) then the standard stream reading will convert the standard line termination sequence into a '\n'. On some platforms the end of line termination sequence is '\r\n' and thus if you read a stream from a file you may only see a '\n' character.

split.lex

%option c++
%option noyywrap
%%
\"           return 1;
\r\n         return 2;
[^"\r\n]*    return 3;
%%

main.cpp

#include "FlexLexer.h"

int main()
{
    yyFlexLexer     lexer(&std::cin, &std::cout);
    int             token;

    while((token = lexer.yylex()) != 0)
    {
        std::string  tok(lexer.YYText(), lexer.YYText() + lexer.YYLeng());
        std::cout << "Token: " << token << "(" << tok << ")\n";
    }
}

Build

% flex split.lex
% g++ main.cpp lex.yy.cc

Run (on pre-pared file)

% cat testfile | ./a.exe
Token: 3(Here is some text)
Token: 2(
)
Token: 1(")
Token: 3(here is another line)
Token: 1(")
Token: 2(
)
Token: 3(And another line)

Building on the Rogue Wave SourcePro API you're using, you could use RWTRegex to split the string into tokens:

RWTValOrderedVector<RWCString> tokenize(const RWCString& str)
{
    RWTRegex<char> re("\\r\\n|\"|([^\"\\r]|\\r[^\\n])*|\\r$");

    RWTRegex<char>::iterator it(re, str);

    RWTValOrderedVector<RWCString> result;
    for (; it != RWTRegex<char>::iterator(); ++it) {
        result.append(it->subString(str));
    }
    return result;
}

For details on RWTRegex see http://www.roguewave.com/Portals/0/products/sourcepro/docs/12.0/html/sourceproref/classRWTRegex.html.

Here is a method that uses TR1 regex features.

std::string text("Here is some text\r\n\"here is another line\"\r\nAnd another line");
std::vector<std::string> vec;

std::regex rx("[\\w ]+|\\r\\n|\"");
std::sregex_iterator rxi(text.begin(), text.end(), rx), rxend;

for (; rxi != rxend; ++rxi)
{
    vec.push_back(rxi->str());
}

In my testing, this populates the vector with the 7 substrings in your example. I'm no expert so there may be a more correct regular expression than the one I'm using.

strtok will replace your tokens with NULL. That's why it does not include the tokens.

man strtok for more information. I'm also playing around with strtok and strtok_r as I have incoming char array of the following

Hello~Milktea~This is my message\r\nMessage~I have a good watch~Cartier\r\n

I am going to first strip the ~ (tildes) followed by the \r\n, or vice versa.