I agree with Johan du Toit's answer that going "old school" would be much faster. But, the real difference is not in using C primitive types and techniques, but in knowingly having a contiguous collection of characters rather than calling NextChar to read one at a time.

Machines are different now than they were in the 60's. For something like a configuration file, it's reasonable to be able to read the entire thing into memory first. For a job input file, you can at least read an entire line at a time. Reading an entire line, you can assume that one token is in the contiguous collection that you've read ahead.

An issue with Johan's code is that input is not const. He modifies the buffer to add a nul character, and then restores it.

You can do something similar in C++, using string_view. Instead of atof, use from_chars which not only does not require a nul terminator, but is implementing using a new algorithm that's much faster, and written to be fast as opposed to the C library which was implemented to use common code.

Write your parser to read a whole line at a time into a std::string which I'll call current_line. Then set a string_view to the entire line, called input. Use input.front() to inspect the next character. The function like ScanNumber will modify input by calling input.remove_prefix to consume input.

Instead of a while loop you can use std algorithms and member functions on string_view like find_first_not_of.

Go old school!

1. For parsers where performance is critical, you really can’t beat using old-school C strings. Considering the following code:

char* scanNumber(char* input, double& val) {
    if (isdigit(*input) || *input == '-' || *input == '.') {
       char* ptr = input + 1;
       while (isdigit(*ptr) || *ptr == '-' || *ptr == '.' || *ptr == 'e')
          ptr++;
       char temp = *ptr;
       *ptr = '\0';
       val = atof(input);
       *ptr = temp;
       return ptr;
    }
    return 0;
}

There is no need to copy characters. Simply null-terminate the string where the number ends, use atof to read the number, and then reset the last character.

2. The ctype functions in MSVC are notoriously slow (See this link). You may consider creating your own versions something like this:

bool IsDigit(char c) {
    return c >= '0' && c <= '9';
}

or even faster:

#define IsDigit(X) (((uint32_t)X - '0') < 10u)

Here is an example of how this can be used:

void parse(char* input) {
    double val;
    char* next;
    for (char* ptr = input; *ptr; ) {
        next = scanNumber(ptr, val);
        if (next != 0) {
            addToken(NUMBER, val);
            ptr = next;
            continue;
        }
        switch (*ptr) {
            case ' ':
            case '\t':
            case '\r':
            case '\n':
                // skip whitespace
                ptr++;
                break;
            case ',':
                addToken(SEPERATOR);
                ptr++;
                break;
            default:
                throw std::exception("Invalid character");
        }
    }
}