I suggest using an orientation histogram. First, collect sample points on your curve, ensuring they're far enough apart to be meaningful. Choose a minimum distance of 10 pixels, for instance.
Then iterate on point triplets [A,B,C] and compute the orientation change:
v1 = (B - A).normalize();
v2 = C - B;
f = atan2(-v2.x * v1.x -v2.y * v1.y,
v2.x * v1.y - v2.y * v1.x);
Now f contains the orientation difference between BC and AB. The atan2() here is important to avoid any division by zero.
YouNow you can either compare successive orientation changes like @Jesse Emond[Jesse Emond] suggested. However, computing a histogram will be more robust:
float histo[32];
memset(histo, 0, sizeof(histo));
/* ... */
histo[31 - (int)((M_PI - f) * (32.0 / 2.0 / M_PI))] += weight(A, B, C);
(I use M_PI - f instead of f + M_PI because the M_PI value is inclusive and could lead to an integer overflow).
float weight(A, B, C) is a method that computes the weight of the stroke. A simple weight function would return the length of AC, but you could decide to give long strokes a smaller weight.
Finally, analyse the histogram. If you get most of the values around histo[15] and histo[16] it means the user did a straight line. If they are around histo[13] it means a large counterclockwise circle arc, histo[18] means a clockwise circle, histo[11] means a smaller circle, anything below histo[8] or above histo[24] means there was a brutal change in direction...