You haven't shown what your particular functions are, but this is almost certainly a sampling problem. Gnuplot doesn't really draw curves for functions - it actually computes the functions at multiple points and connects them with straight lines, similarly to what would happen if you were plotting a data file. The number of points that it computes is user settable.
Suppose that I do plot sin(x) and see this:

Here the sampling rate is set pretty low. We can look at the individual points in order to see what is going on.

In order to improve this, I need to increase the sampling rate by using the set samples ? command. The default is 100 (in 5.0 patch level 6). Depending on how rapidly the function changes, higher values may be needed. I usually set it to around 1000 with set samples 1000. This changes the graph to

which produces a much nicer smooth curve. Again, this is just a bunch of points connected by straight lines, but when there are a lot of these, it looks like a smooth continuous curve.
We can look at the individual points again (using a sampling rate of 100 as 1000 is too many to clearly see the points)

We can also see here that there is not much difference between the graph with 1000 points and 100 points. In the case of a sine curve, 100 is enough to see a smooth graph, but with a faster changing curve, we may need more.
The set samples command takes (optionally) two values, but the second value is only used for 3d plots. You can find out more with the help samples command.
set samples 200