I'm not sure whether this is the real issue you have, but I spot an issue which can be the issue you have.

You wrote something like this:

if (pulseFromEncoder)
  if (direction1)
    OCR1A -= 100;
  else
    OCR1A += 100;

Now, let's assume the OCR1A is a unsigned 16 bit integer (not sure about this, should be verified in the atmel header files). Let's assume also that 100 corresponds to 1ms of delay on the timer. This is what happens when you rotate in direction 2:

        OCR1A  Time delay
Rotate:   300      3ms
Rotate:   200      2ms
Rotate:   100      1ms
Rotate:     0      0ms
Rotate: 65436    654ms

See? The value rolls over and the time completely changes.

Now, maybe the issues are other (for instance changing the mode to 15 instead of 4 like EdgarBonet said (no idea on what this means, you should double check the differences on the atmega datasheet), but I strongly suggest you to fix this issue, for instance this way:

const uint16_t OCR1A_Min = 0
const uint16_t OCR1A_Max = 20000
const uint16_t OCR1A_Step = 100

...

if (pulseFromEncoder)
  if (direction1)
  {
    if (OCR1A > OCR1A_Min + OCR1A_Step)
      OCR1A -= OCR1A_Step;
    else
      OCR1A = OCR1A_Min;
  }
  else
  {
    if (OCR1A < OCR1A_Max - OCR1A_Step)
      OCR1A += OCR1A_Step;
    else
      OCR1A = OCR1A_Max;
  }

Sidenote: doesn't your encoder need debounce?

I'm not sure whether this is the real issue you have, but I spot an issue which can be the issue you have.

You wrote something like this:

if (pulseFromEncoder)
  if (direction1)
    OCR1A -= 100;
  else
    OCR1A += 100;

Now, the OCR1A is a unsigned 16 bit integer. A value of 100 corresponds to 1.6ms of delay on the timer (see comment from Edgar Bonet below). This is what happens when you rotate in direction 2:

        OCR1A  Time delay
Rotate:   300      4.8ms
Rotate:   200      3.2ms
Rotate:   100      1.6ms
Rotate:     0      0ms
Rotate: 65436   1047ms

See? The value rolls over and the time completely changes.

Now, maybe the issues are other (for instance changing the mode to 15 instead of 4 like EdgarBonet said (no idea on what this means, you should double check the differences on the atmega datasheet), but I strongly suggest you to fix this issue, for instance this way:

const uint16_t OCR1A_Min = 0
const uint16_t OCR1A_Max = 20000
const uint16_t OCR1A_Step = 100

...

if (pulseFromEncoder)
  if (direction1)
  {
    if (OCR1A > OCR1A_Min + OCR1A_Step)
      OCR1A -= OCR1A_Step;
    else
      OCR1A = OCR1A_Min;
  }
  else
  {
    if (OCR1A < OCR1A_Max - OCR1A_Step)
      OCR1A += OCR1A_Step;
    else
      OCR1A = OCR1A_Max;
  }

Sidenote: doesn't your encoder need debounce?