It's better to read the four bits simultaneously. This can be done using direct port manipulation. I would connect the four most significant bits of the encoder to pins 4 – 7 of the Arduino. As the encoder outputs are open-collector, you have to enable the pullups in setup():

Next step is to decode the Gray code. I let you research how to do that, as you can probably find examples or even ready-made libraries for that.

Next, you have to extend the value in order to know the absolute position of the cable-cam. The encoder only tells you where you are  within a revolution, but it doesn't tell you how many revolutions you did. This is done by comparing the current encoder position with the previous one, taking the difference (which can be positive or negative), and adding it to the previously computed absolute position.

  Something like this:

Beware of the types of the variables. The type of delta, for instance, ensures that the result of the subtraction is wrapped in the correct way to yield a number between −128 and +127. And note that you have to call this function frequently enough: more than once per half revolution. Otherwise it won't know which direction the wheel turned.

int32_t left_limit = INT32_MIN;
int32_t right_limit = INT32_MAX;

// To set the left limit to the current position:
left_limit = absolute_position;
// and likewise for the right limit.

and condition the maximum speed on the distance to the closets limit.

It's better to read the four bits simultaneously. This can be done using direct port manipulation. I would connect pins 9 – 12 of the encoder to pins 4 – 7 of the Arduino respectively. As the encoder outputs are open-collector, you have to enable the pullups in setup():

Next step is to decode the Gray code. I let you research how to do that, as you can probably find examples, or even ready-made libraries for that.

Next, you have to extend the value to a wider bit length in order to know the absolute position of the cable-cam. The encoder only tells you where you are  within a revolution, but it doesn't tell you how many revolutions you did. The extension is done by comparing the current encoder position with the previous one, taking the difference (which can be positive or negative), and adding it to the previously computed absolute position. Something like this:

Beware of the types of the variables. The type of delta, for instance, ensures that the result of the subtraction is wrapped the correct way to yield a number between −128 and +127. Thus, the sign will tell you which is the shortest path (clockwise or counterclockwise) from the previous to the current position.

Note also that you have to call this function frequently enough: more than once per half revolution. Otherwise it won't know which direction the wheel turned. If your loop() runs fast enough (you don't use delay(), right?), you just have to call this function once in loop(). Otherwise you would have to setup a timer interrupt to ensure the updates are frequent enough.

int32_t left_limit = INT32_MIN;
int32_t right_limit = INT32_MAX;

// To set the left limit to the current position:
if (user_pressed_the_relevant_button()) {
    left_limit = absolute_position;
}
// and likewise for the right limit.

and condition the maximum speed on the distance to the closets limit. Assuming you store the speed as a signed number (the sign being the direction), you would constrain it to always be between a negative min_speed and a positive max_speed, which could be updated as follows:

// Slow down if close to a position limit.
if (absolute_position - left_limit < BREAKING_DISTANCE) {
    min_speed = map(absolute_position,
        left_limit, left_limit + BREAKING_DISTANCE,
        0, -ABSOLUTE_MAX_SPEED);
} else {
    min_speed = -ABSOLUTE_MAX_SPEED;
}
if (right_limit - absolute_position < BREAKING_DISTANCE) {
    max_speed = map(absolute_position,
        right_limit - BREAKING_DISTANCE, right_limit,
        +ABSOLUTE_MAX_SPEED, 0);
} else {
    max_speed = +ABSOLUTE_MAX_SPEED;
}
speed_to_apply = constrain(requested_speed, min_speed, max_speed);

Note that this makes the speed limit a linear function of the position. You may want to look into easing functions to have a smoother breaking.