Because the outer while loop's stdin (the input to the script) is set to read from 1_to_5_file.txt, that is inherited to all commands running inside the while loop. One way to work around that is to save the original stdin in a different descriptor, which we can then use inside the loop. In my example I will store it in file descriptor 3.
#!/bin/bash
while read NUMBER
do echo "Prime number (answer y or n)?"
while read ANSWER
do
test "$ANSWER" == "y" && echo "Prime" && break
test "$ANSWER" == "n" && echo "Non-prime" && break
test "$ANSWER" != "y" -a "$ANSWER" != "n" && echo "Type y or n"
done 0<&3
done 3<&0 < 1_to_5_file.txt
The last line makes a copy of file descriptor 0 in file descriptor 3. Then inside the loop file descriptor 3 points to the original stdin. On the inner while loop's last line I then copy file descriptor 3 to file descriptor 0 which thus makes the stdin inside the loop again point to the original stdin.
For the record, "<" is the same as "0<" and ">" is the same as "1>". Thus the 0<&3 could also have been written just <&3.
Another, more elagant solution proposed by @chepner would be to open the 1_to_5_file.txt on file descriptor 3 instead and just change the first read command to read from that one instead:
#!/bin/bash
while read NUMBER <&3
do echo "Prime number (answer y or n)?"
while read ANSWER
do
test "$ANSWER" == "y" && echo "Prime" && break
test "$ANSWER" == "n" && echo "Non-prime" && break
test "$ANSWER" != "y" -a "$ANSWER" != "n" && echo "Type y or n"
done
done 3< 1_to_5_file.txt
Please note that compared to the code in the question, the last line has 3< instead of just 3.