First of all, you need to redraw at least the plot (the Line2D object), otherwise there will be no change in the plot.
Without knowing the purpose of not redrawing, it's hard to judge on an acceptable solution. However, usually you would just redraw the whole canvas.
To set new data, the Line2D.set_data() method can be used as shown in the following. You may press the number key (0,1,2) of the point to remove in the plot.
import matplotlib.pyplot as plt
x = [1, 2, 3]
y = [2, 3, 1]
fig, ax = plt.subplots()
line, = ax.plot(x, y, ls="None", marker="o", color='#1f77b4', ms=10)
def remove_point(event):
try:
key = int(event.key)
xvals = x[:]
xvals.pop(key)
yvals = y[:]
yvals.pop(key)
line.set_data(xvals,yvals)
fig.canvas.draw_idle()
except:
pass
fig.canvas.mpl_connect('key_press_event', remove_point)
ax.set_title("Press number of point to remove")
plt.show()