As Rob Pratt suggested, if you have one binary variable, you can just solve two problems. If you have a few (say $n$) binary variables, and $2^n$ is small, you can just solve a problem for each combination.
If that's impractical, you might look at combinatorial Benders decomposition [1]. You put $x$ and any other integer variables in the master problem; $p$, $y$ and any other continuous variables go in the subproblem. Given a feasible solution $\hat{x}$ to the master problem, you include the constraint $p\ge y$ if $\hat{x}=1$ and omit the constraint if $\hat{x}=0$. If the subproblem is feasible and its optimal value matches the claimed objective component in the master, accept the master solution. Otherwise, generate either a Benders optimality cut or a Benders feasibility cut. The cuts are similar but not identical to the cuts you get in conventional Benders decomposition. The advantage of the combinatorial Benders approach is that you do not need a constant $M$. The disadvantage is that, as an outer approximation approach, Benders may be slower than solving a single MIP model ... but that apparently is not an issue, since your minimal valid $M$ triggers numerical problems.
[1]
Codato, G. & Fischetti, M. Combinatorial Benders' Cuts for Mixed-Integer Linear Programming. Operations Research, 2006, 54, 756-766.