Food for thought but,

1. how are you detecting collision? If using OnCollisionEnter() try using OnTriggerEnter() and vice versa. Works for me sometimes.
2. Maybe try giving the secondary objects a ForceMode.Impulse away from the walls when they are near them? Try tracking the distance between walls and secondary objects and when the distance is near do ___. Slows down performance a bit (or a lot depending on how many secondary objects you have) but one out-of-the-box way to do it

Food for thought but,

1. how are you detecting collision? If using OnCollisionEnter() try using OnTriggerEnter() and vice versa. Works for me sometimes.
2. Maybe try giving the secondary objects a ForceMode.Impulse away from the walls when they are near them? Try tracking the distance between walls and secondary objects and when the distance is near do ___. Slows down performance a bit (or a lot depending on how many secondary objects you have) but one out-of-the-box way to do it

My suggestion as per distance: create an array of gameObjects that stores the transform of each wall. Make a function that, each time it is called, iterates through the array of walls and calculates the distance from ith wall to the player. If distance between the two is less than 'x' then return true.

bool calculateDistance(gameObject[] wallArray){
for(int i = 0; i < wallArray.Length; i++){
if(distance between player and wallArray[i] < 'x') //x being arbitary value
 then return true;
}
return false;
}