You are using a button, which is an UI element. It is not supposed to be used on 3d objects in the game world.
On 3d objects in game world you probably want to use IPointerClickHandler. They work like this:
using UnityEngine;
using UnityEngine.EventSystems;
public class Clickable : MonoBehaviour, IPointerClickHandler
{
public void OnPointerClick( PointerEventData eventData )
{
Debug.Log ("Click recorded");
}
}
Also note that the object should have a collider attached to it, and the scene must have EventSystem and Standalone Input Module attached to some object.
There are more events supported to allow different behaviours.