Retrieve an Entity annotations using JavaScript in CRM 2011?

You can retrieve annotations querying AnnotationSet via OData.

All entities where notes are active should also have a relationship called [name of entity]_annotations that you can $expand

That's possible using OData and JavaScript. Other thing is why do you need it?

UPD. You can use following code as a basic for your functionality:

            var isAttached = false;

            XrmSvcToolkit.retrieveMultiple({
                entityName: "Annotation",
                odataQuery: "$select=AnnotationId&$filter=ObjectId/Id eq guid'" + Xrm.Page.data.entity.getId() + "' and IsDocument eq true",
                async: false,
                successCallback: function (results) {
                    isAttached = results.length > 0;
                },
                errorCallback: function (error) {
                    alert(error.message);
                }
            });

            if (!isAttached) {
                Xrm.Utility.alertDialog("Attach document first!");
                return;
            }