First, some notes:
Even used without assigning a variable, dialogs return a result "record" object containing two key-value pairs: button returned and gave up.
To get the selected button's text (other than Cancel), use button returned of result.
When using giving up after n, use either button returned of result (with text value of "" (empty string)) or gave up of result (with boolean value of true or false).
Clicking the "Cancel" button causes an error (number -128 or "User canceled."). To handle the Cancel button, use a try block.
In my first example, I'm providing a Cancel button in multiple places. This demonstrates that the on error code will work regardless of which dialog in the try block returns the Cancel button. If you don't need a Cancel, then just provide a single button such as buttons "OK".
I'm using parens around some text in my examples, but AppleScript does not require these. AppleScript does require quotes around text.
Tested in both Monterey and High Sierra, and using Mar10Josh's answer, with fixes, as the basis for my examples to make it simple and clear...
try
display dialog ("An example dialog.") buttons {"Cancel", "OK"} giving up after 5
if button returned of result = "OK" then
display dialog ("You clicked OK.") buttons {"OK", "Cancel"}
else if gave up of result then
display dialog ("You let the dialog expire.") buttons {"OK", "Cancel"}
end if
-- handle the Cancel button here
on error "User canceled." -- or you can say: on error number -128
display dialog ("You clicked Cancel.") buttons "OK"
end try
Same thing, stated slightly differently to show some other stuff (text without parens, giving up after option, generic error handling):
try
-- uncomment following line to see non-Cancel-related error result ;)
-- error "foobar!"
display dialog "An example dialog." buttons {"OK", "Cancel"} giving up after 5
if button returned of result = "OK" then
display dialog "You clicked OK." buttons "OK"
else if gave up of result then
display dialog "You let the dialog expire." buttons "OK"
end if
-- handle any error that might occur here;
-- errText and errNum are variable names I chose for these values
on error errText number errNum
if errText is "User canceled." then -- or you can say: if errNum is -128
display dialog "You clicked Cancel." buttons "OK"
else
display dialog "Error: " & errText & return & return & "Number: " & errNum buttons "OK"
end if
end try