Maybe you already understand what's happening, but I've already written this post. Lua doesn't have a ! operator; I guess you meant not. (I wouldn't be surprised if someone has made a patched version of Lua with ! in place of not.)
a(nil, 1) returns not nil and d(1) or print("b", true). Now, not nil evaluates to true and d(1) evaluates to nil, so we have true and nil or print("b", true), which in turn evaluates to nil or print("b", true), and therefore print("b", true) is evaluated.
As to why d(1) evaluates to nil: it returns not 1 and print("c", false) or print("c", true). That is equivalent to not 1 and nil or nil because print always returns nothing when it is called, and nothing is treated as nil by the operators and and or. not x and nil or nil always evaluates to nil whether x is truthy or falsy, so d always returns nil. (The only difference is that if d receives a falsy value, both print calls are evaluated.)
You can verify that print returns nothing by calling type(print('a')): it throws the error "bad argument #1 to 'type' (value expected)", whereas type(nil) returns "nil".