From a C module, the simple thing to do is to create a full userdata with a metatable with a __gc metamethod. Store it in a field in the module's environment so it isn't collected by the GC until the module is unloaded.
According to the manual, only userdata get their __gc metamethod called by the collector, so you can't use a table to hold the module's finalizer.
For a module written in pure Lua that needs a finalizer, you still need to have a userdata to hold it up. The unsupported and undocumented but widely known function newproxy() can be used to create an otherwise empty userdata with a metatable to use for this purpose. Call it as newproxy(true) to get one with a metatable, and use getmetatable() to retrieve the metatable so you can add the __gc metamethod to it.