In my experience, the vast majority of such assembly code is in VCL or RTL units.
If you compile with "Debug .dcu's" disabled then the debugger will not step through this code. However, it will also not step through any other VCL/RTL code either - assembler or Pascal. It will still step through any code that is not part of the VCL/RTL, assuming you have the Debug information option enabled for the project.

Whether turning off VCL/RTL debug units makes debugging any particular issue you may have easier or harder will of course depend on your particular circumstances, but usually what is going on inside the VCL/RTL code is of little consequence unless and until you have eliminated the possibility of some error in your own code and then need to investigate a potential bug in the VCL/RTL itself.
For myself, I have "Debug .dcus" turned OFF unless I need them ON.
Your mileage may vary.