You have to understand the difference between copyrights, patents and trademarks. Please read the quoted text below.
Copyrights protects only "original works of authorship fixed in a tangible medium of expression". It doesn't cover ideas, concepts, processes, facts and algorithms. Copyright covers the "fixed" tangible expression of ideas, concepts, processes, facts and algorithms.
Ideas, concepts, processes and algorithms are protected by patents.
I assume you want to create a software that mimics or copies part of the functionality of some other possibly copyleft-licensed open source software, and this software is in Linux userspace and makes kernel system calls.
Alternatively, you may want to create an open source software that mimics or copies part of the functionality of some other possibly proprietary software, and this software is in Linux user space and makes kernel system calls. Please note that you may not be allowed by the proprietary software's EULA to reverse engineer the software.
You must not copy the source code, documentation, images or videos from the original software, as those are protected by copyright law and may subject your software to the same copyleft licensing terms. However, the ideas, concepts, algorithms, UI/UX flows and/or processes can be copied from the copylefted software, if they are not protected by patents.
So, based on this, I would say tracing kernel system calls, and reimplementing them, would not be infringing copyrights and would not be considered the creation of a derivative work of the original software.
Some examples: NTFS is a proprietary filesystem in Microsoft Windows. SMB is a proprietary network file sharing protocol in Microsoft Windows. Open source Linux software were created for both of these, based on reverse-engineering. This does not infringe Microsoft's copyrights. Patents are another matter to check, of course.
From the U.S. Copyright Office FAQ page on Copyright In General:
What is copyright?
Copyright is a form of protection grounded in
the U.S. Constitution and granted by law for original works of
authorship fixed in a tangible medium of expression. Copyright covers
both published and unpublished works.
What does copyright protect?
Copyright, a form of intellectual property law,
protects original works of authorship including
literary, dramatic, musical, and artistic works, such as poetry,
novels, movies, songs, computer software, and architecture. Copyright
does not protect facts, ideas, systems, or methods of operation,
although it may protect the way these things are expressed. See
Circular 1, Copyright Basics, section "What Works Are Protected."
How is a copyright different from a patent or a trademark?
Copyright protects original works of authorship, while a patent
protects inventions or discoveries. Ideas and discoveries are not
protected by the copyright law, although the way in which they are
expressed may be. A trademark protects words, phrases, symbols, or
designs identifying the source of the goods or services of one party
and distinguishing them from those of others.
When is my work protected?
Your work is under copyright protection the
moment it is created and fixed in a tangible form that it is
perceptible either directly or with the aid of a machine or device.
On the legality of reverse engineering:
For U.S. law on reverse engineering, please see Reverse Engineering and the Law: Understand the Restrictions to Minimize Risks by Rahul Vijh.
Quoting from that article:
In short, to ensure you steer clear of any legal risk of reverse
engineering, it should be performed only to the extent of allowances,
such as for accessing ideas, facts, and functional concepts contained
in the product. Be especially cognizant of EULA agreements that state
“no reverse engineering”, copyright laws, and anti-circumvention
provisions before proceeding to perform any reverse engineering on the
product.
For EU and UK law on reverse engineering, please see Reconciling trade secrets and reverse engineering in the EU and UK by Eversheds Sutherland.
From EU Directive 2016/943 (16):
... Reverse engineering of a lawfully acquired product should be
considered as a lawful means of acquiring information, except when
otherwise contractually agreed. ...
I am not a lawyer and this is not legal advice.