Note that the xml-to-json() function in XSLT 3.0 isn't designed to handle arbitrary XML, it's only designed to handle the "XML representation of JSON" produced by the json-to-xml() function.
You have two options: either transform your XML to a structure of maps and arrays and then serialize this as JSON, or transform it to the XML vocabulary that xml-to-json() accepts.
(The reason for this is well illustrated by your example, where you are trying to keep some of the elements represented as markup. No off-the-shelf conversion is going to do that for you.)
Also note: your expected output isn't JSON. It needs surrounding curly braces to make it JSON: there's also a stray comma that needs fixing.
I would do:
<xsl:template match="root">
<xsl:variable name="temp" as="map(*)" select="
map{ "root": [
map{ "citeID": string(citiID (:sic:)),
"bib": serialize(bib/child::node(), map{"method":"xml", "omit-xml-declaration": true()}
}]}"/>
<xsl:value-of select="serialize($temp, map{"method":"json", "indent":true()})"/>
</xsl:template>
Not tested.