sed is for simple s/old/new/, that is all. You aren't simply doing s/old/new/ so you shouldn't be considering sed. Just use awk:
$ cat tst.awk
function descend( internalStr) {
while( ++i <= length($0) ) {
char = substr($0,i,1)
internalStr = internalStr char
if (char == "{") {
internalStr = internalStr descend()
}
else if (char == "}") {
return internalStr
}
}
}
BEGIN { IGNORECASE=1 }
{
fullStr = externalStr = ""
i = 0
while( ++i <= length($0) ) {
char = substr($0,i,1)
externalStr = externalStr char
if (char == "{") {
gsub(/\<bla\>/,"KUI",externalStr)
fullStr = fullStr externalStr descend()
externalStr = ""
}
}
gsub(/\<bla\>/,"KUI",externalStr)
print fullStr externalStr
}
.
$ cat file
bla blab blab \cite{bla} \cite[prout]{bla} \footcite[prout][hein]{ bla } Bla aBla
bla \tag1{ bla \tag2{ bla } bla } bla
$ gawk -f tst.awk file
KUI blab blab \cite{bla} \cite[prout]{bla} \footcite[prout][hein]{ bla } KUI aBla
KUI \tag1{ bla \tag2{ bla } bla } KUI
The above uses GNU awk for word boundaries and IGNORECASE. The need for those can be worked around pretty easily with other awks.
Note that it works even for nested tags (the 2nd input/output line).