How to setup rollup for css modules in TypeScript

I managed to import css modules in my Typescript React Class using this plugin from npm.

tsconfig.json

{
  "compilerOptions": {
    "target": "ESNext",
    "outDir": "build",
    "jsx": "react",
    "noImplicitAny": false,
    "removeComments": true,
    "sourceMap": false,
    "module": "ESNext",
    "allowJs": true,
    "moduleResolution": "node",
    "esModuleInterop": true,
    "baseUrl": "src",
    "plugins": [{ "name": "typescript-plugin-css-modules" }]
  },
  "exclude": [
    "node_modules/"
  ],
  "include": [
    "src/*"
  ]
}

I also added the following module file in my src/ folder:

modules.d.ts

declare module '*.module.css' {
    const classes: { [key: string]: string };
    export default classes;
}

It suppressed all warnings and I was able to test my code fine. I have a component which imports a css module located in the same folder:

- src
  - components
    - Text.tsx
    - Text.module.css

And so my component contains the following import line:

import css from './Text.module.css';

I now want to transpile my code to commonjs to use it as a React module in other codes. Here is my rollup config:

package.json

"scripts": {
  "build": "rollup -c && tsc",
  "test": "jest"
}

rollup.config.js

import babel from 'rollup-plugin-babel';
import typescript from 'rollup-plugin-typescript2';
import pkg from './package.json';
import {terser} from 'rollup-plugin-terser';
import autoprefixer from 'autoprefixer';
import postcss from 'rollup-plugin-postcss';

export default [
    // CommonJS
    {
        inlineDynamicImports: true,
        input: './src/index.ts',
        output: [
            {
                file: pkg.main,
                format: 'cjs'
            }
        ],
        external: [
            ...Object.keys(pkg.dependencies || {})
        ],
        plugins: [
            babel({
                exclude: 'node_modules/**'
            }),
            typescript({
                typescript: require('typescript')
            }),
            postcss({
                plugins: [autoprefixer()],
                sourceMap: true,
                extract: true,
                minimize: true
            }),
            terser() // minifies generated bundles
        ]
    }
];

I am able to run yarn build without any errors, however when I look at the built code, the css module file is no longer located next to the Text.js file. Below is a screenshot of the folders generated by build:

All the css has been moved to the lib folder, and in the generated Text.js file:

Is their a way to either preserve the files structure, or to transpile in a way the import points to the correct css file ?

I saw some workarounds with webpack.config.js (running eject script), however I'm not quite easy with it (since it adds a lot of files and dependencies to the project, and I'm not sure how to handle everything well).

Thanks a lot!