Laravel PHPUnit test: Multiple databases issue

In my Laravel application, I have set up two databases in a single server. All specific migrations and model classes have $connection property set up to use blog instance. The app works fine, no rollback issue on migrations.

Here is my .env file:

DB_CONNECTION=mysql
DB_HOST=db
DB_PORT=3306
DB_DATABASE=laravel
DB_USERNAME=laravel
DB_PASSWORD=laravel
DB_CHARSET=utf8mb4
DB_COLLATION=utf8mb4_unicode_ci

DB_BLOG_CONNECTION=mysql
DB_BLOG_HOST=db
DB_BLOG_PORT=3306
DB_BLOG_DATABASE=blog
DB_BLOG_USERNAME=laravel
DB_BLOG_PASSWORD=laravel
DB_BLOG_CHARSET=utf8mb4
DB_BLOG_COLLATION=utf8mb4_unicode_ci

Here is my config/database.php file:

<?php

return [
    'default' => env('DB_CONNECTION', 'sqlite'),
    'connections' => [
        'sqlite' => [
            // not touched
        ],
        'mysql' => [
            // not touched
        ],
        'blog' => [
            // copied from the 'mysql'
            // changed 'driver':
            'driver' => env('DB_BLOG_CONNECTION', 'mysql'),
        ],
];

Now, I want to write some tests for the blog feature. Here is the BlogTest.php file:

<?php

namespace Tests\Feature;

use App\Models\Blog;
use Illuminate\Foundation\Testing\RefreshDatabase;
use Tests\TestCase;

class BlogTest extends TestCase
{
    use RefreshDatabase;

    public function test_basic_structure_of_blogs_list(): void
    {
        Blog::factory()
            ->withCategories(3)
            ->count(3)
            ->create();

        $this->getJson('/api/blog')
            ->assertOk();
    }
}

Here is my phpunit.xml file:

<env name="DB_CONNECTION" value="sqlite"/>
<env name="DB_DATABASE" value=":memory:"/>
<env name="DB_BLOG_CONNECTION" value="sqlite"/>
<env name="DB_BLOG_DATABASE" value=":memory:"/>

Whenever I try to run only that specific test (php artisan test --filter BlogTest) it works. But when I try to run all tests at once (php artisan test --compact) then it fails:

FAILED  Tests\Feature\BlogTest > basic structure of blogs list                                                                                                       QueryException   
  SQLSTATE[HY000]: General error: 1 no such table: blogs (Connection: blog, SQL: insert into "blogs" ...

But when I overwrite setUp() method to this

protected function setUp(): void
    {
        parent::setUp();

        $this->artisan('migrate:rollback');
        $this->artisan('migrate');

        // or only this also works
        $this->artisan('migrate:refresh');
    }

then it works.

My question is why RefreshDatabase trait doesn't take care of running all the migrations properly or it is normal to fail when you use multiple databases?
Or even is it possible that I use in-memory SQLite for both of them and it causes an issue?