In fiat economies financial capital isn't a limiting factor since it can be and is created out of thin air as needed. The need for private citizens' money to grow the economy is often repeated idea but it doesn't hold water when you consider how their money was created in the first place. Specifically, currency issuing governments spend money into existence before being able to tax it. Therefore they don't need to tax in order to spend. If there are the real resources needed for certain economic activity to occur but the limiting factor is the lack of money, a competent government will spend the required money into that sector and the activity will materialize. There's no need to wait for private individuals to accumulate it over time in order to spend it to enable this economic activity. Crucially, even if you wait, the money is still going to come from a government's "printing press."

Other types of capital such as human, intellectual, can limit growth since they're not as easily replaceable. That's why I think your second point about who those people are is important. It is possible that they're knowledgeable workers in different domains. It is also possible that they're people skilled in exploiting others. If we assume the former, losing them isn't ideal. If we assume the latter, then it's a social value judgement of whether you want to have more or fewer of these types in your society, but they're not essential for economic growth.