The difficulty is not rising exponentially, but it does keep going up.
Bitcoin miners have definitely gone from a large group of small miners to a small group of big miners.
In 2010 GPU (graphics card) mining made it hard to mine on CPUs. Sometime 2011 CPU mining became pointless. This meant office PCs could no longer take part in mining. Intel graphics (built into Intel CPUs) and Nvidia were both pretty bad at it. So it really meant mining was now only for gaming PCs with AMD graphics.
In 2013 Bitcoin ASIC (application-specific integrated circuit) miners appeared. Sure, some mined on FPGAs before this, but FPGAs didn't make that great an impact before ASICs arrived and killed off both GPU and FPGA mining. Now bitcoin mining was only for people who bought specialized bitcoin mining hardware.
In 2009 anyone could mine 50-100 bitcoins per day on an old laptop. After the transition to ASICs in 2013 you couldn't mine without specialized hardware.
This has greatly reduced the number of people who are able to mine.
Competition is also getting very fierce and margins are slim.
Some miners design their own ASIC chips and build their own miners with these chips. If you buy your miners from them, how can you compete on acquisition cost of miners?
Some miners have special deals with their power company. Can you get electricity cheap enough to compete?
Some say Bitmain uses Asicboost, which gives them a huge advantage. The only way to compete with Asicboost is to use Asicboost. Do you have a setup capable of using it? Will you risk a lawsuit? Yes, strangely someone has a patent pending on this simple software optimization.
Some mining operations are in very cold places like Iceland. Cooling is probably more expensive for you than for them.
The fiercer the competition, the fewer are able to stay in the game.
I wouldn't call this a bottleneck. But it is a security risk. With the majority of hashrate in China, the Chinese government can 51% attack bitcoin. Bitcoin scalability may also have been achieved more easily without huge mining entities that can veto every proposal to improve bitcoin.