The early sun was surrounded by a disc of dust and gas. Such a disc is not stable, it will develop parts that are are more dense and parts that are less dense. The combination of friction (between bodies of gas and dust) and gravity (pulling together) will tend to make the denser lumps grow denser, eventually coalessing into a ball of metal, rock and gas. This "proto-Earth" was comparable in size to the Earth now.
In the process, most of the rest of the dust and gas either fell onto the ball, or was pushed out of the way. It's not true that meteorites don't become part of the planet. Meteorites do add to the mass of the planet, but this is a slow process.
After this the early Earth collided with another planet. The other planet was destroyed and the remanents formed the Earth and Moon. By now the Earth was roughly the same size as it is now.
There was then a period when the meteorite rate increased greatly. This added a little to the mass of the Earth. The meteorites often were at least partly icy, and this water formed the seas. The rocky an metallic parts of these meteorites were incorporated into the crust of the Earth.
By the time the "Late Heavy Bombardment" finished 3.8 billion years ago, the Earth was almost exactly the same size as now. There has been some gain from later meteorite impacts, and some loss due from the top of the atmosphere, but these are minor.