We tend to think of black holes as some kind of cosmic vacuum cleaner, constantly sucking in all the material around it. And while it is true that if you managed to carefully drop an object into a black hole, you’d never ever get that object back, under normal circumstances, black holes are actually remarkably bad at pulling material that close.
There are two reasons. First, black holes aren’t actually attractive to anything for any reason other than gravity. Much like our solar system is in a stable orbit around the sun, the vast majority of a galaxy is in a stable orbit around the black hole, with no real reason to go plunging towards the centre of the galaxy.
Second, black holes are bad at being astronomical vacuum cleaners because they are very inefficient at getting material close enough to them to cross the event horizon—the point at which the black hole’s gravitational pull becomes so great as to make escape impossible—and add to the mass of the black hole. Even small black holes, which exist in great numbers in a galaxy, are much better at tearing a companion star apart than they are at actually growing their own size by consuming the other star.
Material near a black hole tends to form what’s called an accretion disk—a thin, rapidly rotating disk outside the event horizon of the black hole. The gas trying to get to the black hole will speed up the closer it gets to the black hole, and any jostling between gas particles will heat the gas to incredibly high temperatures. At these temperatures, the gas will start glowing in X-rays, which flow out vertically away from the disk. Sometimes this process also causes huge galactic winds, which pushes material vertically away from the galaxy. A significant fraction of the material which could otherwise have made it to the black hole will get pushed straight back out again before it gets particularly close.
But that is assuming that there is a lot of material near the black hole, actively falling towards the event horizon. The supermassive black holes in the centers of galaxies have an additional problem: there might not even be any material close enough to it in the first place. The Milky Way’s central black hole, for instance, seems to be surrounded by stars, but almost no gas, so there is no accretion disk around our black hole.
In order to be shredded by a black hole, a star would have to come very close to the black hole. We’ve been able to watch the star that orbits the black hole in the centre of the Milky Way orbits once every fifteen years—which is a really short time astronomically speaking. It comes within a light-day—about 26 billion km, which is only five times the distance between the sun and Neptune—of the event horizon, and that is still not close enough to get torn apart or sucked in.
The fastest way for a black hole to grow in size—at least, as far as we know right now—is by crashing into another galaxy. When that happens, after things settle down, the heaviest objects will wind up in the centre, which for two galaxies will be the two black holes. Over time, the two black holes will lose enough energy while orbiting each other to merge into a single black hole. If the other galaxy was about the same mass as the original galaxy, this should double the mass of the black hole in one fell swoop—much more efficient than than by trying to build mass with gas or matter falling into it.