Are you talking about a rock garden in the traditional sense ("artistically" arranged football-to-boulder-sized rocks on a well-drained base, in some sort of hill-like or sloping setting, with alpine type plants among them) or are you talking about just plants-planted-among-gravel as beefy describes?
The following pertains only to the former:
1) A rock garden will be an intractible weed farm if you do not start with absolutely weed-free seed-free soil (that includes being free of grass roots). Otherwise the weeds' roots will hide under your rocks and boulders and you will never be able to really get rid of them.
2) It needs to be pretty well-drained, and thus it will need periodic (possibly frequent, depending on your situation and your choice of plants) watering.
3) The hardest part IMO is to get the rocks arranged so that they look vaguely natural and attractive. This is relatively seldom achieved, if you ask me

Two things that will really help are a) use only one type of rock, not a mixture, and b) set them so that it looks like the majority of each rock is underground (only the 'tip of the iceberg' sticking up) as they would normally occur in nature. This can be expensive and cumbersome if you use round boulders. But attention to making the rock pile look quasi-natural makes a HUGE difference in the appearance of the finished garden. Take a week or more, poking and adjusting at it, til you are really really satisfied before you put in the first plant.
4) Avoid plants that will take over, such as snow-in-summer (Cerastium tomentosum), because of issue #1 with the difficulty of removing the roots of anything once it gets established.
They're more work than a 'regular' flowerbed, and if you have to buy your rocks and fill gravel and such they are also more expense. And in my limited experience (but it matches what I hear from people who have much more rock gardening experience) they are also a bit more work/aggravation to maintain. However, in the right situation, well designed, they can be really attractive. And certainly there are a bunch of plants that grow better (or only) in rock-garden conditions.
If you meant something else, just ignore all the above
Pat