Your first, second, third, maybe even fourth pair of dress pants should be grey. Get them in different shades, fabrics, and weights. Light- to mid-grey are the most versatile and can be worn with almost any jacket without you needing to put too much thought into it. Charcoal is much less wearable, but once you already have a large wardrobe of trousers, I suppose one pair can’t hurt. You can pick these up in an assortment of flannel and twill wools. The former looks and feels better, but the latter will be harder wearing. Flannels worn day in day out won’t last long, but if you plan to wear them often, get them in worsted flannel instead of woolen. Finally, you can pick up a few seasonal greys, such as an open weave tropical wool for summer, so that your legs can breathe, and an extra heavyweight wool for winter, so that they’ll stay warm.
At some point, however, you’ll have enough grey trousers and need some variety. I suggest turning to a solid brown first. Like how brown is a wonderful color for sport coats, I think it exhibits the same richness and warmth in trousers. You can go as light as tan, or get something as dark as the golden-cast pair you see above (no longer in stock, unfortunately, but the company does have something similar). These can be worn with sport coats in navy, olive, or if the shade differs enough, even brown. For example, the dark pair pictured here could be worn with some kind of tan checked jacket for a nice autumnal look. Regardless of the kind of jacket you choose, dress shirts should probably be kept to a light blue if you’re going to the office. If it’s the weekend, try putting on a navy flannel cotton shirt or maybe even a long-sleeve polo.
Either way, pick up some brown trousers at some point. This and grey are really the only colors you need.