It's polite to keep exterior textures to 256 and interior textures to 128 unless you really need some amazing detail, and in historical sims, of course, sometimes we do - an amaaaazing interior wall might need 256 or even 512, and a great painting might take 256. But we're all guilty of using bigger textures than required and I think that's just how it is in SL where we're (almost) all amateurs and not game design pros. You say you repeat a number of textures - look at the sizes. Somewhat following the 80/20 rule but also observations over the years in-world, I would say easily 80% of textures in a sim could be reduced at least two sizes (from say 1024 to 256, or 256 to 64), as long as the textures are on your computer or you have them full perm this is quickly done in any graphics program that has simple resizing. Try this out in a few areas and see how it looks.
I agree, if you have some older builds that you've used out of necessity, take a look at them piece-by-piece. Sculpties were a great move forward when they came but were easily abandoned for mesh because of the vertices issue (although there is still a vertices issue with much amateur mesh, it's difficult enough to learn a modelling program, learning how to reduce vertices is another step which is not often known). If you have old donuts or other tortured prims in a build that you could replace with mesh building pieces, texture to match, and then make it all convex prim, I would definitely urge that. Not always possible but the fewer vertices you have in your view, the "lighter" the drag on your system and on your visitors'.
In the end, though, what really lags us are, on the client side: Particles, and other overly high graphics preferences. I am in LOVE with the new graphics save settings, I think I have about 10 sets saved for a variety of occasions. The individual user needs to make certain that his or her settings are not so high on, say, particles or distance that he is paralyzed by lag. With the new save settings ability, you can easily switch to a lower (or higher) setting as circumstances dictate or allow. (Personally, I rarely find particle effects I find impressive and/or necessary, so I keep this setting pretty low all the time. Very helpful.)
On the server side: Avatars. Physics creates lag, so even an av wearing an invisible texture and no attachments is going to create more lag than most prims put together. And we DO want people in our sims ... sooo that's just something we're going to have to live with. Does that mean it's not worthwhile to make builds clean and trim textures down? I don't think so. I think that's just good building practice.
It does sound like textures here, if most of your textures are at 512/1024. Good luck, I hope this helps somewhat!