Removing and adding people from your community list, loosely referred to as community management, is a continuous activity on Twitter. Spare some of your time every week to manage your Twitter community. You can start with a small one and build it with time.
Depending on the available time and business goals, you may set your first objective to find 20 or 50 new following each week. Do not follow too many people at a given time. Twitter spammers are known to follow a big number of people within a short time. Someone may report you as a spammer if you act like one. Avoid this by all means as it may lead to suspension of your account by Twitter.
Twitter is usually keen on the aggressiveness of some users in following each other. Spare some few hours in your regular community management to review new followers. You can look through the profile and the stream of tweets of every new follower and then write a personal customized message instead of sending an auto-reply DM (direct message) to new followers. Connect with your Twitter community and slowly build your interactions as a good person so as to become a good member of the community.