As with SEO, it's going to take some time to see the fruits of your labor with social media. Twitter is about having conversations with your customers, potential customers, etc. It's not necessarily about the number of followers you have, but about the level of engagement you have with those followers. It's not only the quality of content/information you're sending as EGOL mentions, but also the type of people you're interacting with.
Looking at your current Twitter stream, you engage with people quite a bit, but the topics are all over the place. You don't have to ONLY talk about men's clothing, but decide exactly who your audience is (men who buy nice clothes online, or maybe women looking to buy something for their man) and figure out what they talk about. Find a similar type of store as your own and look at their followers. Follow their followers (not necessarily all of them, but this is a good place to start) and see what they're talking about. Try not to just tweet about your site, but tweet interesting articles or facts as well. Again, make sure you take the time first to find out what your audience is interested in. If it's men's clothing, even better. 
As an example, I have a friend who runs a plus size women's blog and she gained followers by finding the top bloggers in the niche and interacting with them. She also found plus size women's clothing stores and went through their followers and found women she was interested in talking to. After a few months of having conversations with these bloggers and other women she became a part of that community. Her twitter stream is also a personal account so she talks about anything and everything but she focuses on women's fashion and mainly plus size fashion. People who follow her know they're not going to get bombarded with spam and only showing her own blog posts. She's going to tweet great content she finds from across the web related to what many of her followers are also interested in.
If you can take this example and figure out how to get into the community where your prime audience is then you've found a great start.
Hope this helps!