Welcome to the Q&A Forum

Is your site verified in Google Webmaster Tools, and Bing's Webmaster Tools? If not, get it verified, then look at your account. There's a wealth of information there, including any notices from the search engines about anything they find that might be suspect (Google now even sends a note for buying/selling links), and you can get some better statistics about your site when it's verified. I would first look in your webmaster tools, and see if the engines have flagged anything for you. If so, then you know what to work on first. Second, step back and think about what purpose your website has for your business, is it meeting that purpose, and what are useful metrics for you. In May, Matt Cutts did a video response to a person asking why toolbar PR isn't updated more frequently (available at http://www.youtube.com/watch?v=jZ9pFUSVG9g) and mentioned that while they internally keep a close eye on correct PR, they're not as worried about what shows up in the toolbar. They have some flags if something with super-high PR goes down to super-low PR in the toolbar, but a move from 4 to 3 doesn't fall under that. So, it's probably good to be thinking about your site, but think beyond the toolbar PR.