In my experience, on average Google will only index about 80% of your sitemap. Do you already have links to those pages or is this a new site? Generally, I give it two full weeks to see where it is at and see if any errors come back.
Are you using WMT to verify submission or site:operator?
As far as time goes. Make sure that your site's crawl rate is optimized too. GoogleBot works on a kind of time "budget" and can't crawl everything all the time.
Check for Dup Content. Google doesn't "hit" your site but will choose one version or another to index if the content is similar or nearly similar.
These are some starting points but 4 days isn't very long for that many pages. Give it some time and keep us posted on how it's going.