Welcome to the Q&A Forum

Thanks a lot, Dr.Pete

Hi, Andy, very kind of you. Unfortunately my project is about CZ, SK, HU countries but I am actually interested in whether such tools exists for other parts of thw world. Thanks.

I am interested in the following data:

• searches in google and youtube

• limited to certain countries

• focusing on a precise segment (beauty): the question is NOT what other word people add next to the search term "beauty". My quest is for searches related to beauty (cosmetics rather than fashion),

• top bloggers /vloggers in a certain region and topic

• top e-commerce sites / webshops in a certain region / segment.

Thanks a lot in advance!

Hi, everyone: problem solved.

Here is what I did: I created a seperate sitemap-xml and linked to all the new pdfs.

I updated the general sitemap.xml and linked to the new sitemap as well.

I (re)submitted both sitempas via the Webmaster Tools.

Within a few hours, most of pdfs got indexed and the overall quality of search has improved dramatically. Thanks for all your help.

Try this, I hope it helps: http://ready.mobi/launch.jsp?locale=en_EN

Hi! With this file, you can create a Google-friendly sitemap for any given folder almost automatically. No limits on the number of files. Please note that the code is the courtesy of @frkandris who generously helped me out when I had a similair problem. I hope it will be as helpful to you as it was to me

• Copy / paste the code below into a text editor.
• Edit the beginning of the file: where you see seomoz.com, put your own domain name there
• Save the file as getsitemap.php and ftp it to the appropriate folder.
• Write the full URL in your browser: http://www.yourdomain.com/getsitemap.php
• The moment you do it, a sitemap.xml will be generated in your folder
• Refresh your ftp client and download the sitemap. Make further changes to it if you wish.

===  CODE STARTS HERE ===

define(DIRBASE, './');define(URLBASE, 'http://www.seomoz.com/'); $isoLastModifiedSite = "";$newLine = "\n";$indent = " ";if (!$rootUrl) $rootUrl = "http://www.seomoz.com"; $xmlHeader = "$newLine"; $urlsetOpen = "<urlset xmlns="&quot;http://www.google.com/schemas/sitemap/0.84\&quot;"  =""   <="" span="">xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance\" xsi:schemaLocation="http://www.google.com/schemas/sitemap/0.84 http://www.google.com/schemas/sitemap/0.84/sitemap.xsd\">$newLine";$urlsetValue = "";$urlsetClose = "</urlset>$newLine"; function makeUrlString ($urlString) {   return htmlentities($urlString, ENT_QUOTES, 'UTF-8');} function makeIso8601TimeStamp ($dateTime) {   if (!$dateTime) {       $dateTime = date('Y-m-d H:i:s');   }   if (is_numeric(substr($dateTime, 11, 1))) {       $isoTS = substr($dateTime, 0, 10) ."T"                .substr($dateTime, 11, ."+00:00";   }   else {       $isoTS = substr($dateTime, 0, 10);   }   return $isoTS;} function makeUrlTag ($url, $modifiedDateTime, $changeFrequency, $priority) {   GLOBAL $newLine;   GLOBAL $indent;   GLOBAL $isoLastModifiedSite;   $urlOpen = "$indent<url>$newLine";</url>   $urlValue = "";   $urlClose = "$indent$newLine";   $locOpen = "$indent$indent<loc>";</loc>   $locValue = "";   $locClose = "$newLine";   $lastmodOpen = "$indent$indent<lastmod>";</lastmod>   $lastmodValue = "";   $lastmodClose = "$newLine";   $changefreqOpen = "$indent$indent<changefreq>";</changefreq>   $changefreqValue = "";   $changefreqClose = "$newLine";   $priorityOpen = "$indent$indent<priority>";</priority>   $priorityValue = "";   $priorityClose = "$newLine";    $urlTag      = $urlOpen;   $urlValue     = $locOpen .makeUrlString("$url") .$locClose;   if ($modifiedDateTime) {    $urlValue .= $lastmodOpen .makeIso8601TimeStamp($modifiedDateTime) .$lastmodClose;    if (!$isoLastModifiedSite) { // last modification of web site        $isoLastModifiedSite = makeIso8601TimeStamp($modifiedDateTime);    }   }   if ($changeFrequency) {    $urlValue .= $changefreqOpen .$changeFrequency .$changefreqClose;   }   if ($priority) {    $urlValue .= $priorityOpen .$priority .$priorityClose;   }   $urlTag .= $urlValue;   $urlTag .= $urlClose;   return $urlTag;} function rscandir($base='', &$data=array()) {  $array = array_diff(scandir($base), array('.', '..')); # remove ' and .. from the array /  foreach($array as $value) : / loop through the array at the level of the supplied $base /    if (is_dir($base.$value)) : / if this is a directory /     $data[] = $base.$value.'/'; / add it to the $data array /     $data = rscandir($base.$value.'/', $data); / then make a recursive call with the     current $value as the $base supplying the $data array to carry into the recursion /    elseif (is_file($base.$value)) : / else if the current $value is a file /     $data[] = $base.$value; / just add the current $value to the $data array */    endif;  endforeach;  return $data; // return the $data array } function kill_base($t) {   return(URLBASE.substr($t, strlen(DIRBASE)));} $dir = rscandir(DIRBASE);$a = array_map("kill_base", $dir); foreach ($a as $key => $pageUrl) {       $pageLastModified = date ("Y-m-d", filemtime($dir[$key]));       $pageChangeFrequency = "monthly";       $pagePriority = 0.8;        $urlsetValue .= makeUrlTag ($pageUrl, $pageLastModified, $pageChangeFrequency, $pagePriority); } $current = "$xmlHeader$urlsetOpen$urlsetValue$urlsetClose"; file_put_contents('sitemap.xml', $current); ?>

=== CODE ENDS HERE ===

It may be a good idea to include all the pdf files on the sitemap, even if it is a troublesome process.

Otherwise it just takes too long for Google to index them.

What still surprises me is that even for a site search, you need to win the 'indexing battle'. I thought that Google indexes everythig within the map for the 'sake of the site search' and displays the results when a visitor is searching within the site. Less fancy softwares are actually doing the job. I thought a Google Site Search provides something even better.

Last crawl - thanks, great info.

yes, all new pdfs are linked from the html files.

This the summary page of one article: http://www.ejmh.eu/5archives_ppr_jaggle_061.html

In the middle of the page, you see 'download full text' - this is from where the individual papers (pdf) are linked.

You are great, thanks for your time. Yeah, I did check things out with this google command: there are pdf's listed but these are all old pdfs I have put up a long time ago. None of the pdfs I have put up recently are among those indexed.

Do you think that only those urls come up through a customized site search that are indexed by Google? Does Google not crawl the site and make a list of urls for the sake of the search purely? (Zoom search does it, for example) In theory, there could be two different type of 'crawls': one for the site search and one for the larger world, searching in the browser.

As for the settings...can you plase help me further: what exactly would you change?

Thanks for the tip, it's a good one. But they are all 100% texts.

Hi, everyone: problem solved.

Here is what I did: I created a seperate sitemap-xml and linked to all the new pdfs.

I updated the general sitemap.xml and linked to the new sitemap as well.

I (re)submitted both sitempas via the Webmaster Tools.

Within a few hours, most of pdfs got indexed and the overall quality of search has improved dramatically. Thanks for all your help.