Setting A Custom User Agent in Screaming Frog

[Malika1]

Hi Thomas,

Thanks for responding, much appreciated!

Does that mean, if I type in something like -

HTTP request user agent -

Crawler access V2

&

Robots user agent

Crawler access V2

This will work too?

[BlueprintMarketing]

No, you Cannot just put anything in there. The site has to recognize it and ask why you are doing this?

I have listed how to build and already built in addition to what your browser will create by using useragentstring.com

Must be formatted correctly and have it work with a header it is not as easy as it sometimes seems but not that hard either.

You can make & use this to make your own from your Mac or PC

http://www.useragentstring.com/

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2747.0 Safari/537.36

how to build a user agent

• https://developer.mozilla.org/en-US/docs/Web/HTTP/Gecko_user_agent_string_reference
• https://developer.mozilla.org/en-US/docs/Setting_HTTP_request_headers
• https://msdn.microsoft.com/en-us/library/ms537503(VS.85).aspx

Lists of user agents

https://support.google.com/webmasters/answer/1061943?hl=en

http://www.user-agents.org/

https://msdn.microsoft.com/en-us/library/ms537503(v=vs.85).aspx

[BlueprintMarketing]

if you formatted it correctly see below

User-Agent = product *( RWS ( product / comment ) )

and it was received by your headers yes you could fill in the blanks and test it.

https://mobiforge.com/research-analysis/webviews-and-user-agent-strings

http://mobiforge.com/news-comment/standards-and-browser-compatibility

[BlueprintMarketing]

this gives you a very clear breakdown of  user agents and their  set of syntax rules.  The following is valid example of user-agent that is full of special characters,

read this please http://www.bizcoder.com/the-much-maligned-user-agent-header

 

user-agent: foo&bar-product!/1.0a$*+ (a;comment,full=of/delimiters

references but you want to pay attention to the first URL

http://www.howtogeek.com/113439/how-to-change-your-browsers-user-agent-without-installing-any-extensions/

https://developer.mozilla.org/en-US/docs/Web/HTTP/Gecko_user_agent_string_reference

| Mozilla/5.0 (X11; Linux i686; rv:10.0) Gecko/20100101 Firefox/10.0 |

http://stackoverflow.com/questions/15069533/http-request-header-useragent-variable

[BlueprintMarketing]

please let me know if I did not answer the question or you have any other questions

[Malika1]

Hi Thomas,

That's a lot of useful information there. I will have a go on it and let you know how it went.

Thanks heaps!

[BlueprintMarketing]

happy I could be of help let me know if there's any issue and I will try to be of help with it. All the best

[MattRoney]

Hi Malika! How'd it go? Did everything work out?

[Malika1]

Hi Matt,

I havent had a luck with this one yet.

[Malika1]

I think I want something that is dumbed down to my level for me to understand. The above tutorials are great but not being a full time coder, I get lost while reading those.

[Malika1]

Hi Thomas,

would you have a simpler tutorial for me to understand? I am struggling a bit.

Thanks heaps in advance

[BlueprintMarketing]

Hi Malika,

Think about screaming frog and what it has to detect in order to do that correctly it needs the correct  user agent syntax for it will not be able to make a crawl that would satisfy people.

Using a proper syntax for a user agent is essential and I have tried to be non-technical in this explanation I hope it works.

the reason screaming frog needs the user agent because the user-agent was added to HTTP to help web application developers deliver a better user experience.  By respecting the syntax and semantics of the header, we make it easier and faster for header parsers to extract useful information from the headers that we can then act on.

Browser vendors are motivated to make web sites work no matter what specification violations are made.  When the developers building web applications don’t care about following the rules, the browser vendors work to accommodate that.  It is only by us application developers developing a healthy respect

When the developers building web applications don’t care about following the rules, the browser vendors work to accommodate that.  It is only by us application developers developing a healthy respect

It is only by us application developers developing a healthy respect for the standards of the web, that the browser vendors will be able to start tightening up their codebase knowing that they don’t need to account for non-conformances.

For client libraries that do not enforce the syntax rules, you run the risk of using invalid characters that many server side frameworks will not detect.  It is possible that only certain users, in particular, environments would identify the syntax violation.  This can lead to difficult to track down bugs.

I hope this is a good explanation I've tried to keep it very to the point.

Respectfully,

Thomas

[BlueprintMarketing]

Setting a custom user agent determines things like HTTP/2 so there can be a big difference if you change it to something that might not take advantage of something like HTTP/2

Apparently, it is coming  to Pingdom very soon just like it is to Googlebot

http://royal.pingdom.com/2015/06/11/http2-new-protocol/

This Is an excellent example of a user agent's ability to modify the way your site is crawled as well as how efficient it is.

https://www.keycdn.com/blog/https-performance-overhead/

It is important to note that we didn’t use Pingdom in any of our tests because they use Chrome 39, which doesn’t support the new HTTP/2 protocol. HTTP/2 in Chrome isn’t supported until Chrome 43. You can tell this by looking at the User-Agent in the request headers of your test results.

Pingdom user-agent

Note: WebPageTest uses Chrome 47 which does support HTTP/2.

Hope that clears things up,

Tom

ujNvaad.png t49XhGi.png