Crawl de page par Google Bot via une recherche web

[Lynard]

Bonjour,

Je suis à la recherche d'informations concernant le crawl de google Bot (desktop & mobile).
Depuis 4 jours, je remarque que 2 pages du site sont crawlées par Gbot via une url de recherche web .
exemple :
https://www.google.fr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CFw ... YU6FSNz4kw

Ce sont des pages de catégorie de produit.
Il lit toute la page de la fin du header et s'arrete avant le footer.
Donc js/css et toutes les images.

Il les crawl sur le desktop et Iphone+Android.

Est ce que quelqu'un a déjà eu cela ?
Merci

 

[loubet]

pas très clair.
possible d'avoir des extraits de log si c'est la source ?

 

[Lynard]

Merci Loubet

Je suppose que tu veux voir les ip et user agent ?

 

[Lynard]

Voici un exemple
Il lit ensuite toutes les images des produits (google-image...)

66.249.88.165 - - [05/Aug/2015:07:07:30 +0200] "GET /mapage.html HTTP/1.1" 200 27893 "https://www.google.fr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=4&ved=0CFQQFjAB&url=https%3A%2F%2Fwww.mondomaine.com%2Fmapage.html&ei=EZrBVbIZq7SrA-WkAQ&usg=AFQjCNH3cIhW2Y4cuWkUIr5y6ax7Y4sj6g" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
66.249.88.165 - - [05/Aug/2015:07:07:30 +0200] "GET /components/com_products/products.css HTTP/1.1" 200 40900 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
66.249.93.137 - - [05/Aug/2015:07:07:30 +0200] "GET /mapage.html HTTP/1.1" 200 32842 "https://www.google.fr/url?sa=t&rct=j&q=&esrc=s&source=web&cd=1&ved=0CFAQFjAB&url=https%3A%2F%2Fwww.mondomaine.com%2Fmapage.html&ei=EZrBVYDZJ73D5APsJQ&usg=AFQjCNH3cIhW2Y4cuWkUIr5y6ax7Y4sj6g" "Mozilla/5.0 (Linux; Android 4.2.2; en-us; SAMSUNG SGH-M919 Build/JDQ39) AppleWebKit/535.19 (KHTML, like Gecko) Version/1.0 Chrome/18.0.1025.308 Mobile Safari/535.19"
66.249.88.165 - - [05/Aug/2015:07:07:30 +0200] "GET /plugins/content/allvideos/includes/players/quicktimeplayer/AC_QuickTime.js HTTP/1.1" 200 8928 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
66.249.88.155 - - [05/Aug/2015:07:07:30 +0200] "GET /plugins/content/allvideos/includes/players/wmvplayer/silverlight.js HTTP/1.1" 200 21473 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
66.249.88.155 - - [05/Aug/2015:07:07:30 +0200] "GET /plugins/content/allvideos/tmpl/css/template.css HTTP/1.1" 200 6202 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
66.249.88.155 - - [05/Aug/2015:07:07:30 +0200] "GET /plugins/content/allvideos/includes/allvideos.js HTTP/1.1" 200 10316 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
66.249.88.160 - - [05/Aug/2015:07:07:30 +0200] "GET /plugins/content/allvideos/includes/players/wmvplayer/wmvplayer.js HTTP/1.1" 200 27582 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
66.249.88.165 - - [05/Aug/2015:07:07:30 +0200] "GET /templates/mobile_iphone/css/mj_iphone.css HTTP/1.1" 200 12072 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
66.249.88.155 - - [05/Aug/2015:07:07:30 +0200] "GET /media/lengow.js HTTP/1.1" 200 8756 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
66.249.88.155 - - [05/Aug/2015:07:07:30 +0200] "GET /media/system/js/mootools.js HTTP/1.1" 200 78081 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
66.249.83.181 - - [05/Aug/2015:07:07:30 +0200] "GET /plugins/content/allvideos/includes/players/wmvplayer/wmvplayer.js HTTP/1.1" 200 27582 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (Linux; Android 4.2.2; en-us; SAMSUNG SGH-M919 Build/JDQ39) AppleWebKit/535.19 (KHTML, like Gecko) Version/1.0 Chrome/18.0.1025.308 Mobile Safari/535.19"
66.249.83.184 - - [05/Aug/2015:07:07:30 +0200] "GET /plugins/content/allvideos/tmpl/css/template.css HTTP/1.1" 200 6202 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (Linux; Android 4.2.2; en-us; SAMSUNG SGH-M919 Build/JDQ39) AppleWebKit/535.19 (KHTML, like Gecko) Version/1.0 Chrome/18.0.1025.308 Mobile Safari/535.19"
66.249.83.184 - - [05/Aug/2015:07:07:30 +0200] "GET /plugins/content/allvideos/includes/players/wmvplayer/silverlight.js HTTP/1.1" 200 21473 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (Linux; Android 4.2.2; en-us; SAMSUNG SGH-M919 Build/JDQ39) AppleWebKit/535.19 (KHTML, like Gecko) Version/1.0 Chrome/18.0.1025.308 Mobile Safari/535.19"
66.249.88.160 - - [05/Aug/2015:07:07:31 +0200] "GET /templates/mobile_iphone/jqtouch-src/jqtouch/jqtouch.min.css HTTP/1.1" 200 22264 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
66.249.83.178 - - [05/Aug/2015:07:07:30 +0200] "GET /components/com_products/products.css HTTP/1.1" 200 44045 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (Linux; Android 4.2.2; en-us; SAMSUNG SGH-M919 Build/JDQ39) AppleWebKit/535.19 (KHTML, like Gecko) Version/1.0 Chrome/18.0.1025.308 Mobile Safari/535.19"
66.249.83.184 - - [05/Aug/2015:07:07:30 +0200] "GET /media/system/js/mootools.js HTTP/1.1" 200 78081 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (Linux; Android 4.2.2; en-us; SAMSUNG SGH-M919 Build/JDQ39) AppleWebKit/535.19 (KHTML, like Gecko) Version/1.0 Chrome/18.0.1025.308 Mobile Safari/535.19"
66.249.83.184 - - [05/Aug/2015:07:07:31 +0200] "GET /templates/mobile_smartphone/resources/styles/baseStyles.css HTTP/1.1" 200 23487 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (Linux; Android 4.2.2; en-us; SAMSUNG SGH-M919 Build/JDQ39) AppleWebKit/535.19 (KHTML, like Gecko) Version/1.0 Chrome/18.0.1025.308 Mobile Safari/535.19"
66.249.88.160 - - [05/Aug/2015:07:07:31 +0200] "GET /templates/mobile_iphone/jqtouch-src/jqtouch/img/backButton.png HTTP/1.1" 200 1147 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
66.249.88.165 - - [05/Aug/2015:07:07:31 +0200] "GET /templates/mobile_iphone/jqtouch-src/jqtouch/img/toolbar.png HTTP/1.1" 200 532 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
66.249.88.160 - - [05/Aug/2015:07:07:31 +0200] "GET /templates/mobile_iphone/jqtouch-src/themes/apple/img/pinstripes.png HTTP/1.1" 200 481 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"
66.249.83.184 - - [05/Aug/2015:07:07:31 +0200] "GET /templates/mobile_smartphone/css/mj_xhtml.css HTTP/1.1" 200 5056 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (Linux; Android 4.2.2; en-us; SAMSUNG SGH-M919 Build/JDQ39) AppleWebKit/535.19 (KHTML, like Gecko) Version/1.0 Chrome/18.0.1025.308 Mobile Safari/535.19"
66.249.83.181 - - [05/Aug/2015:07:07:31 +0200] "GET /templates/mobile_smartphone/css/custom.css HTTP/1.1" 200 5349 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (Linux; Android 4.2.2; en-us; SAMSUNG SGH-M919 Build/JDQ39) AppleWebKit/535.19 (KHTML, like Gecko) Version/1.0 Chrome/18.0.1025.308 Mobile Safari/535.19"
66.249.88.165 - - [05/Aug/2015:07:07:31 +0200] "GET /templates/mobile_iphone/jqtouch-src/jqtouch/img/toolButton.png HTTP/1.1" 200 885 "https://www.domaine.com/mapage.html" "Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A403 Safari/8536.25"

 

[loubet]

à part les ip qui semblent être des ip de google, rien n'indique que c'est un robot, cela peut être 2 personnes travaillant pour google qui sont allés sur ton site suite à une recherche sur google.

 

[Lynard]

Oui, possible, mais ils scannent 2 pages par jours, mais pas les mêmes, on dirait qu'ils scannent le site par thématique.

Ce serait un jour, bah j'aurai rien dit, mais là :
Depuis 4 jours ils prennent le menu et scanne chaque page du menu.
Exemple mon menu c'est maison/Jardin/Garage/Maison de vacances
ils ont scanné maison, jardin garage et maison de vacances, et cet apm, ils scannent les sous catégories.
En plus :
- ils ne regardent pas le header (donc logo+recherche+liens de contact),
- ils commencent à la balise nav (le menu),
- ils s'arretent avant la balise footer.(y a un menu et des images)