PetalBot is an automatic program of the Petal search engine. The function of PetalBot is to access both PC and mobile websites and establish an index database which enables users to search the content of your site in Petal search engine and present content recommendations for the user in Huawei Assistant and AI Search services, both services are powered by Petal Search enigine. You can identify crawling from Petal by analyzing the User-agent field.
1.2 What Is the User-Agent of PetalBot
Product name
User-Agent
Full User-Agent
PetalBot(PC)
PetalBot
Mozilla/5.0 (compatible;PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
PetalBot(mobile)
PetalBot
Mozilla/5.0 (Linux; Android 7.0;) AppleWebKit/537.36 (KHTML, like Gecko) Mobile Safari/537.36 (compatible; PetalBot;+https://webmaster.petalsearch.com/site/petalbot)
1.3 How to Block PetalBot from Visiting Your Site
PetalBot complies with the Internet robots protocol. You can use the robots.txt file to completely prevent PetalBot from accessing your website, or to prevent PetalBot from accessing some files on your website.
Note: Banning PetalBot from accessing your site will make the pages on your site and all search engine services provided by Petal unsearchable in the Petal search engine.
setup recommendations
You can set different crawling rules according to different user-agents of each product, and you can directly prevent the crawling of PetalBot. The following robots can prevent Petal crawling or conditional allow:
User-agent: PetalBot
Disallow: /
User-agent: PetalBot
Allow: /w/api/
Disallow: /trap/
1.4 How Much Pressure Does PetalBot Bring to a Web Server
In order to have a better retrieval results for the target resources, PetalBot needs to maintain a certain amount of crawling of your website. We try not to bring an unreasonable burden on the website, and we will make adjustments based on comprehensive factors such as server capacity, website quality, and website updates. If there is any unreasonable behavior of PetalBot’s visiting, please send your concerns to petalbot@huawei.com.
1.5 How to Judge PetalBot Crawling
You can verify if a web crawler accessing your server really is PetalBot.
To verify PetalBot as the caller:
1. Run a reverse DNS lookup on the accessing IP address from your logs, using the host command.
2. Verify that the domain name is in aspiegel.com.
3. Run a forward DNS lookup on the domain name retrieved in step 1 using the host command on the retrieved domain name. Verify that it is the same as the original accessing IP address from your logs.
Example:
$ host 114.119.128.10
10.128.119.114.in-addr.arpa domain name pointer petalbot-114-119-128-10.petalsearch.com
$ host petalbot-114-119-128-10.petalsearch.com
petalbot-114-119-128-10.petalsearch.com has address 114.119.128.10
1.6 Why My Website Can Still Be Found in Petal When It Has Been Already Added to robots.txt
It takes time to update data in the search engine index database. Although PetalBot has stopped accessing the pages on your site, it may take several months to clear the page index information that has already been established in the Petal search engine database. Besides, please check if your robots configuration is correct.If your rejection is very urgent, please make your request to petalbot@huawei.com.
