• Home
  • Linux
  • UNIX
  • Website Development
  • SAP Basis
  • SAP Solution Manager
  • SAP PI/XI
  • SLD

Файл robots.txt для сайта WordPress

Публикую используемый мной robots.txt с запретом работы некоторым поисковым ботам.

User-agent: Exabot
Disallow: /
User-agent: IDBot
Disallow: /
User-agent: Wotbox
Disallow: /
User-agent: Baiduspider
Disallow: /
User-agent: Baiduspider-image
Disallow: /
User-agent: Baiduspider-video
Disallow: /
User-agent: Baiduspider-news
Disallow: /
User-agent: Baiduspider-favo
Disallow: /
User-agent: Baiduspider-cpro
Disallow: /
User-agent: Baiduspider-ads
Disallow: /
User-agent: Baidu
Disallow: /
User-agent: MJ12bot
Disallow: /
User-agent: AhrefsBot
Disallow: /
User-agent: dotbot
Disallow: /
User-agent: Riddler
Disallow: /
User-agent: SemrushBot
Disallow: /
User-agent: LinkpadBot
Disallow: /
User-agent: BLEXBot
Disallow: /
User-agent: FlipboardProxy
Disallow: /
User-agent: aiHitBot
Disallow: /
User-agent: trovitBot
Disallow: /
User-agent: *
Disallow: /cgi-bin
Disallow: /wp-admin
Disallow: /wp-includes
Disallow: /wp-content/plugins
Disallow: /wp-content/cache
Disallow: /wp-content/themes
Disallow: /trackback
Disallow: */trackback
Disallow: */*/trackback
Disallow: */*/feed/*/
Disallow: */feed
Disallow: /*?*
Crawl-delay: 10
Host: <хост.нэйм>
Sitemap: http://<сайт.нэйм>/wp-sitemap.xml
« Linux Mint 19.3 зависает намертво! – Решение
Linux Mint 20: Не работают сочетания горячих клавиш с Win(Super L) на русской раскладке »
Еще по теме:
  • Convert the first letter of each line to uppercase
  • Восстановление синхронизации сообщений в Viber для Linux
  • Secure Store AS Java Error
This entry was posted in Website Development and tagged robots.txt, wordpress.
Оставить Ответ
Click here to cancel reply.

  • Курсы валют

    RUB/USD - 63,5428

    RUB/EUR - 66,3644

  • Tags
    ABAP AIX AI_RUNTIME_JCOSERVER Archlinux awk BPE brtools CHARM client copy cp CUA dev_w logs download basket download or upload files drivers EarlyWatch Alert epson ESR file system find firefox FTP GRUB HA systems hot keys HTTPS ibus icm Internet Explorer Java kernel keyboard language Linux Linux Mint logs Managed System Configuration mount mpv NWA Open Source ORACLE pacman PI Cache READ_USER RFC authorization robots.txt roles RWB RZ20 rz21 SAPCAR sapgui saposcol SAP Web Dispatcher SAP по русски Secure store Service Desk shared memory Shutter SLD sm21 sm59 SMD agent SOLAR_PROJECT_ADMIN spam ST03 STAD STMS STZAC Super L SUSE swap terminal text editors timezone tp Ubuntu Unix client Upgrade USB UTF-8 viber waterfox wordpress xterm YouTube youtube-dl zoom Альтернатива Windows Горячие клавиши Зависания Клавиша Win Кодировка Отмена перехода на зимнее время Температура процессора загрузка Excel минимальные требования к аппаратному обеспечению
  • Posts with most comments
    • SAP Solution Manager не работает отправка сообщения в SAP (4)
    • Error: Remote access to the NetWeaver Administrator is forbidden. See SAP Note 1451753 for details. (4)
    • AI_RUNTIME_JCOSERVER – Timeout while waiting for a connection to be established (2)
    • Script for UnSAPCAR Multiple SAR files in single shot (2)
    • Problem with stop/start saposcol (2)

© 2011 - 2022 30 years in IT. All Rights Reserved.