tekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 6 days agorobots.txt is a suggestionusenet.lolimagemessage-square23fedilinkarrow-up11arrow-down10
arrow-up11arrow-down1imagerobots.txt is a suggestionusenet.loltekeous@usenet.lol to Lemmy Shitpost@lemmy.world · 6 days agomessage-square23fedilink
minus-squareekZepp@lemmy.worldlinkfedilinkEnglisharrow-up0·6 days agoAll this robot.txt stuff “perplex” me.
All this robot.txt stuff “perplex” me.