Lee Duna@lemmy.nz to Technology@lemmy.worldEnglish · 1 year agoEdge 118: updated on-page search may send data to Microsoftwww.ghacks.netexternal-linkmessage-square35fedilinkarrow-up1249arrow-down15
arrow-up1244arrow-down1external-linkEdge 118: updated on-page search may send data to Microsoftwww.ghacks.netLee Duna@lemmy.nz to Technology@lemmy.worldEnglish · 1 year agomessage-square35fedilink
minus-squareJuki@lemmy.worldlinkfedilinkEnglisharrow-up14·1 year agoHow much of that is a workaround to feed client rendered webpages into LLMs and bypass robots.txt etc
minus-squareBuddahriffic@lemmy.worldlinkfedilinkEnglisharrow-up2·1 year agoI mean, if you want to go around what the site wants you to do, you can just ignore robots.txt. Or use it to find the juicy stuff the site would rather you didn’t.
How much of that is a workaround to feed client rendered webpages into LLMs and bypass robots.txt etc
I mean, if you want to go around what the site wants you to do, you can just ignore robots.txt. Or use it to find the juicy stuff the site would rather you didn’t.