Sunshine (she/her)@piefed.ca to Fedibridge@lemmy.dbzer0.comEnglish · edit-22 months agoReddit will block the Internet Archivewww.theverge.comexternal-linkmessage-square7linkfedilinkarrow-up196arrow-down11file-textcross-posted to: reddit@lemmy.worldreddit@lemmy.mltechnology@hexbear.nettechnology@lemmygrad.mlredditwasfun@lemmy.worldnews@lemmy.worldtechnology@lemmy.worldtechnology@piefed.socialtechnology@lemmy.zip
arrow-up195arrow-down1external-linkReddit will block the Internet Archivewww.theverge.comSunshine (she/her)@piefed.ca to Fedibridge@lemmy.dbzer0.comEnglish · edit-22 months agomessage-square7linkfedilinkfile-textcross-posted to: reddit@lemmy.worldreddit@lemmy.mltechnology@hexbear.nettechnology@lemmygrad.mlredditwasfun@lemmy.worldnews@lemmy.worldtechnology@lemmy.worldtechnology@piefed.socialtechnology@lemmy.zip
minus-squareRiskable@programming.devlinkfedilinkEnglisharrow-up21·2 months agoSo let me get this straight: Instead of wasting Reddit’s bandwidth, AI companies have been scraping the wayback machine. Because of this, Reddit is going to block the wayback machine from crawling it’s site which will ensure the AI companies crawl Reddit, directly. …Because if you think they’re suddenly going to stop crawling reddit—robots.txt be damned—you’re dreaming.
So let me get this straight: Instead of wasting Reddit’s bandwidth, AI companies have been scraping the wayback machine.
Because of this, Reddit is going to block the wayback machine from crawling it’s site which will ensure the AI companies crawl Reddit, directly.
…Because if you think they’re suddenly going to stop crawling reddit—robots.txt be damned—you’re dreaming.