• cmnybo@discuss.tchncs.de
    link
    fedilink
    English
    arrow-up
    0
    ·
    4 months ago

    Some Lemmy instances disallow indexing in robots.txt, however indexers can choose to ignore that and actually blocking them takes a lot more effort.