zlacker

>>quicks+(OP)
A while back I changed my search engine's crawl data to be ZSTD compressed JSON. It's a bit finnicky to work with, but I'm beginning to realize just how powerful this is.

Could literally just do

  find -name \*.zstd -exec zstdcat {} \; |
    jq 'first(select(.doc|select(.!=null)|.[].headers|select(.!=null)|test("[xX]-[aA]dblock-[kK]ey")))'

and it spewed out samples of domains with a header like X-Adblock-Key. (I'm not great with JQ, so there's probably a better way of doing this, but this unga bunga approach works too)

Specifically, today I did some research on a few tags and headers supposedly associated with "Acceptable Ads" (a standard for showing ads through complicit adblockers), and ended up with a fairly reliable fingerprint for a network of domain squatters that have been a nuisance in my search engine database. Turns out they're basically the only ones that use the headers and tags I was looking at, so now I'm onto their IP-ranges as well.

>>margin+37
I don't have much context about your technical requirements but can I ask why JSON instead of a more indexable format?

>>higero+bk
It's a tradeoff between ease of writing, and ease of reading for indexing, and freeform analytical usecases like this. JSON caters to all fairly well.

It's one file per domain, so looking at specific urls is no prob with this setup.