zlacker

I can confirm. NYT shows a five-second redirect delay: "wget https://t.co/4fs609qwWt". It redirects to gov.uk immediately: "wget https://t.co/iigzas6QBx"

>>kens+(OP)
I almost didn't believe OP, because it's so comically inept and petty. But, I can also confirm in some private testing there is a deliberate delay.

>>kens+(OP)
Agree/confirmed - just recorded a number of different nytimes urls that pass through t.co, all 4.7s+. various cnbc and google articles through t.co were ~130-200ms response time from t.co specifically (not total redirect->page load).

>>kens+(OP)
Im not getting the same time delay with curl

- `time wget https://t.co/4fs609qwWt` -> `0m5.389s`

- `time curl -L https://t.co/4fs609qwWt` -> `0m1.158s`

>>kens+(OP)
It’s about 4.5 seconds for me

https://imgur.com/a/qege0O9

>>kens+(OP)
Oddly enough the delay is reduced to 1 second by using curl's useragent string (wget --user-agent='curl/8.2.1' https://t.co/4fs609qwWt)

>>jquery+V
"because it's so comically inept and petty"

This is precisely why I did believe OP. This is Elon Musk we're talking about.

>>zagreb+T1
Any DNS resolver libraries have a 4.5 second timeout? Maybe their infrastructure is just rotting.

>>lamont+Z3
It'd be weird that it's limited to specific outbound domains in that case.

>>craftk+72
Seeing this makes me wonder if it's some sort of server-side header bidding ad server gone haywire, rather than something nefarious. Why would they only delay browser agents otherwise?

>>ilikeh+I4
Browsers are generally tolerant of long TTFB. Automation, on the other hand, is sometimes quite brittle.

>>kens+(OP)
Could it just be rotting infrastructure? I.e there is some logic on most visited domains to allow ease of moderation, that logic is read heavy and is now bucking under skew.

Or even like some junior dev removed an index

>>snake_+x7
A few years ago I remember their URL shortener on android app directing somewhere that my hostfile adblocker would catch (like an analytics domain or something). This made it so first click on certain twitter links would fail, but if I clicked it again it would go successfully. Ultimately I never researched it deeply enough but my guess is they had some sort of handler that would log whether loading their analytics service failed and serve up the direct link on the second attempt.

>>minima+C4
domains load balanced across different servers in some overly complicated distributed topology with only one of them busted?

although seems unlikely it just happens to be the NYT.

>>lamont+Z3
glibc defaults to 5 sec, but the server wouldn't need to resolve the redirect domain -- that'd be the client's job. Unless it's doing so for some proprietary reason, of course.

Or did you mean failing to resolve some internal service's hostname?

>>joecoo+V7
I've experienced similar behavior on iOS, not sure if it still does that though.

>>ilikeh+I4
Perhaps their own internal tooling also relies on the t.co redirector?

>>jquery+V
Considering how common it is to deliberately break (not just with a "you log in now" modal, but not loading content, spinning forever, breaking layouts, etc) websites for non-logged-in mobile users, that's exactly how petty in imagine them to be. Twitter and Reddit do it, and Imgur comes and goes so I can't decide if for them it's deliberate or just incompetence.

>>kens+(OP)
I tested some substack.com links and there's a delay on those too.

>>joecoo+V7
Seeing that right now with NYT links. First click takes a few seconds to redirect, second click is almost instant

>>zagreb+T1
4521ms according to curl

  curl -A "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:109.0) Gecko/20100101 Firefox/117.0" -I "https://t.co/4fs609qwWt"
  x-response-time: 4521

>>ilikeh+I4
Probably a phishing/malware scan gone wrong then. NYTimes has Twitterbot in its robots.txt which might be related?

Even if it's deliberate, I don't see how people can complain. Google has outright blocked Breitbart for years. They prevent results from that domain from appearing at all unless you specifically force it with site: and apparently HN does the same. Politically motivated censorship and restricting "reach" is just how Silicon Valley rolls. Pre-Musk Twitter did freeze the New York Post's account and many other much worse things. It'd be a shame for Musk to be doing this deliberately, even though it seems unlikely. But that's the problem with creating a culture where that sort of behavior is tolerated, isn't it? One day it might be turned around on you.

>>kens+(OP)
Is there some cache going on? On my first attempt, there is a 5 second delay. When I try it second time immediately it works without the 5 second delay. But if I try again after an hour, 5 second delay again!

>>nvm0n2+4p
Does the value Breitbart adds to the internet outweigh the negatives of turning people into dangerous fascists by weaponizing misinformation? No.

Does the value added by sources like the NYT outweigh the negatives of being occasionally biased or outright wrong? Yes.

>>yohann+ys
Safari seems to be caching it for me, but I can reproduce the delay every time with curl - so long as the user agent doesn't include the string "curl".

>>spider+ht
NYT has arguably done far more damage to the world (Iraq, etc.) than Breitbart has.

>>_a9+N1
And now add browser user-agent to the curl request and watch how slow it gets.

- `time curl -A "Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Firefox/81.0" -L https://t.co/4fs609qwW` -> 4.730 total

- `time curl -L https://t.co/4fs609qwWt` -> 1.313 total

Same request, the only difference is user-agent.

>>lamont+Q8
I don't think it's a problem with rotting infrastructure. Using curl those requests are quick, unless you pass user-agent from browser. 4.7s with firefox user-agent and 1.3s without it. Using twitter link to the same NYT article.

>>spider+ht
There is no objective, public, or shared "value" at play here.

The only "values" that matter are the personal whims of whoever happens to own Twitter, or Google or Facebook.

>>kens+(OP)
Now do additional testing by adding and setting the http referer to t.co or twitter. Is it Twitter, or is it NYtimes doing this?

>>runlev+2a
> Or did you mean failing to resolve some internal service's hostname?

Yeah, something more like that where the internal service is somehow 'sharded' due to some overly complicated distributed database nonsense, and there's a DNS lookup that is failing. Of course that'd mean the DNS lookup wasn't cached, so you're taking that normal latency on every single hit, which would be terrible architecture. The curl-vs-wget performance isn't explained by that though (although that's a bit weird in and of itself, and might suggest that they had to allow that for some internal tool that they didn't want to punish).

> glibc defaults to 5 sec,

The timeout being close to 5 seconds is what made me wonder about it. Its just off though.

>>piaste+Uv1
You’re not making any sense, you’re just trying to sound contrarian.

>>menset+T71
> NYT has arguably done far more damage to the world (Iraq, etc.) than Breitbart has.

NYT may have more reach and definitely isn't neutral, but it's a far cry from the nonsense that Breitbart publishes. It's nakedly partisan.

>>craftk+72
Could this be explained by the UA derived redirect behaviour described in this other comment on the thread? >>37130982

>>spider+ht
Does the value the NYT adds to the internet outweigh the negatives of turning people into dangerous communists by weaponizing misinformation? No. Does the value added by sources like the Breitbart outweigh the negatives of being occasionally biased or outright wrong? Yes.

>>kens+(OP)
I don't see it:

  % curl -gsSIw'foo %{time_total}\n' -- https://t.co/4fs609qwWt https://t.co/iigzas6QBx | grep '^\(HTTP/\)\|\(location: \)\|\(foo \)'
  HTTP/2 301 
  location: https://nyti.ms/453cLzc
  foo 0.119295
  HTTP/2 301 
  location: https://www.gov.uk/government/news/uk-acknowledges-acts-of-genocide- committed-by-daesh-against-yazidis
  foo 0.037376

>>mzs+k42
I think Twitter, err, X, just turned off the delay now that it's getting big media attention. I could reproduce it over and over again a little earlier, but now I can't anymore: >>37138161

[Edit:] I'm still seeing it with threads.net:

  curl -v -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15' https://t.co/DzIiCFp7Ti

>>lapcat+rc2
I don't see it with your URL either:

  % curl -gsSIw'foo %{time_total}\n' https://t.co/DzIiCFp7Ti | grep '^\(HTTP/\)\|\(location: \)\|\(foo \)'
  HTTP/2 301 
  location: https://www.threads.net/@chaco_mmm_room
  foo 0.123137

Doesn't matter if I do a HTTP/2 HEAD or GET:

  % curl -gsSw'%{time_total}\n' https://t.co/DzIiCFp7Ti 
  0.121503

HTTP/1.1 also shows no delay:

  % curl -gsSw'%{time_total}\n' --http1.1 https://t.co/DzIiCFp7Ti
  0.120044

I chalk this up to rot at X/twitter that is being fixed now that it was noticed.

>>mzs+gi2
> I don't see it with your URL either

That's because you're not spoofing the User-Agent to be a browser rather than curl.

>>lapcat+Wi2
Oh that's it, thanks! In fact it it returns a 200 not 301 then:

  % curl -gsSw'%{time_total}\n' -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15' https://t.co/DzIiCFp7Ti
  <head><noscript><META http-equiv="refresh" content="0;URL=https://www.threads.net/@chaco_mmm_room"></noscript><title>https://www.threads.net/@chaco_mmm_room</title></head><script>window.opener = null; location.replace("https:\/\/www.threads.net\/@chaco_mmm_room")</script>4.690000
  % curl -gsSIw'%{time_total}\n' -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15' https://t.co/DzIiCFp7Ti
  HTTP/2 200 
  ...
  content-length: 272
  ...
  x-response-time: 4524
  ...
  
  4.660211

The delay is not there for nyti.ms (anymore) but once you use the Safari UA it's handled as 200 response:

  % curl -gsSIw'foo %{time_total}\n' -A 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/16.6 Safari/605.1.15' https://t.co/4fs609qwWt https://t.co/iigzas6QBx | grep '^\(HTTP/\)\|\(location: \)\|\(foo \)'
  HTTP/2 200 
  foo 0.126043
  HTTP/2 200 
  foo 0.037255

It really does seem that twitter is adding a 4.5s delay to some sites from web browsers. Could be malicious, could be rot...

>>djvdq+Pj1
your URLs are different.

>>wakame+0q2
Only because I copied the first one incorrectly to put it here. I haven't selected full command, so there is a missing "t" at the end of first link

>>darkso+MZ1
I can tell you all the ways in which you’re wrong, but something tells me you wont trust anything anyone says unless it confirms your biases.

>>adhesi+qc
You can add Instagram and YouTube to the list of websites that are painful to use from a mobile browser

>>kens+(OP)
They both load instantly for me.

>>XCSme+N54
It's already been reverted.

>>mzs+Wp2
The specific logic with user agents is that it happened (I think they've ended it now?) whenever the word "curl" was not in your user agent string. If the substring "curl" was contained anywhere in your user agent string, it did not have a delay. I cannot imagine how it could rot in that specific way non-maliciously.

>>piaste+Uv1
Just because this is a very difficult question doesn't mean we can throw our hands up and pretend it doesn't exist. Many things in life are very difficult and yet worth solving anyway.

>>howint+lF5
I didn't say the concept cannot exist: I said it's not at play here.

What gets a website censored, in the modern corporation-dominated Internet, is going against the interests and preferences of Big Tech owners - and nothing else. Nobody with any power is bound to look out for the public interest, however defined; ICANN is perhaps the only exception that comes to mind.

We can waste our time and attention debating over which targets were more or less deserving of censorship, based on our personal ideas of public interest. But as long as Big Tech is allowed to exist in its current form, we're like powerless peasants arguing about the decisions of kings.

>>adhesi+qc
it's no doubt intentional if the site has an app. They can scoop up significantly more data about you from their app

>>piaste+vl6
“And nothing else”

That’s not true and you know it. Don’t ignore facts man.

>>Pengui+hw1
You can do that if you want; I don't take orders.