Description
When using httpx v1.7.1, some websites respond with 403 Forbidden getting us false results, but using curl with the same URL returns a normal response 200 OK with full HTML content.
##for example:
httpx -u https://hola.example.com -no-decode -debug
While with curl:
curl -k -L https://hola.example.com -i
You get:
HTTP/1.1 200 OK
... (headers)
<html> ... full page content ... </html>
Actual behavior
httpx consistently gets 403 (Access Denied) on this site and several similar Akamai-protected domains
Additional context
-
Happens even with -H "User-Agent: ...", -random-agent, or -H "Accept-Encoding: identity".
-
May be related to subtle differences in TLS fingerprinting / HTTP2 vs HTTP1 negotiation that Akamai WAF detects.
-
Sites behind Akamai often do bot detection using TLS fingerprints. curl has a different fingerprint than httpx.
Description
When using
httpx v1.7.1, some websites respond with403 Forbiddengetting us false results, but usingcurlwith the same URL returns a normal response200 OKwith full HTML content.##for example:
httpx -u https://hola.example.com -no-decode -debugWhile with
curl:curl -k -L https://hola.example.com -iYou get:
HTTP/1.1 200 OK
... (headers)
<html> ... full page content ... </html>
Actual behavior
httpxconsistently gets 403 (Access Denied) on this site and several similar Akamai-protected domainsAdditional context
Happens even with
-H "User-Agent: ...",-random-agent, or-H "Accept-Encoding: identity".May be related to subtle differences in TLS fingerprinting / HTTP2 vs HTTP1 negotiation that Akamai WAF detects.
Sites behind Akamai often do bot detection using TLS fingerprints.
curlhas a different fingerprint thanhttpx.