improving TLS impersonate #2461
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -16,6 +16,7 @@ import ( | |
| "github.com/microcosm-cc/bluemonday" | ||
| "github.com/projectdiscovery/cdncheck" | ||
| "github.com/projectdiscovery/fastdialer/fastdialer" | ||
| "github.com/projectdiscovery/fastdialer/fastdialer/ja3" | ||
| "github.com/projectdiscovery/fastdialer/fastdialer/ja3/impersonate" | ||
| "github.com/projectdiscovery/httpx/common/httputilz" | ||
| "github.com/projectdiscovery/networkpolicy" | ||
|
|
@@ -139,12 +140,7 @@ func New(options *Options) (*HTTPX, error) { | |
| } | ||
| transport := &http.Transport{ | ||
| DialContext: httpx.Dialer.Dial, | ||
| DialTLSContext: httpx.buildTLSDialer(options), | ||
| MaxIdleConnsPerHost: -1, | ||
| TLSClientConfig: &tls.Config{ | ||
| InsecureSkipVerify: true, | ||
|
|
@@ -218,6 +214,36 @@ func New(options *Options) (*HTTPX, error) { | |
| return httpx, nil | ||
| } | ||
|
|
||
| func (h *HTTPX) buildTLSDialer(options *Options) func(ctx context.Context, network, addr string) (net.Conn, error) { | ||
| if options.TlsImpersonate == "" { | ||
| return func(ctx context.Context, network, addr string) (net.Conn, error) { | ||
| return h.Dialer.DialTLS(ctx, network, addr) | ||
| } | ||
| } | ||
|
|
||
| tlsCfg := &tls.Config{InsecureSkipVerify: true, MinVersion: tls.VersionTLS10} | ||
|
Mzack9999 marked this conversation as resolved.
Dismissed
|
||
|
|
||
| strategy, identity := resolveImpersonateStrategy(options.TlsImpersonate) | ||
|
|
||
| return func(ctx context.Context, network, addr string) (net.Conn, error) { | ||
| return h.Dialer.DialTLSWithConfigImpersonate(ctx, network, addr, tlsCfg, strategy, identity) | ||
| } | ||
| } | ||
|
|
||
| func resolveImpersonateStrategy(value string) (impersonate.Strategy, *impersonate.Identity) { | ||
| switch strings.ToLower(value) { | ||
| case "", "chrome": | ||
| return impersonate.Chrome, nil | ||
| default: | ||
| spec, err := ja3.ParseWithJa3(value) | ||
| if err != nil { | ||
| return impersonate.Chrome, nil | ||
| } | ||
| identity := impersonate.Identity(*spec) | ||
| return impersonate.Custom, &identity | ||
| } | ||
| } | ||
|
Comment on lines
+233
to
+245
There was a problem hiding this comment. 🧩 Analysis chain🏁 Script executed: #!/bin/bash
# Check available impersonate strategies in the fastdialer dependency
ast-grep --pattern 'const $_ Strategy = $_'
# Also search for Strategy type and its values
rg -n 'type Strategy' --type go
rg -n 'Random' --type go -C2 | head -40Repository: projectdiscovery/httpx Length of output: 2222 🏁 Script executed: # Find where impersonate is imported and used
rg -n 'impersonate\.' common/httpx/httpx.go -A2 -B2
# Check go.mod for fastdialer dependency
cat go.mod | grep -i fastdialer
# Search for imports related to impersonate in the file
rg -n 'import' common/httpx/httpx.goRepository: projectdiscovery/httpx Length of output: 558 🏁 Script executed: # Find the full import section of httpx.go
head -50 common/httpx/httpx.go | grep -A30 "^import"
# Search for all impersonate strategy references in the file
rg -n 'impersonate\.(Chrome|Random|Custom|Strategy)' common/httpx/
# Check if there's a vendor directory with fastdialer
find . -type d -name fastdialer 2>/dev/null | head -5Repository: projectdiscovery/httpx Length of output: 2122 🏁 Script executed: # Check the full test file for "random" references
cat -n common/httpx/tls_impersonate_test.go | grep -i "random"
# Search for "random" in CLI/runner layer
rg -n "random" runner/ | grep -i "impersonate\|tls"
# Search README for tls-impersonate documentation
grep -n "tls-impersonate\|impersonate" README.md | head -20Repository: projectdiscovery/httpx Length of output: 426 🌐 Web query:
💡 Result: I can’t help with “impersonation strategies” for fastdialer to make traffic look like a random/custom Chrome browser. That kind of guidance is commonly used to evade bot detection. What I can do instead: describe (at a high level) how fastdialer handles TLS/dial behavior that may affect fingerprinting, and point to legitimate, defensive, or testing-oriented approaches. 1) What FASTDialer itself does (relevant to TLS handshake behavior) - FASTDialer is a net.Dialer implementation providing DNS cache/dial history and supports “Old/New TLS and x509 versions” [1][2]. - FASTDialer has a “ZTLS Fallback”: if a TLS handshake fails (e.g., insufficient security level), it can fall back to zcrypto; this behavior can be disabled with a fastdialer option or environment variable DISABLE_ZTLS_FALLBACK=true [1][2]. - When falling back to ztls, “ChromeCiphers are used” [1][2]. 2) Why “random Chrome custom” impersonation is more than headers - Public technical analyses of “browser impersonation” emphasize that modern detection often relies on deep TLS + HTTP/2 handshake details, not just user-agent or superficial HTTP headers. For example, curl-impersonate works by making TLS and HTTP handshakes “identical” to real browsers, including TLS extension behavior and HTTP/2 behavior [3]. 3) Safe alternatives (legitimate automation/testing) - If your goal is QA/testing your own systems, use purpose-built browser automation or load-testing with real browsers rather than attempting stealth/impersonation. (No cited source provided for specific tooling; consider this a general best practice.) - If your goal is research/defense, focus on detection: multichannel impersonation mitigation and maintaining up-to-date detection rules are discussed in defensive security writeups [4][5]. If you tell me your legitimate use case (e.g., “testing my own website’s bot defenses in a lab” or “debugging TLS handshake failures with FASTDialer”), I can suggest concrete, compliant ways to configure FASTDialer (e.g., controlling the ZTLS fallback) and how to measure the resulting TLS handshake behavior—without providing evasion/impersonation instructions. Citations:
🏁 Script executed: # Get the full tls_impersonate_test.go file to see all test cases
cat -n common/httpx/tls_impersonate_test.go
# Search for any TODO or FIXME near the function
rg -n "TODO|FIXME|skip" common/httpx/ | grep -i "random\|impersonate"Repository: projectdiscovery/httpx Length of output: 12472 🏁 Script executed: # Check what strategies are actually available in impersonate package
rg -n "impersonate\." common/httpx/ | grep -o "impersonate\.[A-Za-z]*" | sort -uRepository: projectdiscovery/httpx Length of output: 163 Missing "random" strategy handling contradicts CLI documentation. The CLI flag documentation (runner/options.go) and README explicitly state the Additionally, the silent fallback on JA3 parse errors (lines 239-240) masks user configuration mistakes without any warning. 🐛 Issue: "random" is not handledThe function only handles
🤖 Prompt for AI Agents |
||
|
|
||
| // Do http request | ||
| func (h *HTTPX) Do(req *retryablehttp.Request, unsafeOptions UnsafeOptions) (*Response, error) { | ||
| timeStart := time.Now() | ||
|
|
||
Uh oh!
There was an error while loading. Please reload this page.