httpx logo

What is httpx?

httpx is a fast and multi-purpose HTTP toolkit made by Project Discovery that allows running multiple probes using the retryablehttp library. It is designed to maintain result reliability with an increased number of threads. httpx can be used to obtain web server information, such as headers, download pages and take screenshots of targets. httpx is perfect for validating http/https servers for large scopes on bugbounty programs or performing asset management / penetration testing.

httpx Installation

How to install httpx:

go install -v github.com/projectdiscovery/httpx/cmd/httpx@latest

httpx Project Discovery Tutorial

After installation the following simple httpx tutorial will get you up and scanning web servers:

cat targets.txt | httpx

For more options and real world httpx examples see the bottom of this document.

httpx Supported Probes

The type of data httpx can obtain from target web servers:

Probes Default check Probes Default check
URL true IP true
Title true CNAME true
Status Code true Raw HTTP false
Content Length true HTTP2 false
TLS Certificate true HTTP Pipeline false
CSP Header true Virtual host false
Line Count true Word Count true
Location Header true CDN false
Web Server true Paths false
Web Socket true Ports false
Response Time true Request Method true
Favicon Hash false Probe Status false
Body Hash true Header Hash true
Redirect chain false URL Scheme true
JARM Hash false ASN false

TIP: Take Screenshots with httpx

To take screenshots with httpx use -screenshot or -ss

httpx Cheat Sheet

httpx Input Commands

COMMAND DESCRIPTION

-l, -list string

input file containing list of hosts to process

-rr, -request string

file containing raw request

-u, -target string[]

input target host(s) to probe

httpx Probe Commands

COMMAND DESCRIPTION

-sc, -status-code

display response status-code

-cl, -content-length

display response content-length

-ct, -content-type

display response content-type

-location

display response redirect location

-favicon

display mmh3 hash for '/favicon.ico' file

-hash string

display response body hash (supported: md5,mmh3,simhash,sha1,sha256,sha512)

-jarm

display jarm fingerprint hash

-rt, -response-time

display response time

-lc, -line-count

display response body line count

-wc, -word-count

display response body word count

-title

display page title

-bp, -body-preview

display first N characters of response body (default 100)

-server, -web-server

display server name

-td, -tech-detect

display technology in use based on wappalyzer dataset

-method

display http request method

-websocket

display server using websocket

-ip

display host ip

-cname

display host cname

-asn

display host asn information

-cdn

display cdn/waf in use (default true)

-probe

display probe status

httpx Headless Options

COMMAND DESCRIPTION

-ss, -screenshot

enable saving screenshot of the page using headless browser

-system-chrome

enable using local installed chrome for screenshot

-esb, -exclude-screenshot-bytes

enable excluding screenshot bytes from json output

-ehb, -exclude-headless-body

enable excluding headless header from json output

httpx Match in Response

Allows httpx to match something in the server response header / body / http response code or url etc.

COMMAND DESCRIPTION

-mc, -match-code string

match response with specified status code (-mc 200,302)

-ml, -match-length string

match response with specified content length (-ml 100,102)

-mlc, -match-line-count string

match response body with specified line count (-mlc 423,532)

-mwc, -match-word-count string

match response body with specified word count (-mwc 43,55)

-mfc, -match-favicon string[]

match response with specified favicon hash (-mfc 1494302000)

-ms, -match-string string

match response with specified string (-ms admin)

-mr, -match-regex string

match response with specified regex (-mr admin)

-mcdn, -match-cdn string[]

match host with specified cdn provider (cloudfront, fastly, google, leaseweb, stackpath)

-mrt, -match-response-time string

match response with specified response time in seconds (-mrt '< 1')

-mdc, -match-condition string

match response with dsl expression condition

httpx Extract Regex Strings

Allows httpx to extract regex strings from the reponse.

COMMAND DESCRIPTION

-er, -extract-regex string[]

display response content with matched regex

-ep, -extract-preset string[]

display response content matched by a pre-defined regex (mail, url, ipv4)

httpx Filters

Filter by response code, length, server version, error page, url etc

COMMAND DESCRIPTION

-fc, -filter-code string

filter response with specified status code (-fc 403,401)

-fep, -filter-error-page

filter response with ML based error page detection

-fl, -filter-length string

filter response with specified content length (-fl 23,33)

-flc, -filter-line-count string

filter response body with specified line count (-flc 423,532)

-fwc, -filter-word-count string

filter response body with specified word count (-fwc 423,532)

-ffc, -filter-favicon string[]

filter response with specified favicon hash (-ffc 1494302000)

-fs, -filter-string string

filter response with specified string (-fs admin)

-fe, -filter-regex string

filter response with specified regex (-fe admin)

-fcdn, -filter-cdn string[]

filter host with specified cdn provider (cloudfront, fastly, google, leaseweb, stackpath)

-frt, -filter-response-time string

filter response with specified response time in seconds (-frt '> 1')

-fdc, -filter-condition string

filter response with dsl expression condition

-strip

strips all tags in response. supported formats: html,xml (default html)

httpx Rate Limiting

Limit the number of requests httpx can make per second / per minute and configure the number of threads.

COMMAND DESCRIPTION

-t, -threads int

number of threads to use (default 50)

-rl, -rate-limit int

maximum requests to send per second (default 150)

-rlm, -rate-limit-minute int

maximum number of requests to send per minute

Misc httpx Commands

COMMAND DESCRIPTION

-pa, -probe-all-ips

probe all the ips associated with same host

-p, -ports string[]

ports to probe (nmap syntax: eg http:1,2-10,11,https:80)

-path string

path or list of paths to probe (comma-separated, file)

-tls-probe

send http probes on the extracted TLS domains (dns_name)

-csp-probe

send http probes on the extracted CSP domains

-tls-grab

perform TLS(SSL) data grabbing

-pipeline

probe and display server supporting HTTP1.1 pipeline

-http2

probe and display server supporting HTTP2

-vhost

probe and display server supporting VHOST

-ldv, -list-dsl-variables

list json output field keys name that support dsl matcher/filter

httpx Update

How to update httpx + how to disable auto update.

COMMAND DESCRIPTION

-up, -update</code>

update httpx to latest version

-duc, -disable-update-check

disable automatic httpx update check

httpx File Output

httpx output file options.

COMMAND DESCRIPTION

-o, -output string

file to write output results

-oa, -output-all

filename to write output results in all formats

-sr, -store-response

store http response to output directory

-srd, -store-response-dir string

store http response to custom directory

-csv

store output in csv format

-csvo, -csv-output-encoding string

define output encoding

-j, -json

store output in JSONL(ines) format

-irh, -include-response-header

include http response (headers) in JSON output (-json only)

-irr, -include-response

include http request/response (headers + body) in JSON output (-json only)

-irrb, -include-response-base64

include base64 encoded http request/response in JSON output (-json only)

-include-chain

include redirect http chain in JSON output (-json only)

-store-chain

include http redirect chain in responses (-sr only)

-svrc, -store-vision-recon-cluster

include visual recon clusters (-ss and -sr only)

httpx Config Options

COMMAND DESCRIPTION

-config string

path to the httpx configuration file (default $HOME/.config/httpx/config.yaml)

-r, -resolvers string[]

list of custom resolver (file or comma separated)

-allow string[]

allowed list of IP/CIDR's to process (file or comma separated)

-deny string[]

denied list of IP/CIDR's to process (file or comma separated)

-sni, -sni-name string

custom TLS SNI name

-random-agent

enable Random User-Agent to use (default true)

-H, -header string[]

custom http headers to send with request

-http-proxy, -proxy string

http proxy to use (eg http://127.0.0.1:8080)

-unsafe

send raw requests skipping golang normalization

-resume

resume scan using resume.cfg

-fr, -follow-redirects

follow http redirects

-maxr, -max-redirects int

max number of redirects to follow per host (default 10)

-fhr, -follow-host-redirects

follow redirects on the same host

-rhsts, -respect-hsts

respect HSTS response headers for redirect requests

-vhost-input

get a list of vhosts as input

-x string

request methods to probe, use 'all' to probe all HTTP methods

-body string

post body to include in http request

-s, -stream

stream mode - start elaborating input targets without sorting

-sd, -skip-dedupe

disable dedupe input items (only used with stream mode)

-ldp, -leave-default-ports

leave default http/https ports in host header (eg. http://host:80 - https://host:443)

-ztls

use ztls library with autofallback to standard one for tls13

-no-decode

avoid decoding body

-tlsi, -tls-impersonate

enable experimental client hello (ja3) tls randomization

-no-stdin

Disable Stdin processing

httpx Debug Options

COMMAND DESCRIPTION

-health-check, -hc

run diagnostic check up

-debug

display request/response content in cli

-debug-req

display request content in cli

-debug-resp

display response content in cli

-version

display httpx version

-stats

display scan statistic

-profile-mem string

optional httpx memory profile dump file

-silent

silent mode

-v, -verbose

verbose mode

-si, -stats-interval int

number of seconds to wait between showing a statistics update (default: 5)

-nc, -no-color

disable colors in cli output

Optimizations

Improve the performance of httpx tune the settings to the target environment.

COMMAND DESCRIPTION

-nf, -no-fallback

display both probed protocol (HTTPS and HTTP)

-nfs, -no-fallback-scheme

probe with protocol scheme specified in input

-maxhr, -max-host-error int

max error count per host before skipping remaining path/s (default 30)

-ec, -exclude-cdn

skip full port scans for CDN/WAF (only checks for 80,443)

-eph, -exclude-private-hosts

skip any hosts which have a private ip address

-retries int

number of retries

-timeout int

timeout in seconds (default 10)

-delay value

duration between each http request (eg: 200ms, 1s) (default -1ns)

-rsts, -response-size-to-save int

max response size to save in bytes (default 2147483647)

-rstr, -response-size-to-read int

max response size to read in bytes (default 2147483647)

Real World httpx Examples

DNSX to httpx

Run domains through dnsx to confirm resolution, then through httpx to confirm a 200 response from the webserver:

dnsx -d roots.txt -w <key,words> | httpx -sc -mc 200

httpx Follow Redirects

For httpx to follow redirects use:

httpx -follow-redirects

httpx Screenshot

Take a screenshot of targets that return 200 response:

httpx -sc -mc 200 -ss

Basic Recon

httpx -t 200 -random-agent -nc -silent -timeout 8 -sc -server -title -o httpx.out

Conclusion

We hope this httpx cheat sheet was useful in covering the usage of this excellent HTTP toolkit by Project Discovery for performing recon against web servers and applications.