Skip to content

GSA Proxy Scraper internal Proxy: Problems with Cloudflare http

jojodatjojodat germany
edited September 2019 in GSA Proxy Scraper
Hi,
i noticed when i want to access a url via http protected with cloudflare via the internal proxy i get:

504 Gateway Timeout

unable to find suitable sub-proxy


after gsa tried some proxies.
It only happens when you try the url with http and not https.
Example url to test:
http://uam.hitmehard.fun/

Comments

  • SvenSven www.GSA-Online.de
    When I try to open this in browser, I get a normal 200 reply without https redirects or cloudflare.
  • jojodatjojodat germany
    edited September 2019
    Strange - as the site is only made to test the cloudflare under attack mode.

    When i try to access it via browser (firefox) with GSA internal proxy set. I see in http debugger that GSA is trying a proxy - cloudflare replies the challege (jumping dots) on a 503 page and gsa continues to next proxy (you never see the challenge page in the browser). And after some it stops and sends the 504 to the browser.

    Firefox with a manually set http (connect) proxy displays the 503 - the browser solves the js challenge displayed there and then continues to the real http page (200 there)

    Over https it's working as normal
  • SvenSven www.GSA-Online.de
    im afraid i cant do much here really.
  • jojodatjojodat germany
    Too bad. 

    Was hoping to you would not jump to next proxy on 503 but check if it contains "Server: cloudflare" Header or a "CF-RAY" header then deliverthe page to the browser. 

    But ok, then i have to try to change the library i use for cf bypass to prefer https instead of http.
  • SvenSven www.GSA-Online.de
    hmm that would be something to do yes. can you post the whole packet received by the proxy call on your end?
  • jojodatjojodat germany
    edited September 2019
    Sure
    here the whole process:
    https://imgur.com/KgmpdiY

    here a 503 
    https://pastebin.com/innj8x53

    and i noticed if a recaptcha challenge gets displayed (proxy has a bad rating), they send a 403:
    https://pastebin.com/veqxv97t
  • SvenSven www.GSA-Online.de
    ok, next update should improve this detection.
  • jojodatjojodat germany
    Wow - great!
    Thank you.
  • jojodatjojodat germany
    Just updated and tested. Works great. 
    Thank you
  • SvenSven www.GSA-Online.de
    your welcome, glad it works as expected.
Sign In or Register to comment.