Ruby curl with follow redirects

Just a technical FYI; this took a little digging to find in the documentation.

If you’re using Curl::Easy in ruby to download html (or results from our API), then FYI the default is to NOT follow redirects. If you want to follow redirects and download the page contents of the target, you’ll need to set the option option


to true.

Here’s a code snippet :

  def download_url url
    res = ""

    tries = 0
      tries += 1
      easy =
      easy.timeout = 30
      easy.follow_location = true
      easy.url = url
      res = easy.body_str 
    rescue Exception => e  
      retry unless tries > 2
      puts "#{url} failed, returning empty string, #{e.message}"

    ic ='UTF-8//IGNORE', 'UTF-8')
    res = ic.iconv(res + ' ')[0..-2]

    return res.downcase

Comments are closed.

%d bloggers like this: