Ruby curl with follow redirects

Just a technical FYI; this took a little digging to find in the documentation.

If you’re using Curl::Easy in ruby to download html (or results from our API), then FYI the default is to NOT follow redirects. If you want to follow redirects and download the page contents of the target, you’ll need to set the option option

easy.follow_location

to true.

Here’s a code snippet :

  def download_url url
    res = ""

    tries = 0
    begin 
      tries += 1
      easy = Curl::Easy.new
      easy.timeout = 30
      easy.follow_location = true
      easy.url = url
      easy.perform
      res = easy.body_str 
    rescue Exception => e  
      retry unless tries > 2
      puts "#{url} failed, returning empty string, #{e.message}"
    end

    ic = Iconv.new('UTF-8//IGNORE', 'UTF-8')
    res = ic.iconv(res + ' ')[0..-2]

    return res.downcase
  end

Advertisement