Can access URL with browser but not with working code

Question

I want to check if a bunch of URLs are working. So, I wrote some code (given below) to do that. It works for sites like google.com. When I apply it to my scenario, it fails.

I am logging into a VM. From this VM, I can open the desired URL in a browser. When I try to check if I can connect to the URL with code, it fails. The URL obtained by my code from file is correct and works on browser. So, an error in the URL is ruled out.

My server urls look like this -

ab-web-internal-test-005.myweb.com

How do I debug this issue and enable my code to connect to the URL?

This is the exception:

C:/mycode/>ruby LinkTester.rb
C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:878:in `initialize': No connection could be made because the target machine actively refused it. -
 connect(2) (Errno::ECONNREFUSED)
        from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:878:in `open'
        from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:878:in `block in connect'
        from C:/Ruby200-x64/lib/ruby/2.0.0/timeout.rb:52:in `timeout'
        from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:877:in `connect'
        from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:862:in `do_start'
        from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:851:in `start'
        from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:582:in `start'
        from C:/Ruby200-x64/lib/ruby/2.0.0/net/http.rb:477:in `get_response'
        from LinkTester.rb:9:in `connect_to_url'
        from LinkTester.rb:38:in `block in <main>'
        from LinkTester.rb:37:in `each'
        from LinkTester.rb:37:in `<main>'

Code -

require "net/http"
require "uri"

def connect_to_url(url)
  response = nil
  encoded_uri = URI.encode(url)
  uri = URI.parse(encoded_uri)

  response = Net::HTTP.get_response(uri)
  http = Net::HTTP.new(uri.host, uri.port)
  response = http.request(Net::HTTP::Get.new(uri.request_uri))

  case http_response
  when Net::HTTPSuccess
    puts  uri + "success"
  when Net::HTTPRedirect
    puts  uri + "success"
  else
    puts uri + "failure"
  end
end

def get_urls(file_path)

  array = Array.new
  file = File.open(file_path, "r")
  file.each_line do |line|
    array << line
  end
  file.close
  return array
end

url_file = "C:/mycode/servers.txt"
url_array = get_urls(url_file);

url_array.each do |url|
  connect_to_url(url)
end

the Tin Man · Accepted Answer · 2015-01-10 02:17:32Z

2

It isn't particularly useful to compare a browser and an HTTP agent. Browsers do use similar underlying technology way down deep, but it has a LOT of other code that tries to be resilient and as friendly as a puppy. Unless you know what else a browser does when trying to present something useful to a user, it can be difficult to debug URLs and HTTP. You'd probably do better using OpenURI inside IRB or one of the other gems that supplies a console, or use cURL at the command-line, because then you are in more control of the behavior.

Your code isn't handling redirects. You handle a redirect response with the same message as a success, which isn't true since they're not the same thing but you can't tell which is which.

Net::HTTP is a low-level library for creating HTTP services, but, because it's low-level, you have to tell it how to do everything. That's fine if you are creating your own new service, but for retrieving pages you can do it more easily using OpenURI or one of the other gems to handle your access that DO implement redirection like Curb, Typhoeus, HTTPClient, RestClient, HTTParty, etc.

IF you are going to use Net::HTTP, then you need to implement the full redirect-handling code, which is given in the documentation.

edited Jan 10, 2015 at 2:17

answered Jan 9, 2015 at 23:37

the Tin Man

161k44 gold badges222 silver badges308 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Erran Morad Over a year ago

Thanks !!! "OpenURI inside IRB or one of the other gems that supplies a console" - I am not sure what this means. Can you please tell me what "supplies a console" means and why its good to have one ? Also, why is it better to use IRB ?

the Tin Man Over a year ago

Use IRB to explore your code choices. "Supplies a console" means you can open an IRB session using that particular gem at the command-line and explore what you received. Look at the documentation for the mentioned gems and you'll find one.

Erran Morad Over a year ago

the Tin Man - I am thinking of using Mechanize because it has many good tutorials. But, I am not even able to install the gem due to some strange errors. New post on that.

the Tin Man Over a year ago

Mechanize is fine if you need to fill in forms and locate and click buttons and links. If you're parsing pages to extract information from tables or text from paragraphs, use Nokogiri.

Erran Morad Over a year ago

tin man, may be HTTPClient is good. It has been around since 2007, so it must be quite mature and well developed as compared to Httparty and curb. Besides, I am hoping there will be a lot of tutorials also.

oldhomemovie · Accepted Answer · 2015-01-09 23:45:35Z

1

As per the Tin Man's explanation, will try to answer using the typhoeus notion:

require 'typhoeus'

File.readlines('C:/mycode/servers.txt').each do |server_uri|
  puts Typhoeus::Request.new(server_uri).run.code
end

answered Jan 9, 2015 at 23:45

oldhomemovie

15.2k16 gold badges68 silver badges104 bronze badges

5 Comments

Erran Morad Over a year ago

The gem can't be installed.

gem install typhoeus ERROR:  Could not find a valid gem 'typhoeus' (>= 0), here is why: Unable to download data from https://rubygems.org/ - SSL_connect returned=1 errno=0 state=SSLv3 read server  certificate B: certificate verify failed (https://rubygems.org/latest_specs.4.8.gz)

I can make the gem installation run in my own computer though. However, it fails late due to nmake and some other issues which have nothing to do with server certs.

oldhomemovie Over a year ago

@BoratSagdiyev normally I wouldn't recommend this, but a as quick solution – try using http://rubygems.org as the gem source (note http instead of https)

Erran Morad Over a year ago

gmile - Please tell me how I can do that. What is the command ? Thanks.

Erran Morad Over a year ago

gmile - Thanks ! That works. One caveat though. The gem is installed, but the documentation is not because of cert errors.

Parsing documentation for typhoeus-0.7.0 WARNING:  Unable to pull data from 'https://rubygems.org/': SSL_connect returned=1 errno=0 state=SSLv3 read server certificate B: certificate verify failed (https://rubygems.org/latest_specs.4.8.gz)

Btw, why don't you recommend it ?

oldhomemovie Over a year ago

@BoratSagdiyev I would recommend https against http only to make sure the data is transmitted via a secure channel.

Collectives™ on Stack Overflow

Can access URL with browser but not with working code

2 Answers 2

5 Comments

5 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

5 Comments

Your Answer

Sign up or log in

Post as a guest

Related