Comments on: Quick and Dirty URL Validation

By: Bob Aman

Bob Aman — Thu, 12 Feb 2009 02:06:49 +0000

If I wanted to break this, I think I’d write a custom “webserver” that responded to a HEAD request with an unending stream of headers. Server never closes the connection, just keeps sending you headers until the client finally gives up, assuming it ever does.

By: Trevor

Trevor — Wed, 11 Feb 2009 22:22:31 +0000

Hmm… yes… I think I got it now? I pasted it from running code, so I hope so :)

By: name

name — Wed, 11 Feb 2009 21:49:39 +0000

There is still one slash left unescaped.

By: Trevor

Trevor — Wed, 11 Feb 2009 20:58:54 +0000

This is my favorite URL format validator right now:

http://github.com/henrik/validates_url_format_of/tree/master

URI.regexp didn’t catch a lot of invalid stuff I tried to trow at it.

By: Trevor

Trevor — Wed, 11 Feb 2009 15:52:05 +0000

Ah yes. Fixed, thanks!

By: name

name — Wed, 11 Feb 2009 07:11:58 +0000

Btw, regexp is wrong. You should escape slashes:
regexp = url.match(/https?:\/\/([^\/]+)(.*)/)

By: Trevor

Trevor — Wed, 11 Feb 2009 02:13:47 +0000

Thanks, Walter. That looks like a good plugin to consider if you’re worried about overkill but need more than quick and dirty.

By: Walter McGinnis

Walter McGinnis — Tue, 10 Feb 2009 21:51:58 +0000

Sorry, meant to add link to the source. It can be found here:

https://modzer0.cs.uaf.edu/repos/hank/code/http_url_validation_improved/

By: Walter McGinnis

Walter McGinnis — Tue, 10 Feb 2009 21:51:16 +0000

This is the same basic concept of the older http_url_validation_improved plugin. It’s not on github, so perhaps that is why you missed it.

We use it in Kete (http://kete.net.nz) to achieve what you are after. It does look like it could use that URI.regexp refactoring added to it.

One nice thing is that it will check for allowed content types with configuration. It also gives more finegrained feedback when validation fails.

By: Trevor

Trevor — Tue, 10 Feb 2009 20:53:35 +0000

As an aside, found this easy way to do basic URL format validation:

http://www.ruby-doc.org/core/classes/URI.html#M004840

require ‘uri’
validates_format_of :uri, :with => URI.regexp

Very nice – no need for a custom regexp :)

By: Trevor

Trevor — Tue, 10 Feb 2009 20:24:39 +0000

Hmm… yeah, I guess you could use Timeout to help, but it does seem like making requests of arbitrary websites may have unindented consequences.

I’m not sure what you could do about receiving large header responses. I suppose a timeout could help there, too.

Here’s where I would start looking in terms of doing timeout stuff:

http://www.ruby-doc.org/core/classes/Timeout.html
http://www.slashdotdash.net/2008/02/15/ruby-tidbit-timeout-code-execution/

Although I came across these not too long ago:

http://blog.segment7.net/articles/2006/04/11/care-and-feeding-of-timeout-timeout
http://blog.headius.com/2008/02/rubys-threadraise-threadkill-timeoutrb.html

So, I’m not sure if timeout is safe to use or not :)

By: Tim Connor

Tim Connor — Tue, 10 Feb 2009 18:12:43 +0000

I like tenderlove’s refinement of sending back a very large response (is there somewhere you can cram base64 encoded video into a HTTP response header, maybe, to additionally add in the illegal/copyrighted content problems), when you finely do respond, to help chew up memory, too.

By: Jacob Harris

Jacob Harris — Tue, 10 Feb 2009 18:05:05 +0000

This does allow for a really simple denial-of-service attack on any site using it. Basically, you write a very simple server (you can do it in Sinatra if you like) that sleeps for a really long time on a head request to a specific URL. Then you just submit the URL multiple times to the form validating this URL. Repeat until you preoccupy ever Mongrel or cause Passenger to spawn enough Apache processes to thrash. Of course, you could add a timeout on the check, but I could also use Mechanize… ;)

By: Tim Connor

Tim Connor — Tue, 10 Feb 2009 17:59:05 +0000

Trevor, you may be entirely right, I do not know. My only point is counting on everyone else on the internet on honor “this MUST NOT have side effects” or whatever the language is, causes things like the Google Accelerator deleting lots of people info fiasco.

By: Tim Connor

Tim Connor — Tue, 10 Feb 2009 17:57:20 +0000

Ah I forgot to add the merely annoying one

http:/serverthattimesout.com

By: Daniel Berger

Daniel Berger — Tue, 10 Feb 2009 12:03:35 +0000

require ‘uri’
URI.parse(url).host

By: Trevor

Trevor — Tue, 10 Feb 2009 02:21:33 +0000

Yeah, I think the HEAD request is harmless. Maybe I’m wrong, though.

This technique I’m talking about isn’t a method for preventing spam or anything – it’s just a quick way to validate URLs are accessible (e.g. not http:/sdf38830.com or something nonsensical like that).

By: Caden

Caden — Tue, 10 Feb 2009 02:10:16 +0000

Well that was completely bizarre. Try adding more foil Tim. You need more foil.

By: Tim Connor

Tim Connor — Tue, 10 Feb 2009 02:08:43 +0000

Umm, I didn’t mean for those to actually be turned into links, sorry.

By: Tim Connor

Tim Connor — Tue, 10 Feb 2009 02:07:36 +0000

My last attempt got swallowed by your spam filter, I think.

Not a problem for your server getting hacked, but more in this line

http:/alqueda.com
http:/www.kiddiepr0n.com

http:/www.wellsfargo (thanks for the loan of your fat pipes for my DDOS)

http:/www.vulnerableserver.com/troublesome_url (at least they got your IP as the one that brought them down).

Basically, while I think HEADs will be mostly harmless, this still does leave you as an anonymous proxy in at least one way, for people who may know what to actually exploit (unlike me).

By: Trevor

Trevor — Tue, 10 Feb 2009 01:28:39 +0000

Err… I’m not sure how exactly this could be a security problem. Perhaps an example would help?

By: Tim Connor

Tim Connor — Tue, 10 Feb 2009 01:22:30 +0000

Abusable? Sure HEAD isn’t *supposed* to do anything, but I’ll bet money there are sites out there that have URLs you shouldn’t be blindly hitting. I guess a simple timeout wouldn’t be the end of the world, but could be moderately annoying. Also, I could make your servers show connections to TERRORISTS or kiddie porn servers, if I knew where to connect to them (which I don’t).

All in all, I’m not sure you want to leave your server connecting up to another one to a blind check.