Running with Ruby

Tag: Ruby 2.1 (page 1 of 2)

Rack/Rails middleware that will add rel=”nofollow” to all your links

Few years ago I wrote a post about adding rel=”nofollow” to all the links in your comments, news, posts, messages in Ruby on Rails. I’ve been using this solution for a long time, but recently it started to be a pain in the ass. More and more models, more and more content – having to always declare some sort of filtering logic in the models don’t seem legit any more. Instead I’ve decided to use a different approach. Why not use a Rack middleware that would add the nofollow rel to all “outgoing” links? That way models would not be “polluted” with stuff that is directly related only to views.

Nokogiri is the answer

To replace all the rel attributes, we can use Nokogiri. It is both convenient and fast:

require 'nokogiri'

doc = Nokogiri::HTML.parse(content)

doc.css('a').each do |a|
  a.set_attribute('rel', 'noindex nofollow')
end

doc.to_s

Small corner cases that we need to cover

Unfortunately there are some cases that we need to cover, so simple replacing all the links is not an option. We should not add nofollow when:

  • There’s already a rel defined on an anchor
  • There are local links that should be “followable”
  • There are local links with a full domain in them
  • We want to narrow anchor seeking to a given css selector (we want to leave links that are in layout, etc)

If we include all of above, our code should look like this:

require 'nokogiri'

doc = Nokogiri::HTML.parse(content)
scope = '#main-content'
host = 'mensfeld.pl'

doc.css(scope + ' a').each do |a|
  # If there's a rel already don't change it
  next unless a.get_attribute('rel').blank?
  # If this is a local link don't change it
  next unless a.get_attribute('href') =~ /\Awww|http/i
  # Don't change it also if it is a local link with host
  next if a.get_attribute('href') =~ /#{host}/

  a.set_attribute('rel', 'noindex nofollow')
end

Hooking it up to Rack middleware

There’s a great Rails on Rack tutorial, so I will skip some details.

Our middleware needs to accept following options:

  • whitelisted host
  • css scope (if we decide to norrow anchor seeking)

So, the initialize method for our middleware should look like this:

# @param app [SenpuuV7::Application]
# @param host [String] host that should be allowed - we should allow our internal
#   links to be without nofollow
# @param scope [String] we can norrow to a given part of HTML (id, class, etc)
def initialize(app, host, scope = 'body')
  @app = app
  @host = host
  @scope = scope
end

Each middleware needs to have a call method:

# @param [Hash] env hash
# @return [Array] full rack response
def call(env)
  response = @app.call(env)
  proxy = response[2]

  # Ignore any non text/html requests
  if proxy.is_a?(Rack::BodyProxy) &&
    proxy.content_type == 'text/html'
    proxy.body = sanitize(proxy.body)
  end

  response
end

and finally, the sanitize method that encapsulates the Nokogiri logic:

# @param [String] content of a response (body)
# @return [String] sanitized content of response (body)
def sanitize(content)
  doc = Nokogiri::HTML.parse(content)
  # Stop if we could't parse with HTML
  return content unless doc

  doc.css(@scope + ' a').each do |a|
    # If there's a rel already don't change it
    next unless a.get_attribute('rel').blank?
    # If this is a local link don't change it
    next unless a.get_attribute('href') =~ /\Awww|http/i
    # Don't change it also if it is a local link with host
    next if a.get_attribute('href') =~ /#{@host}/

    a.set_attribute('rel', 'noindex nofollow')
  end

  doc.to_s
# If anything goes wrong, return original content
rescue
  return content
end

Usage example

To use it, just create an initializer in config/initializers of your app with following code:

require 'nofollow_anchors'

MyApp::Application.config.middleware.use NofollowAnchors, 'mensfeld.pl', 'body #main-content'

also don’t forget to add gem ‘nokogiri’ to your gemfile.

Performance

Nokogiri is quite fast and based on benchmark that I did, it takes about 5-30 miliseconds to parse the whole content. Below you can see time and number of links (up to 488) per page. Keep that in mind when you will use this middleware.

perf

TL;DR – Whole middleware

require 'nokogiri'

# Middleware used to ensure that we don't allow any links outside without a
# nofollow rel
# @example
#   App.middleware.use NofollowAnchors, 'example.com', 'body'
class NofollowAnchors
  # @param app [SenpuuV7::Application]
  # @param host [String] host that should be allowed - we should allow our internal
  #   links to be without nofollow
  # @param scope [String] we can norrow to a given part of HTML (id, class, etc)
  def initialize(app, host, scope = 'body')
    @app = app
    @host = host
    @scope = scope
  end

  # @param [Hash] env hash
  # @return [Array] full rack response
  def call(env)
    response = @app.call(env)
    proxy = response[2]

    if proxy.is_a?(Rack::BodyProxy) &&
      proxy.content_type == 'text/html'
      proxy.body = sanitize(proxy.body)
    end

    response
  end

  private

  # @param [String] content of a response (body)
  # @return [String] sanitized content of response (body)
  def sanitize(content)
    doc = Nokogiri::HTML.parse(content)
    # Stop if we could't parse with HTML
    return content unless doc

    doc.css(@scope + ' a').each do |a|
      # If there's a rel already don't change it
      next unless a.get_attribute('rel').blank?
      # If this is a local link don't change it
      next unless a.get_attribute('href') =~ /\Awww|http/i
      # Don't change it also if it is a local link with host
      next if a.get_attribute('href') =~ /#{@host}/

      a.set_attribute('rel', 'noindex nofollow')
    end

    doc.to_s
  rescue
    return content
  end
end

Ruby – Installation and compilation from sources on Debian

If you don’t want to use Ruby from your repositories and you don’t want (or can’t because of some reason) to use RVM, here’s a quick howto download and compile Ruby from sources (all the commands should be executed with sudo or as a super user):

First a small update and some neccessery libs:

sudo su
apt-get update

# Not sure if all of them are required but nothing bad will happen if you just install them

apt-get install build-essential bison openssl libreadline6 libreadline6-dev \
libyaml-dev libxml2-dev libxslt-dev zlib1g zlib1g-dev libssl-dev autoconf \
libc6-dev ncurses-dev libaprutil1-dev libffi-dev libcurl4-openssl-dev libapr1-dev

Then we should go here and download the most recent stable version:

# Still as root
cd ~
wget http://cache.ruby-lang.org/pub/ruby/2.1/ruby-2.1.2.tar.gz

Of course we need to unpack it:

tar -xvf ruby-2.1.2.tar.gz
# go to where it was unpacked
cd ruby-2.1.2/

and now the whole installation process:

./configure  
make  
make test  
make install 

and a small cleanup:

cd ~
rm -rf ruby-2.1.2*

To test it, just:

ruby -v
# output: ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-linux]
Olderposts

Copyright © 2017 Running with Ruby

Theme by Anders NorenUp ↑