And you thought all the clever caching names were taken.

What is it

ActsAsCachola is a plugin that lets you cache any class method by simply prepending ‘cachola_’ to the method name when calling it. Here’s how it works:

Given the following model:

class Internet < ActiveRecord::Base
  acts_as_cachola
 
  def self.get_a_million_numbers
    1.upto(1_000_000).inject([]){ |numbers, x| numbers << x }
  end
end

Now you can call the method, ‘cachola_get_a_million_numbers,’ and the return value of ‘get_a_million_numbers’ will be cached automatically.

Note that if the method accepts arguments, each unique call will have its own key in the cache. For example:

class Internet < ActiveRecord::Base
  acts_as_cachola
 
  def self.get_numbers(to_number)
    1.upto(to_number).inject([]){ |numbers, x| numbers << x }
  end
end

Calling Internet.cachola_get_numbers(100) and Internet.cachola_get_numbers(500) will result in two keys (with different values) stored in the cache.

The cached method is then expired automatically when the class in which the plugin has been included is saved or destroyed. It’s restored to the cache the next time it’s called.

Now, what if your Internet class method ‘get_a_million_numbers’ depends on other objects getting saved or destroyed? That’s the other thing I wanted to make easier. Rather than setting up observers or sweepers, you can add the following to the other model:

class WhereAmI < ActiveRecord::Base
  acts_as_cachola_notifier => [:internet]
end

Now when your WhereAmI model is ether saved or destroyed, the cached methods in the Internet model will be deleted.

Installation

script/plugin install git://github.com/rbrant/acts_as_cachola.git

Where is this going from here?

Not sure. It does what I need it to do right now. It’s something I’ve found myself doing on two different projects that I thought would just make my life easier.

Project Info

ActsAsCachola is hosted on Github: http://github.com/rbrant/acts_as_cachola, where your contributions, forkings, comments and feedback are greatly appreciated. Please do add tests if you want me to pull in any changes.

I had to process some pretty big xml docs recently from the USPTO. Each doc is about 60mb and (oddly enough) contains several thousand individual documents all concatenated. So the document isn’t valid xml..but that’s a different story.

The reason for writing this was to show a quick demo of how to use SAX to process a large XML file. You can read about SAX here, but basically, SAX (Simple API for XML) is an event-driven model that solves the problem of having to read an entire tree structure into memory which can be realllly sloooow, and instead reads the stream of data and raises events along the way.

The code below uses the Nokogiri library (which as a side note has this odd, albeit entertaining tagline: “XML is like violence – if it doesn’t solve your problems, you are not using enough of it.”). Most other XML parsing libraries also have SAX implementations.

What the code does below is looks for the root node of each doc and builds a string for each individual document. After the doc has been assembled, the doc can be processed via the more pleasant:

doc = Nokogiri::HTML(xml)
serial = doc.css("application-reference document-id doc-number").inner_text

So this ends up being sort of a hybrid and much, much faster than loading the entire doc at once. It would be faster not parsing the doc again at all but the docs have too much nested complexity that requires the ability to use xpath to get at what I need.

Download now or preview on posterous

what-matters-now-2.pdf (3073 KB)

Posted via email from Rich’s posterous

It’s easy to forget what you’ve learned and what tools you used from project to project. I thought it might be worthwhile to sort of sum up these things either on a weekly basis or project basis. I had a lot of fun on a recent project and thought it would be a good place to start. I recently built what is described as a ‘tool for intelligently searching US patent application Image File Wrappers (IFWs).’

Technically, the system allows users to upload PDF documents and have their content indexed and made searchable. The documents are reasonably sized, averaging 25 megs each with several hundred pages. So once uploaded to the server, they are handed to delayed job to be processed in the background. I’m using collective idea’s fork after watching Ryan Bates’ screencast on delayed job that points out this fork has a few generators and rake tasks not part of the original.

In order to index the document, the PDF needs to be examined by OCR (optical character recognition) software. But before the OCR software can do its OCR-ing, it needs to have an image to examine. So we need to convert the individual PDF pages into images. To accomplish that, I used ghostscript. It’s really easy to use and fast. You can hand ghostscript the document, and it will churn out a an image of each PDF page, in the resolution of your choice. I’m using 300×300, which seems to be a nice balance between processing time, space, and readability/ocr results.

Once the document has been converted into images the OCR software, tesseract-ocr, will iterate through each image and produce a text file with the contents of the page. Now, with a directory full of text files, it’s time to store the contents of each page in the database. That’s where sphinx and thinking sphinx come into play. Sphinx is the full text search engine and thinking sphinx is a ‘concise and easy-to-use Ruby library that connects ActiveRecord to the Sphinx search daemon, managing configuration and searching.’ I actually started the project with ferret/acts_as_ferret, but after reading so many good reviews of sphinx, and my own problems with ferret, I switched. The only downside is that setup is a little trickier and thinking sphinx doesn’t automatically update the index the way acts_as_ferret does, so there’s a cron job that handles that. The indexer is super fast though, so frequent indexing isn’t a problem.

The site also offers multiple file download, and I used the rubyzip library which makes it simple to zip up a bunch of docs into one.

As for design, we used a theme from themeforest.net. I was impressed by the quality of generic templates they have. They aren’t free, but are dirt cheap – $5 or $10  for most. I’ve used

It’s a Rails application, so as for gems/plugins, the usual suspects are there: acts_as_commentable, exception_notification, restful_authentication, role_requirement, mislav-will_paginate, attachment_fu, and a few others: slicehost, thinking-sphinx, delayed_job, and rubyzip.

The real jewel in the list is ‘slicehost‘ which gives you a bunch of rake tasks for setting up your slice at slicehost, which is my favorite hosting provider.

One other thing worth mentioning was an issue with the delayed_job process not stopping properly during deploys, so I kept getting multiple instances of delayed job running because the one running during the deploy never stopped. It was noted on github (with the solution below) in the issues section but I can’t find it now. Basically, the restart task looks like this:

desc "Restart the delayed_job process"
  task :restart, :roles => :app do
  stop
  wait_for_process_to_end('delayed_job')
  start
  end
end
 
def wait_for_process_to_end(process_name)
  run "COUNT=1; until [ $COUNT -eq 0 ]; do COUNT=`ps -ef | grep -v 'ps -ef' | grep -v 'grep' | grep -i '#{process_name}'|wc -l` ; echo 'waiting for #{process_name} to end' ; sleep 2 ; done"
end

As a final note, all the software used in this project is open source. I’m constantly reminded of that and impressed by it. My thanks to all who have contributed to the software used in this project!