Getting paperclip, heroku and aws s3 non-us region bucket working together

As I have found out earlier, images uploaded via paperclip to heroku will go missing after each push of code to heroku.

One way to keep images uploaded via paperclip permanent is to use a 3rd party storage service like s3 or keep it on a ftp server. I went for s3, mainly want to learn dealing with Amazon Web Services, might turn to ftp server later on if needed. If you would like to go the ftp server way, there are gem like FTP Storage for Paperclip and PaperclipFTP to make life easier.

  1. Add paperclip and aws-sdk to your gemfile
    gem 'paperclip'
    gem 'aws-sdk'

    Follow by the usual

    bundle install
  2. To start working with s3, first you need to signup for an aws account.
  3. Once you have an account, login then go to My Account/Console > AWS Management Console
    aws_mgt
  4. Select s3 from a host of services
  5. Create a bucket
    “A bucket is a container for objects stored in Amazon S3. When creating a bucket, you can choose a Region to optimize for latency”, that is what mentioned in the guide, so I chose Singapore as it is closest to my location.
    region
    Now if you have chosen “US standard” region, things will work out much straightforward, any non-us region will need some tweaks in the configuration. The documentation in heroku did mentioned that some international users may need to override the default URL structure and place the bucket’s name “domain-style” in the URL. Following that guide didn’t solve my problem though. I googled around, found a few more useful guides

    • Stackoverflow on setting env variables
    • Techspry on s3 configuation
    • DCChua on how to get Paperclip and AWS-S3 Singapore (and European) buckets working

    Coupled with the heroku documentation, I finally get my setup working combining them and change a bit here and there.

  6. Get your aws access key and secret access key from the Security Credential option
  7. In ./config/production.rb, defined this environment variables
      config.paperclip_defaults = {
    			    :storage => :s3,
    			    :s3_credentials => {
    			      :bucket => ENV['AWS_BUCKET'],
    			      :access_key_id => ENV['AWS_ACCESS_KEY_ID'],
    			      :secret_access_key => ENV['AWS_SECRET_ACCESS_KEY']
    			    },
    			    :path => ":class/:id/:basename_:style.:extension",
    			    :url => ":s3_sg_url"
    		  }
  8. Then set them by doing these in terminal from your apps
    heroku config:set AWS_BUCKET=your_bucket_name
    heroku config:set AWS_ACCESS_KEY_ID=your_access_key_id
    heroku config:set AWS_SECRET_ACCESS_KEY=your_secret_access_key

    You can of course set these values in the ./config/production.rb file, but then it is not protected if your repo is not private.

  9. In ./config/initializer, create paperclip.rb, and add these lines
    Paperclip.interpolates(:s3_sg_url) do |att, style| 
    "#{att.s3_protocol}://s3-ap-southeast-1.amazonaws.com/#{att.bucket_name}/#{att.path(style)}"
    end

git commit, push and then push to heroku, images uploaded from your apps should now store in s3.

why attr_accessor

I keep bump into this problem, where I pick up some basic long ago, after a while, get too used to it and forgot the reason why something is done in a particular way.

If I have written about it, then I have a better understanding and longer memory. Like the collection_select method which I drawn a illustration and post about it sometimes ago, it stuck in my memory till now.

This idea is not really new, you learn by teaching, or at least write or blog about it. All that help to clarify your thought.

And so, to help me internalized the reason behind attr_accessor, I quickly googled and read thru this from stackoverflow, yet, am gonna write about it as a process of internalization. Ha.

You keep writing attr_accessor, one day you totally forgot why we did attr_accessor at the first place. Why attr_accessor?

class Human
  def name=(str)
    @name = str
  end
 
  def name
    @name
  end
end

You do that to assign variable to a value, then call the value. This is common repetitive task. Programming is all about eliminating repetitive tasks. So you can do this in ruby

class Human
  attr_reader :name
  attr_writer :name
end

where attr_reader replace name method part, and attr_writer replace the name=(str) method. Still, often attr_writer and attr_reader often come in pair, why not just do it in one step.

class Human
  attr_accessor :name
end

I created a diagram hopefully could better illustrate this in one glance.

Evolution of attr_accessor
Evolution of attr_accessor

That is it. Already felt that now this tie deeper into my memory.

A marathon, not short sprint

A marathon, not short sprint
A marathon, not short sprint.
Image from stock.xchng, author cienpies.net.

I wish by now I have something positive to write as a fitting part 2 to the post wrote last year, you know, happy ending or something like that. The sad fact is, after numerous try, despite seemingly getting closer each time; I am still here one year later, still looking every possible ways to work with rails or web development work, full-time, oh, actually even part-time (So ping me if you have any web development works, especially ruby or ruby on rails stuff)

I have Interviewed for 4-5 opportunities, learned a lot, and got to meet some local startup people, developers, etc *but* none of that lead to anything, so far.

My first ever rails freelance gig was developing a simple web apps for a client. It was cancelled like 3 weeks into development. The client decided to look for a local developer instead. He said it is not my problem just that he preferred to work with a developer from local. One year down the road, the site is still not up. I guess the client decided not to pursue with his idea anymore.

Then another freelance gig come knocking, I was required to sign a NDA even. I was so thrilled with the opportunity, their idea is cool, it is like another facebook in the making. The core teams were working on the main features, and I was told that I would involve real soon. Now it is almost a year later, nothing happen still, haven’t heard any news from the team, their site haven’t changed for months. Not sure if they still working on the project.

Finally a local startup opening for Rails, not freelance gig but a chance of full-time employment, wow! So off I go for interview, we overcome the usual stumbling block by cutting down my salary (of course, not before serious discussion with the stakeholder – my wife, always grateful for her understanding), I offer to work with them for a few weekends in advance, so both sides can see if we are a fit. Think you can guess the outcome by now. 2 weekends down, and I got a sorry message, it is not something to do with me, but their internal decision. That is what I’ve been told at least. They did offer to pay up for my time, though I rejected.

There are one or two more freelance opportunities in between, that didn’t materialized. The common theme here is, “It is not your problem really”. The project just got cancelled, or stagnant, or fading out, or because of some internal decision I was not hired. But I refuse to buy the “It is not your problem” statement. There is no benefit in believing it is just plain bad luck, there is no benefit in believing it is just some internal decision to not hire. That is like admit defeat and say, okay, there is nothing I can do with it now. I can only work on things that I can control, which is *me*. I haven’t made myself so good, that there is no excuse at all not to hire me, of course that is my problem.

Stay positive, working on my own projects, writing even more codes, reading even more codes. Keep doing all that. This is a marathon, not short sprint. Even if no one take a chance on me, the effort I put in will lead me to somewhere.

I have to believe in that, cause there aren’t any other options anyway. Ha.

Uploaded images go missing in heroku

No doubt heroku is the greatest thing since sliced bread, with easy deployment and stuff, and it is free to start with. Perfect for stingy developer like me. So I am working on a rails apps, deployed to heroku, with paperclip for image upload. Things work fine, except images I uploaded keep on missing. I thought heroku wouldn’t store image for free user, ha.

A quick search around internet, sure enough it is stackoverflow come with an answer top in the list. It turn out that only images in the commit and pushed to heroku are kept, example logo or icon used in the apps. Uploaded images will be removed with each push to heroku, clearly images uploaded via paperclip fall into the latter category.

Solution is to use a 3rd party storage, like Amazon Simple Storage Service. Of course, aws is free to start with as well, a free account includes 5Gb storage, 20k Get requests, 2k put request, yada yada, another great thing for stingy developer, hurray!

Autorun mess up bundle install

I still feel like working with rails in Linux feel easier and faster. What’s with all the homebrew thing in Mac, feel like you have to go through a lot of troubles just to complete something simple. But working with rails in Windows is another story altogether. Initiatives like Railsinstaller and Devkit help a lot to get Windows to be less of a pain, but still…

So I have install railsinstaller, and Devkit come installed with it. And I have things running pretty normal, till lately I try to create a new rails apps, doing a bundle install lead to error saying

Gem files will remain installed in C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9 .1/gems/json-1.7.6 for inspection. 
Results logged to C:/RailsInstaller/Ruby1.9.3/lib/ruby/gems/1.9.1/gems/json-1.7. 6/ext/json/ext/generator/gem_make.out An error occured while installing json (1.7.6), and Bundler cannot continue. Make sure that gem install json -v '1.7.6' succeeds before bundling.

I tried to reinstall railsinstaller, and manually install Devkit. No working. Then tried to uninstall railsinstaller, and have ruby installer manually, then Devkit, make sure Devkit path is under ruby root, still the problem persist. Finally I found the solution in stackoverflow, from a link in treehouse’s blog . The thing is Devkit doesn’t work with Autorun interfaces, so if you have previously have this setup, in my case it was the I setup for syntax highlighting in command prompt, then you will have to remove it.

Just type this into command prompt

REG QUERY "HKCU\Software\Microsoft\Command Processor"
REG QUERY "HKLM\Software\Microsoft\Command Processor"

Should get something like below

HKEY_CURRENT_USER\Software\Microsoft\Command Processor
    CompletionChar        REG_DWORD    0x9
    DefaultColor          REG_DWORD    0x0
    EnableExtensions      REG_DWORD    0x1
    PathCompletionChar    REG_DWORD    0x9

These columns actually stand for Key, Type and Value. If there is a Key named AutoRun, then this could be the root cause of the problem. In my case, I think the Autorun key was set when I setup ansicon.
Run this to remove it,

REG DELETE "HKCU\Software\Microsoft\Command Processor" /v AutoRun

Close command prompt, and bundle install again in a new command prompt session, things should work.

Futnotes running locally

Redo the whole process. Clone repo again, bundle install now work right away, as all the prerequisites are there.
Instead of running rake tasks separately, which fail previously with long list of errors

rake db:create:all
rake db:migrate
rake db:seed

I will just do this, which is a shortcut for rake db:create; rake db:schema:load, and rake db:seed

rake db:setup

And it work!

Precompile Assets,

rake assets:precompile

lead to an error saying “Segmentation fault while running ‘rake assets procompile'”. A suggestion in stackoverflow saying that the rootcause might be execjs, change to therubyracer gem should solve the problem. Even though there is no execjs gem dependency in the gemfile, I still add in therubyracer gem, bundle install again, now rake assets:precompile just work, and I have futnotes running locally. Now off to get the test done

Futnotes – setting up the repo

Cloning the repo

git clone https://github.com/Futnotes/dev_test-2.git devtest

Mixtures of errors doing bundle install though
First it is capybara

Using capybara (1.1.2) 
Installing capybara-webkit (0.12.1) with native extensions 
Gem::Installer::ExtensionBuildError: ERROR: Failed to build gem native extension.

According to these sources thoughtbot and stackoverflow qt need to be installed first, before install capybara.
To install qt, you must link to libpng, simply link to linpng will fail due to permission,

Error: Could not symlink file: /usr/local/Cellar/libpng/1.5.13/lib/pkgconfig/libpng15.pc
/usr/local/lib/pkgconfig is not writable. You should change its permissions.

To fix that

sudo chown -R kahfei /usr/local/lib/pkgconfig

then only follow by command below

brew update
brew link libpng
brew install qt

bundle install still fail though

An error occurred while installing rmagick (2.13.1), and Bundler cannot continue.
Make sure that `gem install rmagick -v '2.13.1'` succeeds before bundling.

It seems that lot of people having this same problem. Some of the suggestion work for some people, but none of it fixed my problem. Apparently this is a rmagick bug, as it does not support the newer version of ImageMagick. rmagick has not been updated for two years, so if there is an option, change to other gem to work with image, like minimagick

After hours of scouring the internet, and trying a few different approaches, using the magick-installer to install ImageMagick, then

bundle install

work again. No actually I lie,  bundle install yield another error

ERROR: While executing gem … (ArgumentError) marshal data too short

There are suggestions to remove ~./gem folder altogether and reinstall all the gem. But in my case, bundle install work on other rails apps, just not this one. So I clone again the repo to a new folder, now bundle install work!

Now creating development, testing and production environment. I already have postgresql installed locally, so I will use postgresql, my database.yml looks something like this

development:
adapter: postgresql
encoding: unicode
database: futnotes_development
pool: 5
username: kahfei
password: 
test:
adapter: postgresql
encoding: unicode
database: futnotes_test
pool: 5
username: kahfei
password: 
 
production:
adapter: postgresql
encoding: unicode
database: futnotes_production
pool: 5
username: kahfei
password:

now to create them

rake db:create:all

All three database created

rake db:migrate

with a long list of errors, it start with something like this

PG::Error: ERROR:  column "position" does not exist
LINE 1: ...RE "competitions"."name" = 'Premiership' ORDER BY position, ...

require file from same directory in ruby

Requiring a file sitting in the same directory wouldn’t work with this

require 'something'

Seems like Ruby 1.9 remove current directory from load path as I read from Stackoverflow

To make it work you could

require './something'

or use require_relative,

require_relative 'something'

The thing is,

load 'something.rb'

still work. So does that mean load and require have different load path?