Jun 14, 2013

Checkvist mobile: notes on WIP

Some more notes along the way.

More on browser caches

Refreshing cached pages and templates is a PITA. But I've found a great Google Chrome's DevTools "Disable cache" option.

Also, when using remote debugging of Chrome on Android, one have to use Android's settings to reset browser caches, "reset cache" command from DevTools didn't work for me.

Routing

Adding a login page inevitably lead to the need to navigate through the app, so I need to add routing support.

I've tried can.route from CanJS - but it doesn't support HTML5 history API yet. So I'm going to use this simple routing library instead.

Given that I have a single-page app, with different URLs which actually point to the same index.html file, I have to patch NGinx configuration to redirect all pages to /index.html internally. To do this, I used information from this post.

Login

For actual login, I had to add CORS support to the Checkvist site. After that, AJAX requests to Checkvist API work nicely.

Internationalization

At some point, I understood that I want to have I18N in the app. So I went with a very simple custom solution: create a separate JSON file with mapping between messages and message codes + simple utility function which does the actual mapping in EJS templates. No need for external library :)

Testing

Time to add tests. Because it is a right thing, and right now I want to make sure that such thing as token expiration is handled correctly.

So, I added QUnit support for Javascript testing, JQuery.mockajax for mocking AJAX requests, added service messages extension for TeamCity continuous integration.

Automated testing task in the Rakefile, to run tests for every test_*.html file under "tests" directory:

    task :test do
      Dir.foreach("test") do |name|
        if name =~ /test_.*.html$/
          system("cd test && phantomjs run.qunit.js ./#{name}")
        end
      end
    end

Feel on a safe side.

AJAX loading of view templates

To allow AJAX loading of local files (in tests), start Chrome on Mac with this command:

open -a "Google Chrome" --args --allow-file-access-from-files

Loading lists from the server

Using CanJS can.Model to implement lists model for the Checkvist. The model is cached locally, into IndexedDB and updated from the server. To interact with IndexedDB so far I use DB.js + IndexedDBShim to emulate IndexedDB over WebSQL when native support is not available.

May 17, 2013

More work on Checkvist mobile infrastructure

This is the next post regarding Checkvist mobile project.

Javascript

Trying to figure out some framework requirements. What I need is:

  1. Single JS file from multiple JS files (modules)
  2. Offline app loading
  3. Client-side undoable persistent deferrable commands

For the first point I'm going to use RequireJS (and its optimizer r.js). Also, it looks like later it is cool to use almond to optimize load times.

After some jumps, managed to make requireJS to work with CanJS MVC and Zepto jQuery replacement (and even r.js optimizer works!).

For offline apps loading, offline HTML manifest works just fine. There is a trick, though - to be able to load a non-mentioned resource in the manifest file (in development mode), one have to add

NETWORK:
*

in addition to usual resources section.

Another note: if there is any problem while loading resources from the manifest file, all other files of the app are not re-loaded.
This is quite critical - you have to make sure that all resource files are present.

The third point of the list I defer so far, until real interaction with the server is implemented.

CSS

It is cool to have a single JS file, but I'd like to have the similar for the CSS.

For this, Sass looks rather good, given that it optimizes @imports + gives tons of syntax sugar for CSS.

Adding Sass wasn't difficult:

  1. Make sure ruby is installed
  2. Create Gemfile with

    source 'https://rubygems.org'
    gem "sass"
    gem 'rb-fsevent', '~> 0.9.1'
    gem 'rake'
    
  3. gem install bundler && bundle install
  4. Create Rakefile:

    require 'sass/plugin'
    namespace :sass do
      task :env do
        Sass::Plugin.options[:template_location] = 'www/scss'
        Sass::Plugin.options[:css_location] = 'www/css'
      end
    
      desc 'Updates stylesheets if necessary from their Sass templates.'
      task :update_and_compress => :env do
        `rm www/css/*`
        Sass::Plugin.options[:style] = :compressed
        Sass::Plugin.update_stylesheets
      end
    
      desc 'Watch stylesheets'
      task :watch => :env do
        Sass::Plugin.watch
      end
    end
    
    namespace :requireJs do
      desc 'Run optimizer/uglifier for JS code'
      task :optimize do
        `node r.js -o app.build.js`
      end
    end
    
    namespace :cl do
      task :build => ["sass:update_and_compress", "requireJs:optimize"] do
        print "Done!
      end
    end

For automatic conversion of scss files to css peers during the development, I have to keep the following command running:

rake sass:watch

By the way, Sass mixins are rather useful, given that there are tons existing examples, for instance this Retina support mixin.

HTML offline issues

After having continuous trouble with updating HTML5 offline manifest file and double page refresh after each change I decided to go a way when the manifest is added to the page only while building the production version of the site. I've added a very simple task to Rakefile:

task :update_for_offline do
    index = "build/www/index.html"
    updated = IO.read(index).sub("<html>", "<html manifest='checkvist.manifest'>")
    IO.write(index, updated)
end

So far so good. MVC is almost in place, JS and CSS are united and minified in the project. Time for the real meat.

May 9, 2013

Day one, persistent storage and native HTML5 wrapper research

I'm going to keep a log of useful notes on a way to mobile Checkvist app. Not sure how patient I will be, let's see.

Persistent storage

First, I tried to figure out the current options for offline storage in the mobile app.

Looks like the most obvious choice could be WebSQL, but this spec is deprecated since 2011, though supported on all major mobile platforms.

The alternative standard, IndexedDB, is much less wide-spread, and less performant.

Here is a useful overview of offline options, and here are slides from a related talk (with a big list of intermediate libraries for offline storage).

So far, I'm considering using this Ydn-DB library to create an intermediate layer for data persisting.

Native wrapper for HTML5 app

Tried PhoneGap, which is essentially Apache Cordova

This library allows to create a native app, which wraps a Web-browser and which provides a bunch of APIs for accessing native features of the device.

The first thing which stroke me was that I had to use symbolic links to share common part of the app between iOS and Android versions of the same application.

The code of the application is deployed with the native app, but you can make AJAX calls to remote servers (though I didn't test it yet).

By the way, Android plugin for IntelliJ IDEA works with Android Cordova app just fine :)

Checkvist mobile app, take 2

In short, the idea is to create a new mobile app for Checkvist with the following key capabilities:
  • offline read/write operations
  • undo for commands
  • Android/iOS compatibility
  • rich touch interface
  • using device camera to post photos as attachments in Checkvist
  • work with external keyboards
  • app upgrade without need for app approval in iOS store
I'd like to achieve this using pure HTML5/JS/CSS + native app wrapper (mostly for marketing via Apple store/Google Play categories). But anyway, it should be possible to reach the app using http://m.checkvist.com.
The task is big and ambitious, but it looks a great way to learn something new :)

Jan 30, 2013

Hiding sidebar in Checkvist with CSS3

Just wanted to share how I've used CSS3 transformations/transitions to hide sidebar in Checkvist.

Nice effect, and easily implemented - by adding a CSS class to the sidebar element (and hiding it afterwards):

div.hideSidebarProgress {

  -webkit-transform: scale(0.01);
  -webkit-transform-origin: 99% 0%;
  -moz-transform: scale(0.01);
  -moz-transform-origin: 99% 0%;
  -o-transform: scale(0.01);
  -o-transform-origin: 99% 0%;
  -ms-transform: scale(0.01);
  -ms-transform-origin: 99% 0%;
  transform: scale(0.01);
  transform-origin: 99% 0%;

  margin-top: -80px;
  -webkit-transition-property: margin-top, transform;
  -webkit-transition-duration: 1s;
  -webkit-transition-timing-function: ease;

  transition-property: margin-top, transform;
  transition-duration: 1s;
  transition-timing-function: ease;
}

Here I use CSS3 transform to scale the sidebar down, and CSS3 transition for margin-top and for the transform scaling to create the effect.

The benefit - user immediately knows where to look for the "show sidebar" option, when needed.

Nov 13, 2011

Ruby on Rails upgrade from 2.3 to 3.1

Recently I've performed the upgrade of Ruby on Rails which we use in Checkvist project from 2.3 to 3.1. There were tons of issues I've had to overcome, some of them I share here, may be they could save someone's time.

 First, upgrade or not upgrade? This is a tough question. Upgrading of a non-trivial project may take several days, and at the end you may get a system which performs worse than before. So you'd better have a good set of system tests, including performance tests.

 In my case, I used Rails 2.3.11 with my performance-related patch for using it with ruby 1.9.2, and I aimed at migration to Rails 3.1.1 (latest at the time of migration). I really like the assets pipeline feature introduced in the 3.1.x version of Rails, and I wanted to start using ruby 1.9.3. After all, new features, plugins, enhancements are developed against latest version of RoR.

At the time of writing  Checkvist is running RoR 3.1.2, and I'm pretty comfortable with its performance.

And here is a list of notes I made during the upgrade (the more complicated is your project, the more unexpected the issues):

General upgrade notes

  • Upgrade from 2.3 to 3.1 consists, actually, of 2 parts: upgrade to 3.0 and upgrade from 3.0 to 3.1. Some deprecations, which were allowed in 3.0, are obsolete in 3.1 (for instance, you must use the new routes syntax in your project).
  • One step which you can do before "breaking everything" during the upgrade - migrate your 2.3 project to use bundler. See the bundler page for the instructions - it is not that painful. Do not forget to update your Capistrano deployment script to handle bundler.
  • In my case, Checkvist was already compatible with ruby 1.9.2, so I used this version of ruby during the upgrade, and after that, updated ruby to 1.9.3
  • All mentiones of RAILS_ROOT, RAILS_ENV should be replaced with Rails.root and Rails.env, respectively. The same applies to Rails.logger .
  • uninitialized constant Syck::Syck problem should be resolved by running gem update --system
  • Rework all old-style ActionMailers to use the new syntax for sending mails - old syntax doesn't work in Rails 3.0 anymore. Also, check naming of your e-mail templates.
  • All files with extension .rhtml should be renamed to .html.erb

Upgrading to 3.0

    • For upgrade steps, I followed instructions from railscast, and used rails upgrade plugin
    • Upgrade of the routes.rb file may be non-trivial. Having tests for routes makes this process less painful. I don't recommend using automatic converter - it is better to understand what's going on there and make your route rules cleaner.
    • Update/rewrite of environment.rb file is rather simple: old stuff goes mostly to application.rb + session_store.rb; non-typical mime types are specified in mime_types.rb .
    • Gemfile: most gems have new, updated version for Rails 3+, no problems encountered here. All issues described in the early migration tutorials with will_paginate or thinking-sphinx gems are already resolved. 
    • After the update, I've removed some obsolete plugins and initializers (and some monkey-patches for rails 2.3 code). May be you have to review them as well :)
    • I had to add 'dynamic_form' gem to enable old-style helpers for error_message_on and others
    • Ruby files from 'lib' subdirectory were invisible until I uncommented "config.autoload_paths += %W(#{config.root}/lib)" line in application.rb
    • I didn't have lots of link_to_remote and remote_form_for helpers (which disappeared in 3.0), so I updated the code explicitly, using my own AJAX helpers in the corresponding places.
    • Cucumber integration tests. This part was tricky, because I used Cucumber with usual Rails integration tests (no webrat, no capibara). I needed POST and PUT requests. And support for such direct integration was broken in the newer version of Cucumber. But, you can use Cucumber directly with Rack::Test, and get the application response from last_response method of Rack::Test - and this was sufficient to modify my tests accordingly.

    Upgrading to 3.1

    • The path for upgrading to 3.1 was described in this post, but in my case the switch to using assets was not simple. 
    • I didn't use a single application.js and application.css files for the whole application (because in this case, after each update all .js and .css code would be downloaded by users, no caches). I've created several files, more specific, per page/type of page. To accomplish this, section config.assets.precompile of environments/production.rb  should include all paths for all .js and .css files used by your application
    • To deliver asset files in production, you have to change Nginx configuration as described in this great guide

    Performance notes

    • To get a decent app performance, comparable with 2.3.x, use ruby 1.9.3 and  specify garbage collection parameters.
    • It is a bad idea to override often-used attribute method in your model (for instance, to provide a default value). In Rails 3 such code works damn inefficiently - it doesn't create effective accessors for such methods
    • Do not use changes method to get information about which attributes changed - use changed_attributes for that

      So much for the upgrade stories. Will be very glad if you find some of these notes useful in your riding on Rails.

      Jun 13, 2011

      Prototype 1.7 memory leak

      Lately, I've been trying to fix a memory leak in TeamCity. After a long investigation, I found out that DOM elements on the page remains in memory even after a simple construct like:
        element.on("click", Prototype.emptyFunction).stop();

      This code adds a fake event listener on a element and immediately removes it (all using Prototype javascript library).

      I.e., after executing the code and removing the element from the page, it remained in browser memory.
      (BTW, I used Google Chrome memory profiler to find out hanging elements).

      So, the leak was caused by the Prototype's code, and I found out this nasty line:
          CACHE.push(element);

      Ironically, Prototype tries to avoid memory leak in IE by removing all event handlers on page unload, and for that, keeps a collection of the elements in the CACHE array. But, it never removes elements from the array (only on page unload), even when all event handlers are removed from an element. For any heavy ajax application, when elements are added/removed/replaced on the fly, this may be very unpleasant.

      I've fixed the problem in my Prototype fork on GitHub, and added a pull request to Prototype. Hope, it will save someone some hairs.

      You can also download patched prototype.js with the fix.

      Update: my original fix introduced a performance problem in stopObserving method (when many elements were observed on the page). Now, the problem is fixed, all related resources above were updated.

      Feb 13, 2011

      Checkvist downtime postmortem

      Checkvist service was unavailable since Feb, 13 04:13 UTC till 07:40 UTC.

      The total downtime was 3 hours 27 minutes. Users couldn't see/modify their information, but no data corruption occurred.

      We are sorry to everyone who was unable to access their data during that time. We've already taken some measures to prevent such problems in the future, see more details below.

      What went wrong

      • On Saturday, Feb 12, we've updated software on the production server (ruby, passenger, nginx).
      • At 04:13 UTC a daily server maintenance procedure was started. Some parts of the procedure were incompatible with the new installed software, and access to Checkvist was broken.
      • E-mail and SMS notification were sent to us. We didn't see e-mail notification because we were away from computers. The SMS notification was sent to an obsolete phone number :( So the problem was unnoticed until the morning, when we've checked e-mail.

      Problem resolution

      Measures to prevent similar problems in the future

      • Daily maintenance procedure was corrected
      • We've set another time for the daily maintenance, so it would take place in the daytime
      • We've updated our monitoring service. Existing phone number for SMS notifications was corrected, we also added another phone number for the notifications. We've also installed iPhone app which can notify about service failures.
      And finally, why all these explanations. I think it's better to be transparent and honest about the problems than make people guess about system's reliability.




      Jan 28, 2011

      Javascript error reporting in various browsers

      Again, about fighting with Javascript errors.

      I've just found out, that Opera browser in some situations provides the most detailed information about javascript errors. At least Opera 11 is rather good.

      Compare the error reporting in various browsers (all on Mac):

      FireFox 3.6.13+FireBug 1.6.1 - didn't report the problem, neither in FireBug console, nor in FireFox javascript errors window.

      Chrome 8.0.x
      gave this:





      And the winner, Opera 11:





      See the difference?

      Nov 5, 2010

      Internet Explorer AJAX errors debugging in Prototype

      This is a short story I want to put down for those who face the same problem.

      In Checkvist project, there is some not very trivial AJAX code. We use Prototype javascript library for AJAX handling. Prototype allows to specify error dispatcher for AJAX javascript errors, and as a fallback solution, I set window.alert() to report errors.

      Several days ago I got a bug report related to list deletion operation in Checkvist - Internet Explorer showed a couple of alert error dialogs. The problem was intermittent on the production server, and I couldn't reproduce it locally at all.

      Really nice and cool development tools in IE8 couldn't help me. I've had an exception object in my error handler, but there is no way to obtain stacktrace from it and find out where the real problem occurred.

      I tried to rethrow the caught exception so IE would show me what happened - no luck.

      After plenty of digging in Google I found a hint. IE can debug an error occurred in Javascript code only if it wasn't wrapped in try/catch block. So if you handle problems yourself - there is no way to catch the real problem in the Internet Explorer debugger.

      To solve the problem I had to remove all try/catch blocks from Prototype's AJAX code, put it to the staging server, spend another 10 minutes to reproduce the problem, and - voila! - got an invitation dialog from IE8 to debug the uncaught error.

      That was it. The error was fixed, Prototype was reverted to original state, code put to the production.

      If you know another solution for the problem - please comment, I'd be glad to know.


      May 16, 2010

      Feature discoverability

      One of the great approaches to developing usable software is to actually use the product you're working on. It is often called "eating your own dog food". The benefits are obvious - you have clear source of the requirements and priorities - because you're your own customer. A lot of good books for entrepreneurs like "The Art of the Start" or "Rework" promote this approach.

      But there is a trap.

      While adding features you mostly think of their usefulness, speed, productivity. You ponder how you'd use them in the most efficient way. And this sounds good, right?

      In this situation you're thinking as an expert user of your product. As a user who knows the product perfectly well from its day one.

      Features designed for experts are usually hidden. Keyboard shortcuts, advanced interface settings, non-trivial mouse gestures, cool rich functions hidden deeply in the UI. And you, as an expert of your product, might not expose these features enough (especially if you care for UI simplicity and don't want the interface to be bloated).

      And even when you're adding a non-expert feature, you can still hide it somewhere in the interface and your users won't find it. You definitely will - you know where it is. But other people can be unaware of its existence at all, or (worse) miss it even after searching for the feature.

      The most relevant recipe to detect such problems early is to perform regular usability testing, especially for the new functionality. Unfortunately, most software development teams don't do this, though even a simple 5 second test may help.

      Here are some other steps to ensure feature discoverability:
      • Provide a logical predictable navigation system in the application, either in the menu or some other way. Many users use the menu as a reference for the application features.
      • Give users a possibility to search for a particular function, either in the reference documentation or search for an action within the UI.
      • Make a screencast and/or blog post to inform your existing users about new cool feature.
      • In the application, create a "Fresh updates" section, which will inform existing users about new versions of the application, and which new features are available in it.
      • Create a "Hint area" in the application, which shows some relevant actions for the current situation.
      • If your application perform some lengthy operations, you may show users some "Tip of the day" screen while such operations in progress.
      • Prepare a printable cheat sheet with the application's keyboard shortcuts, if you provide them.
      We're facing the discoverability issues both in TeamCity and in Checkvist, and going to deal with them in the nearest future, so your feedback is appreciated.
      How do you help your users to find new or advanced features in your application?

      Apr 21, 2010

      What I like about TeamCity 5.1

      Hello,

      I've decided to sum up the most interesting (from my personal point of view) new stuff in TeamCity 5.1 release. I started writing this post in text, but decided that a form of outline is more suitable for that. So here is the outline I've prepared using Checkvist:




      So please, grab it and use! TeamCity Professional Edition is free. TeamCity Enterprise for OS projects is also free.

      And there is 60 days evaluation license for TeamCity Enterprise (you can use this license to upgrade your existing TeamCity Pro installation).

      Enjoy your builds :)

      Mar 12, 2010

      Memory leak fix in Scriptaculos Autocompleter

      The latest released version of scriptaculos (1.8.3) has a really old memory leak. In short, Autocompleter creates a lot of event handlers and never removes them. Given that Checkvist will use tag autocompletion rather intensively, I've decided to fix this problem.

      My solution is attached to the issue at the lighthouse and also available in my fork of scriptaculos.

      May be this fix will be helpful to someone else :)

      Mar 8, 2010

      Calculating the cursor position in textarea with JavaScript

      I've been spending some time writing tag support in Checkvist, and decided to share a bit of related JavaScript code.

      The idea is to allow adding tags with smart syntax: when you write "Call Bob regarding new furniture tomorrow #home" Checkvist will create a task "Call Bob regarding new furniture" with due tomorrow and with tag #home.

      The additional nicety could be the tag completion after the '#' character. But here comes a problem - how to show the completion popup near the cursor position in the textarea? The typical approach used in del.icio.us or GMail is to show the popup under the text field, but it doesn't look good when your cursor is far from the popup, in the beginning of a large text area.

      After some googling I didn't find an existing solution. The only helpful code was detection of the text cursor position relatively to the beginning of the string.

      To convert this position into pixel coordinates, one has to find out how the text is organized within the textarea, where linebreaks are, how many lines are there in the text, and so on. This, in turn, depends on the font metrics of the text and requires answer for the question "What is the length of the given string if it is placed into this textarea?".

      The answer for the last question can be obtained if you create a div with the same font metrics as the original textfield, give it absolute positioning, put a string into it, and take its width. Using such a function, you can model text wrapping in the textarea, and find out the actual X,Y coordinates of the cursor in the textarea.

      You can find my implementation of this approach on GitHub (BTW, a really great place to share open-source code). This library was tested with FF3, Chrome, IE, Opera, Safari. There may be some glitches, but they are rare and I'm pretty comfortable with the results so far. There is no need for Prototype, jQuery or other Javascript libraries to use the code.

      I hope this code will be useful to someone else, and if so, please drop me a line :)

      Aug 22, 2009

      Checkvist Pro plan

      After a long silence, we've finally released Checkvist Pro plan. The detailed post about this is available.

      With this release, I've also upgraded Rails to the version 2.3.3 and migrated Mysql database from MyISAM to InnoDB engine (because we've had problems with data consistency when transactions were interrupted).

      So far so good, performance haven't become worse. And I'm pretty satisfied with upgrade of the app engine from mongrel cluster to Nginx Passenger.

      Now will focus on the daily work and on adding more features to Checkvist.