3 July 2011

With the growing number of offline-capable web applications also grow the requirements developers have. One of those is to be able to upload files while the user is offline or their connection crashes a lot and larger uploads fail. Fortunately, W3C defined a great set of standards that allow web apps to do just that.

This post explains what is happening behind the scenes of this ultimate solution to file uploads.

HTML

HTML required for offline uploads is very simple:

<div class="button disabled">
  <span>Upload a file</span>
  <input type="file" name="files" disabled />
</div>

However, we are going to ensure successful uploads in browsers that do not implement the File API (those will not work while offline, though). The best possible way to do this is to make use of the commonly known iframe trick—that is that we submit a form into an iframe—which spares us page reloading. The HTML structure allowing both offline and online uploads looks like this:

<div id="uploader"> <!-- root element -->
  <form method="post" action="./upload" enctype="multipart/form-data"
    target="upload-target" class="button disabled">
    <span>Upload a file</span>
    <input type="file" name="files" disabled />
  </form>
  <iframe name="upload-target" style="display: none;"></iframe>
</div>

With some styling we can get a pretty good-looking button.

Basic Event Binding

We are going to observe the change event on the input element and initiate the upload process if the user chooses a file.

// Our whole "application" is going to be stored in one global namespace object.
var uploader = {};

window.onload = function () {
  var root_el = document.getElementById('uploader');
  uploader.init(root_el);
};

uploader.init = function (root_el) {
  this.input_el = root_el.getElementsByTagName('input')[0];

  this.input_el.onchange = function () {
    uploader.upload();
  };
};

This script is enough to check for files in non-IE browsers. IE requires the user to click the button of the input but we need to have the whole form element clickable. Another way to let the user choose a file is to call the click method of the input element (this even works when the element is hidden).

We could listen to click events on the form element and call the click method of the input when a click occurs. Then, we could submit the form using the submit method. That is, however, not possible in IE due to security reasons. This fact complicates the process a bit because we need to add a submit button element. The button will cover the whole form element and will be hidden using opacity: 0 (that way it remains clickable).

<div id="uploader">
  <form method="post" action="./upload" enctype="multipart/form-data"
    target="upload-target" class="button disabled">
    <span>Upload a file</span>
    <input type="file" name="files" disabled />
    <button type="submit" disabled>Upload a file</button>
  </form>
  <iframe name="upload-target" style="display: none;"></iframe>
</div>

In the script, we need to detect IE and set event listeners accordingly. The onchange handler is going to be set only for non-IE browsers and for IE, there is going to be an onclick handler on the new button element in which we let the user choose a file.

// Detect IE
var IE = (navigator.userAgent.search('MSIE') !== -1);

uploader.init = function (root_el) {
  this.form_el = root_el.getElementsByTagName('form')[0];
  this.input_el = root_el.getElementsByTagName('input')[0];
  this.button_el = root_el.getElementsByTagName('button')[0];

  if (IE) {
    this.button_el.onclick = function () {
      // Let the user choose a file
      uploader.input_el.click();
      // Prevent form submission is no file was chosen
      return uploader.input_el.value;
    };
  } else {
    this.input_el.onchange = function () {
      uploader.upload();
    };
  }
};

Client-side Database

To ensure support across all modern browsers (WebKit-based, Gecko-based and Opera), we have to use both IndexedDB and WebSQL with IndexedDB being the primary choice. The idea is that when the users chooses a file, its contents are read via the File API (FileReader) and stored in one of the database storage solutions. The application then tries to push the file to the server until it is successfully uploaded.

First, we need to detect which database storage is available.

// Since the IndexedDB standard has not been finished yet, there are several implementations.
var INDEXEDDB_SUPPORT = ('indexedDB' in window)
  || ('webkitIndexedDB' in window) || ('mozIndexedDB' in window);
// Web SQL implementations, on the other hand, use the same property.
var WEBSQL_SUPPORT = ('openDatabase' in window);

// Together with File API support, we can say that a given browser
// is capable of offline file uploads.
var OFFLINE_MODE = (INDEXEDDB_SUPPORT || WEBSQL_SUPPORT)
  && ('files' in document.createElement('input'));

As step 2, we need to connect to the database storage (and build its structure on the first use). Our database will be called uploader and it will have one object store (table) called files.

uploader.init = function (root_el) {
  // ... the same as above ...

  if (OFFLINE_MODE) {
    this._initDatabase();
  } else {
    this.enable();
  }
};

// The control is initially disabled. (See the HTML.)
// We call this method after the initialization process.
uploader.enable = function () {
  this.input_el.disabled = false;
  this.button_el.disabled = false;
  this.form_el.className = 'button';
};

uploader._initDatabase = function () {
  if (INDEXEDDB_SUPPORT) {
  var req = (window.indexedDB || window.webkitIndexedDB
    || window.mozIndexedDB).open('uploader');
    req.onsuccess = function (e) {
      uploader.db = e.target.result;
      uploader._buildDatabase();
    };
    // If the connection fails, we fallback to regular online uploads.
    req.onfailure = function () {
      OFFLINE_MODE = false;
      uploader.enable();
    };
  } else if (WEBSQL_SUPPORT) {
    this.db = window.openDatabase('uploader', '', 'Pending uploads',
      10 * 1024 * 1024); // 10 MB should be enough
    this._buildDatabase();
  }
};

uploader._buildDatabase = function () {
  if (this.db.version === '1.0') {
    this.enable();
  } else if (INDEXEDDB_SUPPORT) {
    var req = this.db.setVersion('1.0');
    req.onsuccess = function (e) {
      uploader.db.createObjectStore('files', {
        autoIncrement: true
      });
      uploader.enable();
    };
    // If we are unable to set up the object store,
    // we fallback to regular online uploads.
    req.onfailure = function (e) {
      OFFLINE_MODE = false;
      uploader.enable();
    };
  } else if (WEBSQL_SUPPORT) {
    this.db.transaction(function (tx) {
      tx.executeSql("CREATE TABLE [files] ( " +
        "[id] INTEGER PRIMARY KEY AUTOINCREMENT, " +
        "[name], [data] " +
      )", [], function () { // success
        uploader.enable();
      }, function () { // failure; fallback to online uploads
        OFFLINE_MODE = false;
        uploader.enable();
      });
    });
  }
};

The target page (./upload) should return an HTML page with a callback that enables the control after the upload finishes. (This approach is generally called JSONP). Such response might look like the following code:

<script>
window.top.uploader.enable();
</script>

Offline uploading

Now that we have the basic environment initialized, we can get into the actual offline uploading. One thing we need is a way to disable the control while there is a running upload operation.

uploader.disable = function () {
  this.input_el.disabled = true;
  this.button_el.disabled = true;
  this.form_el.className = 'button disabled';
};

We already set up the handler that calls out upload method in the initialization so we only need to define this method (and its subsequential methods).

uploader.upload = function () {
  if (OFFLINE_MODE) { // Use File API
    var files = this.input_el.files;
    if (files.length !== 0) { // files selected
      uploader._uploadViaFileAPI(files);
      this.disable();
    }
  }
};

uploader._uploadViaFileAPI = function (files) {
  // Our file input can be set up to accept multiple files.
  Array.prototype.forEach.call(files, function (file) {
    var reader = new FileReader();
    reader.onloadend = function (e) {
      uploader.store(file.name, e.target.result);
    };
    reader.readAsDataURL(file);
  });
};

The two methods above get file contents and pass them to the store method which actually stores them in the database storage.

// Read/write IndexedDB transaction factory method
uploader.createIndexedDBTransaction = function () {
  return this.db.transaction(
    ['files'],
    (window.IDBTransaction || window.webkitIDBTransaction
      || window.mozIDBTransaction).READ_WRITE,
    0
  );
};

uploader.store = function (name, data) {
  if (INDEXEDDB_SUPPORT) {    
    var tx = this.createIndexedDBTransaction();
    var store = tx.objectStore('files');
    var req = store.add({
      name: name,
      data: data
    });
    req.onsuccess = function () {
      uploader.enable();
      uploader.pushQueueToServer();
    };
    req.onfailure = function () {
      uploader.enable();
    };
  } else if (WEBSQL_SUPPORT) {
    this.db.transaction(function (tx) {
      tx.executeSql("INSERT INTO [files] ([name], [data]) VALUES (?, ?)", [name, data], function () {
        uploader.enable();
        uploader.pushQueueToServer();
      }, function () {
        uploader.enable();
      });
    });
  }
};

At this point, we have a control that stores uploads in a database storage or submits them directly to the server if no database storage is available. The only missing piece is a method that would push the formed upload queue to the server. Calls of such method (pushQueueToServer) are already implemented in the store method above.

uploader.pushQueueToServer = function () {
  if (INDEXEDDB_SUPPORT) {
    var tx = this.createIndexedDBTransaction();
    var IDBKeyRange = window.IDBKeyRange || window.webkitIDBKeyRange || window.mozIDBKeyRange;
    var range = (IDBKeyRange.lowerBound || IDBKeyRange.leftBound)(0);
    var req = tx.objectStore('files').openCursor(range);
    req.onsuccess = function (e) {
      var result = e.target.result;
      if (result) {
        uploader.uploadAsBase64ViaXHR(result.value, function () {
          var tx = uploader.createIndexedDBTransaction();
          var req = tx.objectStore('files')['delete'](result.key);
          req.onsuccess = function () {
            uploader.pushQueueToServer();
          };
          req.onfailure = function () {
            uploader.pushQueueToServer();
          };
        });
      }
    };
  } else if (WEBSQL_SUPPORT) {
    this.db.transaction(function (tx) {
      tx.executeSql("SELECT * FROM [files] LIMIT 1", [], function (res) {
        var item = res.rows[0];
        if (item) {
          uploader.uploadAsBase64ViaXHR(item, function () {
            tx.executeSql("DELETE FROM [files] WHERE [id] = ?", [item.id], function () {
              uploader.pushQueueToServer();
            }, function () {
              uploader.pushQueueToServer();
            });
          });
        }
      });
    });
  }
};

What we do is that, if there is one, we get the oldest item from the queue (database) and pass it to the uploadAsBase64ViaXHR method and as a callback, we remove the given item/file from the queue.

The uploadAsBase64ViaXHR method basically just creates a POST request via XMLHttpRequest and passes the execution to a callback function when the request successfully finishes.

uploader.uploadAsBase64ViaXHR = function (item, callback) {
  var data = [
    'name=' + encodeURIComponent(item.name),
    'data=' + encodeURIComponent(item.data)
  ].join('&');

  var xhr = new XMLHttpRequest();
  xhr.open('POST', './upload-base64', true);
  xhr.setRequestHeader('x-requested-with', 'XMLHttpRequest');
  xhr.setRequestHeader('content-type',
    'application/x-www-form-urlencoded; charset=UTF-8');
  xhr.onreadystatechange = function () {
    if (this.readyState === 4) {
      if (this.status < 300) { // success
        callback();
      } else { // failure; try again in a moment
        setTimeout(function () {
          uploader.uploadAsBase64ViaXHR(item, callback);
        }, 5000);
      }
    }
  };
  xhr.send(data);
  xhr = null;
};

A good thing to do is to call the pushQueueToServer method right after database initialization to check if there are any pending uploads from before.

uploader.buildDatabase = function () {
  if (this.db.version === '1.0') {
    // ...
    this.pushQueueToServer();
  } else if (INDEXEDDB_SUPPORT) {
    // ...
    req.onsuccess = function (e) {
      // ...
      uploader.pushQueueToServer();
    };
    // ...
  } else if (WEBSQL_SUPPORT) {
    this.db.transaction(function (tx) {
      tx.executeSql(/* ..., ... */, function () {
        // ...
        uploader.pushQueueToServer();
      }, /* ... */);
    });
  }
};

There is one more thing to do. A few code snippets back, we defined a different behavior for IE. However, the next IE (10) is probably going to feature the File API and IndexedDB storage which would make it possible to perform offline uploads even in IE. To take such future situation into account, we have to modify the onclick handler of the submit button element:

uploader.init = function (root_el) {
  // ...

    this.button_el.onclick = function () {
      // Let the user choose a file
      uploader.input_el.click();
      if (OFFLINE_MODE) {
        uploader.upload();
        // Prevent form submission
        return false;
      } else {
        // Prevent form submission is no file was chosen
        return uploader.input_el.value;
      }
    };

  // ...
};

Also, it might not be a bad idea to possibly broaden the support to older non-IE browsers by extending the upload method a little. (This is also called when database initialization fails.)

uploader.upload = function () {
  if (OFFLINE_MODE) { // Use File API
    // ...

  } else { // Use regular form submission; older non-IE browsers
    var value = this.input_el.value;
    if (value) {
      this.form_el.submit();
      this.disable();
    }
  }
};

Have a look at the complete solution with some logging and control label changing.

When it comes to server-side requirements, we have two upload endpoints—upload and upload-base64. Those, of course, can be merged into one. This is just to show the need to have two different server-side handlers, one for regular multipart uploads and one for simple POST requests with base64-encoded file contents in their bodies.

As to browser support, I tested this in Google Chrome 12, FireFox 4, Opera 11 and IE 8/9. However, it should work in any earlier version of Chrome, FireFox (probably even 1.x but I’m not totally sure about that) and probably in IE 7 but I was not able to start the portable version to check it out.

I hope you found this post helpful. Feel free to share it with your colleagues.

Jan's Noteblog

Offline File Uploads Using File API Explained

HTML

Basic Event Binding

Client-side Database

Offline uploading

Twitter status

Elsewhere