Free Everything

Hacking Content Liberation

Presentation at Open Source Bridge 2016
June 21, 2016
Erik Möller - @xirzon

Slides: https://freeyourstuff.cc/osb2016 (source)

Imagine a world... in which you don't own your data.

Many sites offer ..

  • No export of all your contributions
  • No Creative Commons license
  • Limited APIs

The Network Effect

and how to hack it.

But how does it work?

How does a Chrome extension work?

Firefox WebExtensions (still alpha) are very similar ("designed for compatibility").

Let's look at a plugin!

Reminder: Slides at https://freeyourstuff.cc/osb2016
GitHub repo at: https://github.com/eloquence/freeyourstuff.cc

extension/src/plugins/yelp/index.js 
function retrieveReviews(callback) {
  let page = 1;
  let reviews = {
    head: {},
    data: []
  };
  let firstURL = 'https://www.yelp.com/user_details_reviews_self';
  $.get(firstURL)
    .done(processPage)
    .fail(plugin.handleConnectionError(firstURL));
(Confused by ES6 syntax like let? Here's a cheatsheet.) 
function processPage(html) {
try {
  let $dom = $($.parseHTML(html));
  if (page === 1) {
    let idLink = $dom.find('.user-display-name');
    reviews.head.reviewerName = idLink.text();
    reviews.head.reviewerID =
      (idLink.attr('href').match(/userid=(.*)/) || [])[1];
  }

retrieveReviews(reviews => {
  datasets.reviews = new DataSet(reviews, request.schema.reviews).set;
  chrome.runtime.sendMessage({
    action: 'dispatch',
    data: datasets,
    schema: request.schema
  });
});
Plugins are fragile! How to test them?
Like this: 
var jsonTests = {
  reviews: retrieveReviews
};

The Node.js test runner will:

  1. load Chrome's cookies
  2. run retrieveReviews inside jsdom
  3. diff with any previous results to show inconsistencies
Using jsdom inside Node.js 
let virtualConsole = jsdom.createVirtualConsole().sendTo(console);
const document = jsdom.jsdom(undefined, {
	userAgent,
	cookieJar,
	virtualConsole,
	url: schemas[schemaKey].schema.site.canonicalURL
});
let window = document.defaultView;

service/tests/index.js

What's next?

  • Investigate phantomjs for tests
  • Download multimedia!
  • Your plugin?
  • And a related project:

Credits: