Archive for the ‘ Uncategorized ’ Category

How to show a Percent Complete in Python

from __future__ import division
import sys

sys.stdout.write('\r')
sys.stdout.write('%.2f%% complete' % (number / total * 100,))
sys.stdout.flush()

How to fix ‘url’ requires a non-empty first argument. The syntax changed in Django 1.5

I really hate doing all this junk by hand, so I wrote a sed script to do it for me. Make sure you have a backup first, then run this in your templates directory:

find . -type f -print0 | xargs -0 sed -i 's/ url \([^" >][^ >]*\)/ url "\1"/g'

It’ll go through all of your template files and replace this:

{% url something.else foo bar %}

with this

{% url "something.else" foo bar %}

Be careful, I was a little lazy with this, it might get stuff you don’t want it to if you have a lot of writing in your templates. It’s still going to be easier looking for errors in a diff than doing it by hand, though.

Unfollow everyone on formspring

I’ve been creating accounts on various social networking sites lately for a project we’re working on, and I somehow managed to bulk-follow over a thousand people on formspring.me with their “find friends” tool. Like seriously, I was following hundreds of Asian teenagers — no idea how that happened.

No worries though, I just fired up jash and jquerify (a no-conflict version, since formspring.me uses prototype, which jquery doesn’t always play nice with) to bulk unfollow all of them.

Once you get jquerify and jash up, just put this bad boy in jash, and you’re good to go:

$jq.each($jq(".unfollow"), function() {
	var str = this.id.split('-');
	fspring.unfollow(str[1],this,'People-View');
});
Unfollow all on Formspring - screenshot

Unfollow all on Formspring

Be prepared to wait!!

Spotadoink

As small after-work project inspired by FitFinder (which, unfortunately, has since been taken down per the demands of the UCL), several months ago Montmere Limited launched spotadoink, the 21st century catalyst to flirting.

It was hastily thrown together in just a few short hours, but the concept is fun: when you see someone who is attractive, tell the world where!

Unfortunately it has not seen much use since its launch, but as with anything else, who knows what the future holds.

Facebook Developer Subscriptions – Part 1

So Facebook has an API.  It’s pretty full featured although the documentation seems to be a little on the weak side.

After a little glancing, I thought “I can do this” and followed the sample php code to create a “Login” page for my test site.  I then spent all kinds of time and notebook paper coming up with an elaborate series of cron jobs to use the Access Token to make requests on the user’s behalf so I can periodically check for updates while they’re away.

Then I read the manual.

Facebook offers a subscription service via Real-Time Updates where you can request to be notified of certain changes as they happen.  Facebook doesn’t actually tell you what the changes are, but they notify you that a certain type of change has occurred (new friends, activities, etc.) Cool.

Great.  So all I need to do is get a user authenticated to my application using some example code from the php SDK, let Facebook know what kinds of updates I’m interested in, and set up an endpoint to get that data.  Easy enough, right? Ha.

So the way the Facebook API works is that you request a user’s data using their Access Token (that the developer gets when the user allows a certain application access).  The main problem with using this vanilla approach with subscriptions is that the access token given when using the example code in the php SDK expires after a little while.  Well, I want to get user updates all the time so I need a token that doesn’t expire.  This requires using Extended Permissions.  A quick look through the list tells me I need the “offline_access” permission.

Unfortunately, this is where I realized that the example code in the php SDK starts to lack.  The login URL built into the SDK $facebook->getLoginURL() doesn’t allow you add your requested permission in.  We’ll need to take a few extra steps to get what we’re looking for.

First, we need to construct our own login URL. The format will look something like this:

https://graph.facebook.com/oauth/authorize/client_id={your application ID}&redirect_URL={where to send the user after they do OR don’t authenticate}&scope=offline_access

Using this in place of the function call will cause the user to see something like this when they click on “Login”:

Request for Permissions

When your user clicks “Allow”, they will be redirected back to your redirect_url with an additional parameter called “code”.  Our neverending access token! Well, almost.  Now we need to make another request, server side time, to Facebook to turn this code into an actual access token.  This is done by requesting from a URL like this:

https://graph.facebook.com/oauth/access_token/client_id={your application ID}&redirect_URL={where to send the user after they do OR don’t authenticate}&client_secret={your secret code made when setting up developer account}&code={the code you just got in your return URL from Facebook}

Getting this URL returns the following data:

access_token=178416385509226|bbf513b2bf0c667340e80b40-15123454|9-ha-O6KUwHVtY51PtNxUR_Fkz4

So you’re application will have to parse our your actual access token from the string, but that should be a good start.

Update: after playing with this for a little bit, I found that the facebook session will return the access_token with extended permissions to you when your user is redirected back to your page.  This means you can do something like access $facebook->session->[‘access_token’] and save that instead of having to use curl for that second URL.

Next up: setting up your subscriptions and endpoint.

The ANTI JOIN – all values from table1 where not in table2

One of the less intuitive concepts I come across regularly in SQL is that of the ANTI JOIN. That is, requesting data from a table where some value is not in another table.

The wrong way of doing it is this:

SELECT
*
FROM table1 t1
WHERE t1.id NOT IN (SELECT id FROM table2)

This is very slow and inefficient, because the RDBMS has to compare each row with an entire table of values. The following is much much faster, as it matches up each row once, then simply eliminates the results that are not in the second table:

SELECT
*
FROM table1 t1
LEFT JOIN table2 t2 ON t1.id = t2.id
WHERE t2.id IS NULL

This is what is known as an ANTI JOIN.. And I really wish it were part of standard SQL.. Be careful, though,if table2.id has any NULL rows, you will experience unexpected results. Here’s my proposal, and the following should throw a fatal error if table2.id is not a NOT NULL column:

SELECT
*
FROM table1 t1
ANTI JOIN table2 t2 ON t1.id = t2.id

Too bad I don’t make the rules.

Get the average value from the nearest date in MS SQL

Needed to get the average exchange rate for the nearest invoice to a given date, and I found a quick and easy way to accomplish this in SQL Server:

SELECT TOP 1
   AVG(i.exchrate) / 100 AS exchrate
   , MIN(DATEDIFF(DAY, '" . $date . "', i.invoicedate)) AS accuracy
FROM InvoiceTable i
WHERE
   i.currencycode = '" . $currency_code . "'
   AND DATEDIFF(DAY, '" . $date . "', i.invoicedate) > 0
ORDER BY MIN(DATEDIFF(DAY, '" . $date . "', i.invoicedate))

GROUP_CONCAT and truncation / field serialization in MySQL

I wish there were a true serialize() grouping method in MySQL, as I frequently deal with data that must be viewed as groups of tables or groups of groups of tables, and sometimes even deeper. One hacky way I occasionally use (only when there is really no other alternative, because this sucks) to accomplish this without having to resort to recursion is

GROUP_CONCAT(items, '|||')

Most datasets I work with are sane enough to safely assume I won’t find 3x consecutive pipe characters in a given field. So, I simply explode that field out into an array.

$data = array();
while ($row = $result->fetch()) {
   $row['items'] = explode('|||', $row['items']);
   // or, to eliminate null columns / empty string values:
   //$row['items'] = preg_split('/\|\|\|/', $row['items'], -1, PREG_SPLIT_NO_EMPTY);
   $data[] = $row;
}

One of the problems you will run into, though, as your datasets start to get a little larger, is that this is an insufficient solution since GROUP_CONCAT() truncates each row by default (on most systems) by 1024 characters. Not to worry — there is a very simple fix to this.. simply increase the value of group_concat_max_len.

DB::mysql()->exec("SET SESSION group_concat_max_len = " . PHP_INT_MAX);

Maybe someday there will be a more elegant solution to what should be an unusual problem, as it seems I come up with a nasty solution like this on a weekly basis these days.

Next on the list should be a MODE() aggregate function. Like, for real.. tell me that’s harder to implement than STDDEV_POP().

Transfering MySQL databases between Servers via SSH

Usually, when transfering a MySQL database from one server to another, people pipe the output of mysqldump into a file, archive it if it is large, upload it to the remote host, unarchive it, and then pipe it into the MySQL instance from that shell.

There is a much easier way, although it doesn’t archive the data first.. but usually unless you’re dealing with larger tables, the ease of using this method makes slower transfers very acceptable.

mysqldump -u user -ppassword databasename | ssh username@example.com "mysql -u user -ppassword databasename"