The charset problem

Until recently it was impossible to automatically validate the HTML of the pages on my website. The reason for this was that I had moved from XHTML 1.0 strict to HTML5 (to make working with the canvas easier and to be at the bleeding edge of available technologies.) I had made the recommended changes, so that the first few lines of the source looked like:

<!DOCTYPE html>
<html xmlns='http://www.w3.org/1999/xhtml' xml:lang='en' lang='en' charset="utf-8">
<head>
<meta charset="utf-8" />
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />

This should have fixed things, with a streamlined DOCTYPE to make things even easier, but instead I faced the following error message:

The HTML5 validator error
The HTML5 validator error

The reason was that the parser has to know the charset of the document, but the charset of the document was specified in the document itself (as was common practice for earlier versions of HTML.) This obviously doesn’t make sense, although there are some workarounds. If the charset appears in the first 512 characters the parser should be able to find the information, which is useful for when documents are saved out of context, but still not an ideal solution. After browsing online I found no suitable answer using only HTML and it turns out that the best practice is send some headers that specify the charset:

<?php header('Content-Type: text/html;charset=UTF-8') ; ?>

Of course this header has to be sent before the rest of the document to give the parser a chance to read the document properly in the first place. Once that was added the validator returned no errors:

Success!
Success!

(The warning refers to an experimental feature of the validator known as HTML5 Conformance Checker, and does not reflect an warnings associated with the page itself.)

This is a small fix to a simple problem, very much like the one I described in a previous post about changing standards and HTTP requests. These changes are a little frustrating, but they serve to remind us that we are working in a living medium that is constantly updated and that people take standards and security seriously. It also brings a little joy into my day to make my whole website a little better than it was before. One day I’ll be satisfied with my website, but it’s been development for about a decade now, and I don’t see that process coming to end any time soon.

Neuron (genetic algorithm)

One of my friends wanted to develop a genetic algorithm that was capable of learning and adjusting a neuronal network to suit its environment. I got as far as making a genetic algorithm before abandoning the project as my friend’s interest seemed to move at the same time as my knowledge of the problem was exhausted.

Links

Live page
GitHub repository

Overview

The algorithm creates a population of “animals” that have varying physiology. The physiologies that best match the source of “food” reproduce more frequently. As the source of food changes, so do the inherited physiologies. The physiologies are represented by cells in a table, each one given a weight. They are initially randomly distributed, with the weights representing “investment” in the food source.

The breeding algorithm takes three animals at random, compares their match to the food source, and breeds the two most successful to create a new offspring, while discarding the third. The genes from each parent are selected at random. This algorithm was inspired by a similar study performed by the YouTube user cdk007 in his video “Evolution IS a Blind Watchmaker”.

This was the result of a couple of hours of work, at which point the project was abandoned indefinitely. If I find some useful literature about how to develop a neuronal network capable of learning (ie simulating the lifespan of an animal with a learning process) I may revisit this project. It’s a shame the working title doesn’t actually reflect what the algorithm does.

Challenges

Challenge: The project needed some simple algorithm to get started, to be developed later.
Solution: Thanks to cdk007, I had some good inspiration here! Once the model was created there was no need for additional development- I had a fast evolving algorithm to work with. (Resolved)
Challenge: I wanted to create an algorithm that was very visual and quick to develop.
Solution: At the time I was very experienced with HTML and CSS, so I opted for a series of tables styled with CSS. I was very happy with how it turned out, it makes a nice web toy. (Resolved)

Screenshot

A screenshot of the evolving population
A screenshot of the evolving population

Multiplets

For my thesis I needed to describe many of the know mesons and baryons. To better illustrate their relation to each other I decided to create an SVG image showing a three dimensional representation in isospin and weak hypercharge space.

Links

Live page
GitHub repository

Overview

The multiplets are constructed from a series of points, wires and polygons. These are then rotated using two of three Euler angles (the rotation about the intrinsic \(z\) axis is not useful in this case) and the resulting SVG document is written server side using PHP. The client is then left to render the image.

Like many of my projects from the PhD era this suffers from a common problem: this project was developed when SVG was the best available technology for generating vector graphics on the fly, and as a result requires either a server side response (slow) or embedded Javascript (messy) to operate. This project could benefit from being migrated to the canvas.

Challenges

Challenge: The SVG image had to be rendered in such a way that objects were not obscured in the 3D view.
Solution: This was one of the first projects where I had to write a simple 3D viewer from scratch, and I opted to use the median dstance of an object form the viewer to determine the order in which to draw the objects. There were a few minor tweaks to acount for instances where this was not always successful, but the overall result was efficienct and overall I am pleased with the solution. This is very similar to the techniques I used for the EMC viewer and the aDetector project. (Resolved)
Challenge: This project allows the user to rotate the multiplets and the rotation must be intuitive.
Solution: This is only really meant to be used by physicists who are used to thinking with angles, and used to poor user interfaces, so given the audience I would say that the user interactions is a big success. However the page still makes requests to the server, meaning that the rotations do not happen in real time, and in the case of mutiple requests could lead to a large overhead on the server. In the future this should be handled client side to free up server side resources and speed up the user experience. (Resolved)

Screenshot

The meson multiplet
The meson multiplet

Alarm clock

This project was written very quickly one evening to give myself an emergency alarm clock after my phone was stolen. As a result it is not very user friendly.

Links

Live page 1
GitHub repository

Overview

This project simply plays a YouTube video after a fixed period of time. The alarm plays a fixed number of times at regular intervals.

Challenges

Challenge: Ideally, the user should be able to change the settings for the alarm.
Solution: I didn’t have time to add a user interface (as doing so would have caused me to lose sleep!) (To be revisited.)

Screenshot

A screenshot of the alarm clock
A screenshot of the alarm clock

That weird part of YouTube

One of the most fruitful memes on the internet is “That wierd part of YouTube”. I saw an opportunity to createa a service to find the weirdest videos on YouTube, and I made this webpage.

Links

Live page
GitHub repository

Overview

This page uses the YouTube Javascript API to load pairs of YouTube videos. The user then votes on which one they think is “weirder”, and two new videos are loaded for comparison. The backend is handled using PHP with a MySQL database to store and retrieve the video ids. The user can submit their own videos with client side validation of the URI and server side sanitisation to prevent MySQL injection attacks. The statistics associated with each video are also displayed, showing how many times each “won” or “lost” a comparison.

This project was never actually released outside my webspace, despite my original intentions. This is because YouTube is very clear about its branding policies- a third party cannot use the word “YouTube” without explicit permission, and the meme requires the name “YouTube” to be used. As a result this was only ever shown to my friends and never received much traffic. I may revisit this later and release to the general public with a different name.

Challenges

Challenge: As usual, this uses the YouTube Javascript API and all the difficulties that come with that.
Solution: The YouTube Javascript API is second nature now. This wasn’t difficult at all! (Resolved)
Challenge: The user should be able to recommend videos.
Solution: The interface to add suggestions had to be simple to use, so I made the form as clean as possible. The input had to sanitised as well. The URI is matched to a regular expression client side before submission, and sanitised server side to prevent injection. The server side sanitising is very easy given that it must consist of exactly 11 characters in the set [A-Za-z0-9_\-]. (Resolved)
Challenge: This page needed a catchy motif.
Solution: I wanted an image of YouTube videos falling from the sky, as if the videos has been rejected and thrown onto a pile. I had to emulate the YouTube logo using vector graphics with the canvas, then randomly generate an collection of them that wrapped around in the horizontal direction for the wallpaper. It was fun to make, and I may make a wallpaper generator based on this concept. (Resolved)

Screenshots

Some YouTube weirdness
Some YouTube weirdness
The automatically generated wallpaper
The automatically generated wallpaper

HTTP response error messages

As part of my webspace I wanted to add some humorous error pages to soften the blow when there’s a problem.

Links

Live page
GitHub repository

Overview

This project works in two distinct phases: first the .htaccess file catches HTTP responses and redirects them before the client’s browser can display an error page, and second the PHP script interprets the redirected URI and displays the relevant message. For whimsy, a random comic from Lore Brand Comics is shown. I’ve loosely followed Lore Sjoeberg’s work since the Brunching Shuttlecocks started in the mid ’90s.

Challenges

Challenge: .htaccess must catch a relative path from anywhere in the domain.
Solution: For some reason this eluded me for years. On receiving an HTTP request error, the .htaccess used to redirect to an absolute path, including the full domain name, which meant that the PHP script didn’t know the original URI. Once I fixed this issue I could provide more useful information for the client. (Resolved)
Challenge: The data for each constituency varies in its content and structure.
Solution: The data contains various parties and candidates which vary from constituency to constituency. As a result the data processing must be able to add an arbitrary list of candidates and parties per constituency. This turned out to be trivial, using the database schema, but did add another layer of complexity to the project. (Resolved)

Screenshot

An error 404 message
An error 404 message

Sliders

I was playing a PuzzelScript game involving sliding blocks that i just couldn’t solve. So I wrote an algorithm to solve it.

Links

Live page
GitHub repository

Overview

The algorithm used is a simple backtracking algorithm to find the optimal solution (the one using the fewest moves.) The user can create an arbitrary puzzle for solution, and then the algorithm attempts to find solutions, giving the user progress updates as it does. Once the solution is found it’s written out for the user, and the solution is animated.

Challenges

Challenge: The algorithm itself had to be written and optimised.
Solution: Although the algorithm was relatively easy to write, there is still at least error, as it fails to find a solution for a problem in 12 steps, instead giving a solution in 13 steps. This error has yet to be found. By optimising a few steps, the algorithm was significantly improved. (Resolved, to be revisited)
Challenge: The user should be able to create their own puzzles.
Solution: This was achieved quite easily using my experience making tile based games. I used all the normal mouse event listeners. (Resolved)
Challenge: The solution is animated.
Solution: It turns out that animating the solution was relatively easy once the graphics had been made. The steps are handled using the normal window.setTimeout method. (Resolved)
Challenge: The algorithm should not max out the user’s CPU for more than 30 seconds.
Solution: Using the user’s CPU for too long causes alerts to be raised in most browsers. The algorithm takes time to sleep as it progresses using the window.setTimeout method to free up the user’s CPU from time to time. (Resolved)
Challenge: The progress is animated.
Solution: I’ve used CSS based progress bars in the past before, but this is the first time I’d animated them. The only hard part here was estimating the number of possible combinations of \(n\) moves, and then scaling the bars accordingly. (Resolved)

Screenshots

An example puzzle, animating the solution
An example puzzle, animating the solution
Dynamic status indicator
Dynamic status indicator
Automatically generated solution
Automatically generated solution

Image fader

This is a simple script that fades an image to a given color and transparency. It’s used to make images for use in web design (for example, a background image that needs to fade to match the background color.)

Links

Live page
GitHub repository

Overview

This was made to edit the background images on my website.

Challenges

Challenge: Due to lack of time it only fades from the top at the moment.
Solution: The other three directions will be added as needed! (To be revisited.)

Screenshot

The image fader, fading the London skyline to red
The image fader, fading the London skyline to red

Circle points

This project was thrown together rather quickly to solve a geometrical problem: How can you analytically find the circle which passes through three given points? The construction simply demonstrates that the method works, and it not intended to give any further information to the user.

Links

Live page 1
GitHub repository

Overview

Finding a circle which passes through \(n\) points is fairly easy to do: simply define a \(\chi^2\) and vary the centre and radius of the circle until the \(\chi^2\) is minimised. I didn’t want to pursue a numerical method because it was to be used in the aDetector project to emulate helix reconstruction, where it could be called hundreds or thousands of times per event. Instead I opted to take triplets of points and estimate the circle properties from the triplets, hence I need a fast algorithm to find the circle that passes through three points.

Challenges

Challenge: The The main challenge was working through the algebra to find a solution.
Solution: It took a while to realise that some GCSE level geometry had the answer- a chord bisector is a diameter, so all I had to two was define two chords using the three points, find their perpendicular bisectors, and fint their point of intersection. (Resolved)
Challenge: Two points which are horizontally aligned lead to infinite gradients.
Solution: This was not addressed in the algorithm. Initially I tried to rotate the points to solve the problem, but this seems like a poor solution. Upon writing this report I realise that only one chord can ever be horizontal, leaving two other non-horizontal chords for use. (Solved, not implemented.)

Screenhshot

The construction of a circle, obtained from finding the point of intersection of the chord bisectors
The construction of a circle, obtained from finding the point of intersection of the chord bisectors

Marble Hornets

Marble Hornets is a web series that currently spans the course of about five years. The premise of the series is that a student film maker started filming an amateur movie in 2006, but abandoned the film due to “unworkable conditions on set”. It quickly turns into a horror series that brings with some novel techniques. However what really inspired me to make this project was the asynchronous storytelling narrative, which requires the viewer to piece together the true chronology based on the context of the videos. I’m an active participant on one of the busiest discussion boards for this series and regularly update this project as each new video is released.

Links

Live page
GitHub repository

Overview

So far, the Marble Hornets project has two main aspects to it. The initial work began with the Marble Hornets player, a collection of youtube videos that are manipulated by the youtube JS API. The user can autoplay all the videos in the series, filter based on many parameters, and even create their own playlists. The player is made so that the user can autoplay everything in chronological order in full screen mode. The data was originally stored in Javascript files, but this has since been moved to XML files to make maintenance and data entry easier and more portable. After creating the player I added a lot of further information including links to the wikia pages, youtube videos and forum threads for each video, as well as the twitter feed, a real world map of filming locations and other interesting links, turning the player into a hub of information.

A previous version of the player was adapted to make the my_dads_tapes player, although I lost interest in the series and stopped updating that player. At some point I intend to automate some of the player manipulation so that a user can create their own player for any youtube account, which would automatically source the relevant information and download the thumbnails.

The Marble Hornets player
The Marble Hornets player

The second aspect of the project is more interesting and challenging, and it is the automated creation of infographic images to help clarify the information known about the series. These files are shared with the community on the forums, and help users discuss the series. Videos are split into scenes, which contains characters and items. The scenes are sorted chronologically (although in many cases the chronological ordering is ambiguous, and the consensus opinion is usually taken in these cases) and then the characters and items are represented by arrows which flow from scene to scene. The scenes are arranged in columns to give the location of the characters, or the owners of the items. Users can create and enter their own XML files to make their own infographics, and automatically dump XML snippets by clicking on the scenes they wish to select. The users can filter by characters, camerapersons, items, and seasons. The scenes are colour coded to represent the sources of the footage.

Part of the master timeline
Part of the master timeline
Part of the (even more complicated) items inforgraphic
Part of the (even more complicated) items inforgraphic

Challenges

Challenge: The web series is told out of order, so one of the biggest problems to solve was sorting the scenes in order, when the order was sometimes ambiguous.
Solution: This was solved by following the fan-made “comprehensive timeline” when in doubt, and sorting the scenes with dates, and in the case of multiple scenes per day, by time. The scenes are assigned timestamps, and the videos are assigned release dates. With this in place, the scenes and videos can then be sorted quickly and easily. (Resolved)
Challenge: The data has to be stored in an organised manner.
Solution: Initially this was solved by declaring objects directly. In order to make the data more portable and for other users to contribute, I wrote and parsed xml files, so that nearly all the data is sorted semantically. One of the infographics keeps track of the exchange of posessions between characters, and this data is not yet accounted for in the xml files. (Resolved, to be revisited)
Challenge: This project required extensive use of the youtube js API.
Solution: Thanks to this project I can now queue up videos, make them autoplay, and use a timer to move between sections of video. (Resolved)
Challenge: The video stills had to respond to the user’s actions.
Solution: The video player in this project allows the user to mouseover the video stills, with dynamic changes of style as the user does so, makng it more aesthetically pleasing. This had to be responsive and with little overhead, and the result is rather pleasing. (Resolved)
Challenge: The timeline has to be laid out properly on the page.
Solution: This means that each element must be given enough space to display all the text and images, and then arranged to give the character arrows sufficient space. This has been achieved using a text wrapping function, and parsing the lists of objects multiple times to ensure sufficient spacing. Editing this code is not a pleasant experience though, it could certianly benefit from some refactoring. (Resolved, to be revisited)
Challenge: The player should allow the user to create a custom playlist
Solution: The user can create a playlst filtering by characters etc, and choosing to play in release order or chronological order. The player also allows the user to watch all scenes in order. (Resolved)
Challenge: There have been many many challenges with this project, so I may add more as they occur to me!
Solution: (Resolved)