It was cool to see my VR software being used by the Dallas Cowboys:
To make this a bit more of a technical post, instead of purely a brag, we use a fair amount of machine learning behind the scenes. We look for similar items in the scene between cameras, to match up and align the seams:
We try to ‘autotune’ all the parameters. We adjust both the seam position and the camera warp to minimize the squared RGB pixel difference in the overlapped areas. This is actually quite computationally expensive, so we run this in CUDA to accelerate it.
A way to make a webpage that lets you have an interactive dialogue by choosing options, like the old Interactive Fiction games.
The input file is very simple:
PRINT Hello!
WAIT ; This is a comment. Wait for a key press or mouse click
PRINT I would like to ask you a question.
PRINTW Please don't be nervous. ; The 'W' means WAIT afterwards
PRINT Are you happy?
PRINT [0] Yes
PRINT [1] Not really...
CALL INPUTINT(0, 1)
IF RESULT == 0
PRINTW I'M SO HAPPY THAT YOU'RE HAPPY!
ELSE
PRINTW Then I am miserable 😦
ENDIF
The challenge is to make a webpage that could read that input, and run it, producing the interactive output shown in the video above.
Perhaps have a think if you don’t know how you would implement this. It is perhaps not so obvious.
Here is the approach that I decided to take. Note that I only wanted to spend a weekend on this, so I took various shortcuts that I wouldn’t recommend if writing for production:
Read in the input file in, using javascript with a Tokenizer and Parser.
Output javascript using the above Parser, effectively making a transpiler from the input language to javascript. Call outside javascript functions to ‘PRINT’ etc.
Use ‘eval’ to run the resulting transpiled code. Yes, eval is evil and this is a case of don’t do what I do.
Use Promises, via async and await, to ‘pause’ and allow interaction. Implement ‘WAIT’, ‘PRINTW’ ‘INPUTINT’ etc functions in javascript with Promises, so that they only resolve when the user has clicked, typed a number etc.
Display output by just appending to a list, and displaying that list in React.
1. Read the file in, using Javascript with a Tokenizer and Parser
I used jison. Although the README blurb says that it is for Context Free Grammars, because it is based on BISON which is likewise, it does actually support context. This was vital because the input file language is not actually a context free grammar.
2. Output javascript using the above Parser, effectively making a transpiler from the input language to javascript
The correct way to do this would be to create an Abstract Syntax Tree, however I didn’t want to take too long on this, and instead simply outputted javascript code as a string.
3. Use ‘eval’ to run the resulting transpiled code.
This is very frowned upon, but this was a weekend project, so….
There is one trick that I used here. I wrap the entire program to be ‘eval’ed like:
async function() { ..... }()
This allows the program code inside to use async and await to wait for input, and the eval is returning a Promise. One minor point – when we use eval to evaluate this, we want to catch errors that the Promise throws, to provide clear feedback to the user if there are problems. E.g.
try {
await eval(program);
} catch(e) { ... }
4. Use Promises, via async and await, to ‘pause’ and allow interaction. Implement ‘WAIT’, ‘PRINTW’ ‘INPUTINT’ etc functions in javascript with Promises, so that they only resolve when the user has clicked, typed a number etc.
async function wait() {
await new Promise(
resolve => cb_setClickListener(() => resolve())
)
}
So we subscribe to a click listener via the callback ‘cb_setClickListener’ and then resolve the promise (and thus resume running the program) when the click is published.
Inside the React page, we now listen for clicks and publish it to the callback:
And likewise for keypresses. (Note, I’ve simplified the code here a bit. In the real code, I pass the keypressed etc, so that INPUTINT can listen to a particular key).
5. Display output by just appending to a list, and displaying that list in React.
This way, when presented with a choice, the user can just click a link instead. I maintain two separate output lists, so that I can disable the links once we are done with them.
Conclusion and notes
It worked very nicely! I further extended this to support variables, entering strings, showing pictures, and so on.
The language is actually a real language, but very niche and used almost exclusively by Japanese. It hasn’t seen much acceptance or use outside of Japan.
I became interested in machine learning working for Nokia. I worked on Nokia’s Z Launcher application for Android. You can scribble a letter (or multiple), and it would recognize it and search for it. The app is available for download in the Play Store.
I worked on the Nokia Z Launcher’s handwriting recognition
Specifically I was tasked with optimizing the speed of the recognition. I don’t know if I can state any specifics on how the character recognition was done, but I will say that I managed to increase the speed of the recognition a hundred fold.
But this recognition was actually a relatively simple task, compared to modern day deep neural networks, but it really whet my appetite to understand more.
When Alpha Go beat Lee Sedol, I knew that I simply must understand Deep Neural Networks.
Below is my journey in understanding, along with my reflective thoughts:
I started off naively implementing an LSTM neural network without any training.
I wanted to have a crude appreciation of the problems before I read about the solutions. My results are documented in my previous post here, but looking back it’s quite embarrassing to read. I don’t regret doing this at all however.
Next I did the Andrew Ng Coursera Machine Learning course. This is an 11 week course in the fundamentals. I completed it along with all the coursework, but by the end I felt that my knowledge was about 20 years out of date. It was really nice to get the fundamentals, but none of the modern discoveries were discussed at all. Nothing about LSTM, or Markov Chains, or dropouts, etc.
The exercises are also all done in Matlab/Octave, which has fallen out of favour and uses a lot of support code. I certainly didn’t feel comfortable in implementing a neural network system from scratch after watching the course.
Passed the Coursera Machine learning course with 97.6% score.
The lecturer, Andrew Ng, was absolutely awesome. My complaints, really, boil down to that I wish the course was twice as long and that I could learn more from him! I now help out in a machine learning chat group and find that most of the questions that people ask about TensorFlow, Theano etc are actually basics that are answered very well by Andrew Ng’s course. I constantly direct people to the course.
Next, Reinforcement Learning. I did a 4 week Udacity course UD820.
This was a done by a pair of teachers that are constantly joking with each other. I first I thought it would annoy me, but I actually really enjoyed the course – they work really well together. They are a lot more experienced and knowledgeable than they pretend to be. (They take it in turns to be ignorant, to play the role of a student).
I really wanted to learn about TensorFlow, so I did a 4 week Udacity Course UD730 Again I did all the coursework for it. I thought the course was pretty good, but it really annoyed me that each video was about 2 minutes long! Resulting in a 30 second pause every 2 minutes while it loaded up the next video. Most frustrating.
At this point, I started reading papers and joined up for the Visual Doom AI competition.
I have come quite far in my own Visual Doom AI implementation, but the vast majority of the work is the ‘setup’ required. For example, I had to fix bugs in their doom engine port to get the built-in AI to work. And it was a fair amount of work to get the game to run nicely with TensorFlow, with mini-batch training, testing and verification stages.
I believe I understand how to properly implement a good AI for this, with the key coming from guided policy search, in a method similar to that pioneered by google for robotic control. (Link is to a keynote at the International Conference on Learning Representations 2016). The idea is to hack the engine to give me accurate internal data of the positions of enemies, walls, health, etc that I can use to train a very simple ‘teacher’. Then use that teacher to supervise a neural network that has only visual information, thus allowing us to train a deep neural network with back-propagation. By alternating between teacher and student, we can converge upon perfect solutions. I hope to write more about this in a proper blog post.
I listened to the entire past two years of the The Talking Machines podcast. It highlighted many areas that I was completely unfamiliar with, and highlighted many things that I knew about but just didn’t realise were important.
I did the Hinton Coursera course on Neural Networks for Machine Learning, which perfectly complemented the Andrew Ng Coursera Course. I recommend these two courses the most, for the foundations. It is about 5 years out of date, but is all about the fundamentals.
I did the Computational Neuroscience course. The first half was interesting and was about, well, neuroscience. But the math lectures were in a slow monotonic tone that really put me straight to sleep. The second half of the course was just a copy of the Andrew Ng course (they even said so), so I just skipped all the lectures and did the exams with no problems. I really liked that they let you do the homework in python. It is easier in matlab, but I really wanted to improve my python datascience skills. The exams took way way longer than their predicted times. They would have 20 questions, requiring you to download, run and modify 4 programs, and say that you should be able to do it in an hour!
I completed the IBM Artificial Intelligence class. This is the second most expensive AI class that I’ve done, at $1600 plus $50 for the book. I was really not at all impressed by it. Most of the videos are 1 minute long, or less, each. Which means that I’m spending half the time waiting for the next video to load up. The main lecturer gets on my own nerves – he wears a brightly colored Google Glass for no apparent reason other than to show it off, and speaks in the most patronizing way. You get praised continually for signing up to the course at the start. It’s overly specialized. You use their specific libraries which aren’t at all production ready, and you use toy data with a tiny number of training examples. (e.g. trying to train for the gesture ‘toy’ with a single training example!). Contrast this against the Google Self Driving course:
The Google Self Driving course, which is $2400. This is much better than the IBM course. The main difference is that they’ve made the theme self driving cars, but you do it all in TensorFlow and you learn generic techniques that could be applied to any machine learning field. You quickly produce code that could be easily made production ready. You work with large realistic data, with large realistic neural networks, and they teach you use the Amazon AWS servers to train the data with. The result is code that can be (and literally is!) deployed to a real car.
I needed to rename a whole load of files that were in Japanese. So I wrote a python program that translates the filename using google translate.
It’s not at all fancy, just run it and pass the filenames to translate and rename.
E.g.:
$ ls
こんにちは世界.png
$ sudo pip3 install googletrans
$ translate_rename.py こんにちは世界.png
こんにちは世界.png -> Hello_World.png
$ ls
Hello_World.png
#!/usr/bin/python3
import sys, re, os
from googletrans import Translator
translator = Translator()
sourceLanguage = 'ja'
destLanguage = 'en'
# Set to false to actually rename the files
dryRun = True
def translate_and_rename(filename):
filenameSplit = filename.rsplit('.',1)
translated = translator.translate(filenameSplit[0], src=sourceLanguage, dest=destLanguage).text
translated = re.sub( '[^a-zA-Z0-9.]+', '_', translated).strip().title()
if len(filenameSplit) > 1:
translated += '.' + filenameSplit[1]
if filename == translated:
print(filename, ' (unchanged)')
else:
print(filename, " -> ", translated)
if not dryRun:
os.rename(filename, translated)
def main(argv):
if len(argv) == 1:
print("Need to pass filenames to translate and rename")
for x in argv[1:]:
translate_and_rename(x)
if dryRun:
print()
print(" Dry run only - no actual changes made ")
print()
print("Edit this file and set DryRun to True")
if __name__ == "__main__":
main(sys.argv)
Since many operating systemsmap physical memory, kernel processes, and other running user space processes into the address space of every process, Meltdown effectively makes it possible for a rogue process to read any physical, kernel or other processes’ mapped memory—regardless of whether it should be able to do so. Defenses against Meltdown would require avoiding the use of memory mapping in a manner vulnerable to such exploits (i.e. a software-based solution) or avoidance of the underlying race condition (i.e. a modification to the CPUs’ microcode and/or execution path).
This separation of user and kernel memory space is exactly what I worked on from 2012 to 2014 on behalf on Deutsch Telekom using the L4 hypervisor:
The idea was to give each service its own separate memory space, designing in a way such that you assume that the main OS has been compromised and is not trustworthy (e.g. because of the Meltdown bug). I personally worked on the graphics driver – splitting the kernel graphics driver into two parts – one side for the app to talk to and has to be considered compromised, and one side that actually talks to the hardware.
Here’s my work in action:
Yes, I did actually use Angry Birds as my test. Fully hardware accelerated too 🙂
Unfortunately the problem was that it took too long to port each phone across. It took me a year to port across graphics driver changes, and a similar time for my colleagues to do the other drivers. And then another year for it to actually hit the market. The result is that the phone was always over 2 years out of date by the time it hit the market, which is a long time in mobile phone times.
Still, our software would be immune to this type of bug, and that’s kinda cool. Even if it did fail commercially 😉
I love TypeScript. I use it whenever I can. That said, sometimes it can be… interesting. Today, out of the blue, I got the typescript error in code that used to work:
[06:53:30] typescript: src/mycode.ts, line: 57
Property 'video' does not exist on type 'number | (<U>(callbackfn: (value: Page, index: number,
array: Page[]) => U, thisA...'. Property 'video' does not exist on type 'number'.
The problem is that s.video || s.videoEmbedded isn’t returning a boolean. It’s return a truthy value, but not a boolean. And the lodash typescript developers made a change 1 month ago that meant that filter() would only accept booleans, not any truthy value. And the lodash typescript developers are finding that fixing this becomes very complicated and complex. See the full conversation here:
