Older blog entries for glyph (starting at number 17)

Connecting To A Server On The Internet [Nightmare Difficulty]

The Setup

I recently switched internet service providers, from Comcast to WebPass. Comcast had been progressively upgrading our connection, and I honestly didn’t have any complaints, but WebPass is four times faster. Eight times faster, maybe? It’s so fast that I could demonstrate that my router hardware was the bottleneck and I needed to buy something physically capable of more bits per second. There’s gigabit Ethernet coming straight out of my wall, no modem required. So that’s exciting. The not-exciting part is that WebPass uses what is euphemistically called “carrier grade” NAT to deliver IPv4 service. This means I have no public IPv4 address at all.

This is exactly the grim meathook future that I predicted over a decade ago.

One interesting consequence of having no public IPv4 address is that I could no longer make inbound connections to my home network. When I’m roving around with my laptop, I like to be able to check in with my home servers. Plus, I’m like, a networking dude, or something, so I should know how this stuff works, and being able to connect to networks is a good first step to doing interesting things with them.

The Gear

I’m using OpenWrt (Chaos Calmer) firmware on my home router (although this also works on Barrier Breaker). If you configure its built-in IPv6 stuff naively, connecting to your home network is no problem at all! Every device on your network suddenly has its own public IP address on the Internet.

The Problem

In case it’s not clear: don’t do that. I can guarantee you that all of your computers and some things which aren’t obviously computers (game consoles, “smart” scales, various handheld devices, your “cloud” thermostat, probably your refrigerator, maybe some knives or guns that you didn’t know had internet access: anything that can connect to WiFi) are listening on a surprising array of ports, and every single one of them has some interesting security vulnerability which an attacker could potentially use to do bad stuff. You should of course keep your operating systems and device firmware updated to make sure you’ve always got the latest security patches, but even if you’re perfect at that, many devices simply don’t get updates. So the implicit firewall that NAT gave us is still important; except now we need it as an explicit firewall that actually blocks traffic, and not as an accident of the fact that devices don’t have their own externally-available addresses.

The Requirements

Given these parameters, the desired state of my router (and, I hope, some of yours) is:

  1. Explicitly deny all inbound IPv6 connections to all devices that aren’t expecting them, such as embedded devices and laptops.
  2. Explicitly allow all inbound IPv6 connections to specific devices and ports that are expecting them, like home servers.

Casual Difficulty: Rejecting Incoming Traffic

OpenWrt has a web interface, called “LuCI”, which has a nice firewall configuration application. You’ll want to use it to reject IPv6 inputs, like so:

Reject IPv6 Inputs

It also has a “port forwarding” interface, but forwarding a port is something that only really makes sense in an IPv4 NAT scenario. In an IPv6 scenario you don’t need to “forward” since you already have your own address. Accordingly, the interface populates the various IP address selectors with only your internal IPv4 addresses, and doesn’t work on IPv6 interfaces on the router.

Now we get to the really tricky part: allowing inbound traffic to that known device.

Normal Difficulty: Home Server Naming

First, we need to be able to identify that device. Since residential IPv6 prefix allocations are generally temporary, you can’t rely on your IP address to stay the same forever. This means we need to have some kind of dynamic DNS that updates periodically.

Personally, I have accomplished this with a little script using Apache libcloud that I have run in a crontab. The key part of that script is this command line, which works only on Linux:

1
ip -o -d -6 addr show primary scope global br0

that enumerates all globally-routable IPv6 addresses on a given interface (where the interface is br0 in that script, but may be something else on your computer; most likely eth0).

In my crontab, I have something like this:

1
2
@hourly ./.virtualenvs/libcloud/bin/python .../update_home_dns.py \
    rackspace_us my.dns.user home.mydomain.example.com

And I ran the script once manually to initialize the keyring entry with my API key in it.

Note: I would strongly recommend creating a restricted user that only has access to DNS records for this, if you’re doing it on a general cloud provider like Rackspace or Amazon. On Rackspace, this can be accomplished by going to the “Account” menu at the top right, “User Management”, “Create User”, and selecting the “Custom” radio button in the “Product Access” section. Then, log in as that user, go to the “Account Settings”, and copy the API key.

Nightmare Difficulty: ip6tables suffix matching

Now we can access the address of home.mydomain.example.com from the public Internet, but it doesn’t do us much good; it’s still firewalled off. And since the prefix allocation changes regularly, we can’t simply open up a port in the firewall with a hard-coded address as you might in a regular data-center environment.

However, we can use a quirk of IPv6 addressing to our advantage.

In case you’re not already familiar, let me briefly explain the structure of IPv6 addresses. An IPv6 address is a 128-bit number. That is a really, really big number, which means a really, really big set of addresses; unlike IPv4, your ISP doesn’t need to jealously hoard them and hand them out sparingly, one at a time.

One of the features of IPv6 is the ability to allocate your own individual addresses. When your provider gives you an IPv6 address, they give yo a so-called “prefix allocation”, which is the first half or so - 64 bits - of an address. Devices on your network may then generate their own addresses that begin with those 64 bits.

Devices on your network then decide on their addresses in one of two ways:

  1. If they’re making an outgoing connection, they choose a random 64-bit number and use that as the suffix; a so-called “temporary” address.
  2. If they’re listening for incoming connections, they generate a 64-bit number based on their Ethernet card’s MAC address; a so-called “primary” address.

It is this primary address that the above ip command will print out. (If you see two addresses, and one starts with an f, it’s the other one; this post is already long enough without explaining the significance of that other address.)

Within iptables, the --source and --destination (or -s and -d) matching options are documented as having the syntax “address[/mask]”. The “mask” there is further documented as “... either an ... network mask (for iptables) or a plain number, specifying the number of 1’s at the left side of the network mask.” Normally, sources and destinations are matching network prefixes, because that’s what you would normally want to allow or deny traffic from: a prefix representing a network which in turn represents a specific group of computers. But, given that the IPv6 address for an interface on a given Ethernet card with a given MAC address will have a stable suffix derived from that MAC address, and a prefix who changes based on your current (dynamic) prefix allocation, we can write an ip6tables rule to allow connections based on the suffix by constructing an address that has a mask matching the last 64 bits of the address.

So, let’s say your ip command, above, output an address like 2001:1111:2222:3333:1234:5678:abcd:ef01. This means the stable part of your address is that trailing 1234:5678:abcd:ef01. And that means we want iptables to allow inbound traffic to a specific port. To allow incoming HTTPS on port 443, in the OpenWRT control panel, go to Network → Firewall, click the “Custom Rules” tab, and input a line something like this:

1
2
3
4
5
6
ip6tables -A forwarding_rule \
    -d ::1234:5678:abcd:ef01/::ffff:ffff:ffff:ffff \
    -p tcp -m tcp \
    --dport 443 \
    -m comment --comment "Inbound-HTTPS" \
    -o br-lan -j ACCEPT

This says that we are going to append to the forwarding_rule chain (a chain specific to OpenWRT), a rule matching packets whose destination is our suffix-matching mask with the suffix of the server we want to forward to, for the TCP protocol using the tcp matcher, whose destination port is 443, giving it a comment of “Inbound-HTTPS” so it’s identifiable when we’re looking at the firewall later, sending it out the br-lan interface (again, somewhat specific to OpenWRT), and jumping to the ACCEPT target, because we should let this through.

Game Over

Congratulations! You can now connect to a server on a port, in the wonderful world of tomorrow. I hope you look forward as much as I do to the day when this blog post will be totally irrelevant, and all home routers will simply come with a little UI that lets you connect a little picture of a device to a little picture of a cloud representing the internet, and maybe type a port number in.

Syndicated 2015-09-13 06:30:00 from Deciphering Glyph

Software You Can Use

Python has a big problem. While it’s easy and fun to produce software in Python, it’s hard to produce software that people - especially laypeople who are not professional software developers - can use.

In the modern software ecosystem, there are a few places that you might want to use a program:

  1. On a server, by loading a web page.
  2. On a web page, by running it in your browser.
  3. As a command-line tool, on Mac,
  4. ... Windows,
  5. ... or Linux.
  6. As a desktop application, on Mac,
  7. ... Windows,
  8. ... or Linux.
  9. As a mobile application, on iOS,
  10. ... or Android,
  11. ... Or Windows Phone. (Just kidding.)

Out of these 10 scenarios, Python currently has half of a good deployment story for one of them: running an application on a server, as a back-end. This is a serious problem for the future of Python and one we need to figure out how to face as a community.

Even the “good” deployment story is somewhat convoluted, as you need to know about at least some Linux distribution’s package manager, and native dependencies, and Pip, and virtualenv, and wheels, and probably docker too.

If you want to run a Python application in your browser, your best bet right now is probably Brython. However, brython is still in its infancy, and basic faciltiies like preparing your code for production to achieve acceptable start-up performance, and an explanation of how to use libraries are missing. With big chunks like that missing it’s hard to advocate for Brython’s use in production.

Moving on to scenario 3, this may be one of the best-supported configurations; py2app actually works surprisingly well. But it’s still incredibly confusing for new users. Which native objects (dylibs and frameworks and data files) to bundle are options to py2app itself, and not something that can be handled automatically by libraries. So if, for example, PyGame depends on SDL.framework or libSDL.dylib, you as an application developer need to understand how to figure that out and specify that list.

On Windows, the situation gets worse. To work as a Windows exectuable, you need to bundle the Python interpreter, but unlike in an OS X application, you can’t just copy in a whole directory. So you end up needing a tool like PyInstaller or cx_Freeze. PyInstaller hasn’t seen a release in the last 2 years; it doesn’t support Python 3. It also doesn’t work: if I try to package the most basic Twisted program possible, with pyinstaller 2.1 I get “no module named zope.interface”, and if I try to package it with pyinstaller trunk, I get “no module named itertools”. cx_Freeze similarly can’t figure out how to include zope.interface no matter what I tell it to do. This problem isn’t specific to libraries that I use; most Python projects will run into it.

py2exe, on the other hand, only supports Python 3.3+, and so is unusable with a lot of important python libraries.

For a GUI application for Linux, you might have some small hope of building a distro-specific package that users could install, but that would involve using a distro-specific toolchain that had nothing to do with Python, and you need to repeat that work for Debian, Ubuntu, Fedora, and whatever other distros you want to support.

All of these same tools are what I would use to build a stand-alone command-line executable for Windows, Mac, or Linux, and they all break down in similar ways.

In the mobile space, there is absolutely zero tooling included with the language to even get started there. It might be possible to use Kivy to get a build onto iOS or Android. I haven’t had an opportunity to test those. But they still require you to install Homebrew, and a C compiler, and a whole bunch of fairly specific platform tooling to get started, and there are lots of different ways that can go wrong.

So how do other languages stack up?

  1. In JavaScript, if you want an application in the browser, it’s as simple as ... writing some JavaScript.
  2. In JavaScript, if you want a desktop application, you can just grab Electron and be up and running in a few minutes.
  3. In JavaScript, if you want a command-line UNIX tool, you can grab nar and build something self-contained almost immediately.
  4. And of course, in Go, there’s no way to get anything but a fully functional self-contained executable at the end of the build process. Everything is fully redistributable by default.

As a community, Python needs a clear, well-documented, well-supported, modern way to produce build artifacts that are easy to create and easy to share. We need to have this for all popular platforms and the browser. This is a tricky problem: it requires knowledge of lots of fiddly build details.

This wheel has been re-invented, poorly, a dozen or so times. My list above was just a subset. In addition to py2app, py2exe, pyinstaller, cx_Freeze, and the Kivy bundling tools, we’ve also got terrarium, bbfreeze (which is unmaintained), pipsi, pex, and probably some others I don’t know about.

In order to compete with JavaScript and Go for developers’ attention, Python must be able to become an implementation detail and disappear when the user is running the program. This means that some of these tools (terrarium, pipsi, pex) are not suitable for this purpose because they are envelopes for deployment into an environment with an installed Python interpreter.

All of the tools I’m aware of that are trying to provide fully self-contained execution, though (pyinstaller, cx_freeze, bb-freeze, py2app) are poorly designed because they value optimized distribution size over actually working by default. Rather than reading setuptools metadata and discovering the full set of dependencies which have been declared to be required, all of these tools use weird AST-parsing heuristics and buggy path-traversal hacks to try to construct a guess as to the minimal set of files that might be required, then require the poor application developer to fill in the gaps. This means none of them work with namespace packages, none of them work properly with plugin systems or runtime configuration systems; generally, they don’t work correctly with late binding, which is one of Python’s greatest strengths. Of course, a full Python interpreter with the whole standard library is quite large. If we had a tool that worked well but produced very large executables, we could of course start adding an “optimized mode” to try to crunch things down for production.

And all this is to say nothing of the insanely intricate and detailed knowledge that every Python programmer eventually acquires about the C runtime semantics of their chosen platform. When a C compiler is required but missing, most tools still just emit tracebacks. When a shared library goes missing dues to an OS upgrade or package removal, you just see whatever the dynamic linker thinks to report, no explanation of how to fix it or what to do next.

The Python packaging ecosystem has made great strides in the last few years; Pip, in particular, has gone from a buggy and insecure mess to a mostly workable software delivery mechanism for developers. There are still bugs, but they are getting dealt with at a reasonable clip. However, Pip only delivers software to developers, and still requires you to have a Python runtime, a build environment, and tricky command-line tools to get things in place for development. The Python community has effectively no tools to deliver software to users.

To sum up, we need a tool which:

  1. works by default, including with “tricky” packages with namespace packages, data files, and native dependencies
  2. produces useful, actionable error messages when something is missing and the build can’t be completed (like “you don’t have a C compiler installed” or “you need to install Homewbrew and then brew install openssl”)
  3. can produce both command-line and GUI executables for the mac, windows, and linux (and, for bonus points, a web browser)

The bad news is that I don’t have the time to start this project myself, and I’m not sure who does. The worse news is that every day we don’t have this, more and more people are re-writing their user-facing tools and applications in JavaScript or Go or Swift or Java, to suit their target platform, because it is honestly easier to learn an entirely new programming language and toolchain, and rewrite an entire application than to figure out how to build a self-contained executable in Python right now.

The good news, though, is that it’s a simple matter of programming, and that all the core technologies for doing all the really hard things that need to be done (pip, and zipimport and macholib, for example) already exist. It’s just a simple matter of programming: wiring together the metadata from setuptools, determining native dependencies with something like otool or ldd (or whatever the equivalent is on Windows, I still haven’t figured that out myself), pulling them all into a bundle, tacking the Python interpreter on.

Syndicated 2015-09-12 09:24:00 from Deciphering Glyph

Sometimes, You Just Want A Button

I like making computers do stuff. I find that getting a computer to do what I want it to produces a tremendously empowering feeling. I think Python is a great language to use to tell computers what to do.

In my experience, the most empowering feeling (albeit often, admittedly, not the most useful or practical one) is making one’s own computer do something cool. But due to various historical accidents, the practical bent of the Python community mainly means that we get “server-side” or “cloud” computers to do things: in other words, “other people’s computers”.

If you, like me, are a metric standard tech industry hipster, “your own computer” means “Mac OS X”. Many of us have written little command-line tools to automate things, and thank goodness OS X is enough of a UNIX that that works nicely - and it’s often good enough. But sometimes, you just want to press a button and have a thing happen; sometimes a character grid isn’t an expressive enough output device. Sometimes you want to be able to take that same logical core, or some useful open source code, that you would use in the cloud, and instead, put it into an app. Sometimes you want to be able to learn about writing desktop apps while leveraging your existing portfolio of skills. I have done some of this, and intend to do more of it at work, and so I’d like to share with you some things I’ve learned about how to do it.

Back when I was a Linux user, this was a fairly simple proposition. All the world was GTK+ back then, in the blissful interlude between the ancient inconsistent mess when everything was Xt or Motif or Tk or Swing or XForms, and the modern inconsistent mess where everything is Qt or WxWidgets or Swing or some WebKit container with its own pile of gross stylesheet fruit salad.

If you had a little script that you wanted to put a GUI on back then, the process for getting started was apt-get install python-gtk and then just to do something like

1
2
3
4
5
import gtk
w = gtk.Window("My Window")
b = gtk.Button("My Button")
w.add(b)
gtk.main()

and you were pretty much off to the races. If you wanted to, you could load a UI file that you made with glade, if you had some complicated fancy stuff to display, but that was optional and reasonably straightforward to use. I used to do that all the time, for various personal computing tasks. Of course, if I did that today, I’m sure they’d all laugh at me.

So today I’d like to show you how to do the same sort of thing with OS X.

To be clear, this is not a tutorial in Objective C, or a deep dive into Mac application programming. I’m going to make some assumptions about your skills: you’re a relatively experienced Python programmer, you’ve done some introductory Xcode material like the Temperature Converter tutorial, and you either already know or don’t mind figuring out the PyObjC bridged method naming conventions on your own.

If you are starting from scratch, and you don’t have any code yet, and you want to work in Objective C or Swift, the modern Mac development experience is even smoother than what I just described. You don’t need to install any “packages”, just Xcode, and you have a running app before you’ve written a single line of code, because it will give you a working-by-default template.

The problem is, if you’re a Python developer just trying to make a little utility for your own use, once you’ve got this lovely Objective C or Swift application ready for you to populate with interesting stuff, it’s a really confusing challenge to get your Python code in there. You can drag the Python.framework from homebrew into Xcode and then start trying to call some C functions like PyRun_SimpleString to get your program bootstrapped, but then you start running into tons of weird build issues. How do you copy your scripts to the app bundle? How do you invoke distutils? What about shared libraries that you’ve linked against? It’s hard enough to try to jam anything like setuptools or pip into the Xcode build system that you might as well give up and rewrite the logic; it’ll be less effort.

If you’re a Python developer you probably expect tools like virtualenv and pip to “just work”. You expect to be able to put a file into a folder and then import it without necessarily writing a whole build toolchain first. And if you haven’t worked with it a lot, you probably find Xcode’s UI bewildering. Not to mention the fact that you probably already have a text editor that you like already, and don’t want to spend a bunch of time coming up to speed on a new one just to display a button.

Luckily, of course, there’s pyobjc, which lets you write your whole application in Python, skipping the whole Xcode / Objective C / Swift side-show and just add a little UI logic. The problem I’m trying to address here is that “just adding a little UI logic” (rather than designing your program from the ground up as an App) in the Cocoa / ObjC universe is famously and maddeningly obscure. gtk.Window() is a reasonably straightforward first function to call; [[NSWindow alloc] initWithContentRect:styleMask:backing:defer:] not so much.

This is not even to mention the fact that all the documentation, all the tutorials, and all the community resources pretty much expect you to be working with .xibs or storyboards inside the Interface Builder portion of Xcode, and you’re really on your own if you are trying to do everything outside of that. py2app can help with some of the build issues, but it has its own problems you probably don’t want to be tackling if you’re just getting started; it’ll sap all your energy for actually coding.

I should mention before we finally dive in here that if you really just want to display a button, a text area, maybe a few fields, Toga is probably a better option for you than the route that I'm describing in this post; it's certainly a lot less work. What I am assuming here is that you want to be able to present a button at first, but gradually go on to mess around with arbitrary other OS X native APIs to experiment with the things you can do with a desktop that you can't do in the cloud, like displaying local notifications, tracking your location, recording the screen, controlling other applications, and so on.

So, what is an enterprising developer - who is still lazy enough for my rhetorical purposes here - to do? Surprisingly enough, I have a suggestion!

First, you want to make a new, empty Xcode project. So fire up Xcode, go to File, New, Project, and then:

Just-A-Button-1

Just-A-Button-2

Go ahead and give it a Git repository:

Just-A-Button-3

Now that you’ve got a shiny new blank project, you’ll need to create two resources in it: one, a user interface document, and the other, a Python file. So select File, New, and then choose OS X, User Interface, Empty:

Just-A-Button-4

I’m going to call this “MyUI”, but you can call it whatever:

Just-A-Button-5

As you can see, this creates a MyUI.xib file in your project, with nothing in it.

Just-A-Button-6

We want to put a window and a button into it, so, let’s start with that. Search for “window” in the bottom right, and drag the “Window” item onto the canvas in the middle:

Just-A-Button-7

Just-A-Button-8

Now we’ve got a UI file which contains a window. How can we display it? Normally, Xcode sets all this up for you, creating an application bundle, a build step to compile the .xib into a .nib, to copy it into the appropriate location, and code to load it for you. But, as we’ve discussed above, we’re too lazy for all that. Instead, we’re going to create a Python script that compiles the .xib automatically and then loads it.

You can do this with your favorite text editor. The relevant program looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
import os
os.system("ibtool MyUI.xib --compile MyUI.nib")

from Foundation import NSData
from AppKit import NSNib, NSApplication

with open("MyUI.nib", "rb") as nib_file:
    octets = nib_file.read()
    nib_data = NSData.dataWithBytes_length_(octets, len(octets))

(NSNib.alloc().initWithNibData_bundle_(nib_data, None)
 .instantiateWithOwner_topLevelObjects_(None, None))

from PyObjCTools.AppHelper import runEventLoop
runEventLoop()

Breaking this down one step at a time, what it’s doing is:

1
2
import os
os.system("ibtool MyUI.xib --compile MyUI.nib")

We run Interface Builder Tool, or ibtool, to convert the xib, which is a version-control friendly, XML document that Interface Builder can load, into a nib, which is a binary blob that AppKit can load at runtime. We don’t want to add a manual build step, so for now we can just have this script do its own building as soon as it runs.

Next, we need to load the data we just compiled:

1
2
3
with open("MyUI.nib", "rb") as nib_file:
    octets = nib_file.read()
    nib_data = NSData.dataWithBytes_length_(octets, len(octets))

This needs to be loaded into an NSData because that’s how AppKit itself wants it prepared.

Finally, it’s time to load up that window we just drew:

1
2
(NSNib.alloc().initWithNibData_bundle_(nib_data, None)
 .instantiateWithOwner_topLevelObjects_(None, None))

This loads an NSNib (the “ib” in its name also refers to “Interface Builder”) with the init... method, and then creates all the objects inside it - in this case, just our Window object - with the instantiate... method. (We don’t care about the bundle to use, or the owner of the file, or the top level objects in the file yet, so we are just leaving those all as None intentionally.)

Finally, runEventLoop() just runs the event loop, allowing things to be displayed.

Now, in a terminal, you can run this program by creating a virtualenv, and doing pip install pyobjc-framework-Cocoa, and then python button.py. You should see your window pop up - although it will not take focus. Congratulations, you’ve made a window pop up!

One minor annoyance: you’re probably used to interrupting programs with ^C on the command line. In this case, the PyObjC helpers catch that signal, so instead you will need to use ^\ to hard-kill it until we can hook up some kind of “quit” functionality. This may cause crash dialogs to pop up if you don’t use virtualenv; you can just ignore them.

Of course, now that we’ve got a window, we probably want to do something with it, and this is where things get tricky. First, let’s just create a button; drag the button onto your window, and save the .xib:

Just-A-Button-9

And now, the moment of truth: how do we make clicking that button do anything? If you’ve ever done any Xcode tutorials or written ObjC code, you know that this is where things get tricky: you need to control-drag an action to a selector. It figures out which selectors are available by magically knowing things about your code, and you can’t just click on a thing in the interface-creation component and say “trust me, this method exists”. Luckily, although the level of documentation for it these days makes it on par with an easter egg, Xcode has support for doing this with Python classes just as it does with Objective C or Swift classes.1

First though, we’ll need to tell Xcode our Python file exists. Go to the “File” menu, and “Add files to...”, and then select your Python file:

Just-A-Button-10

Here’s the best part: you don’t need to use Xcode’s editor at all; Xcode will watch that file for changes. So keep using your favorite text editor and change button.py to look like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
import os
os.system("ibtool MyUI.xib --compile MyUI.nib")

from objc import IBAction
from Foundation import NSObject, NSData
from AppKit import NSNib, NSApplication

class Clicker(NSObject):
    @IBAction
    def clickMe_(self, sender):
        print("Clicked!")

with open("MyUI.nib", "rb") as nib_file:
    octets = nib_file.read()
    nib_data = NSData.dataWithBytes_length_(octets, len(octets))

the_clicker = Clicker.alloc().init()

(NSNib.alloc().initWithNibData_bundle_(nib_data, None)
 .instantiateWithOwner_topLevelObjects_(the_clicker, None))

from PyObjCTools.AppHelper import runEventLoop
runEventLoop()

In other words, add a Clicker subclass of NSObject, give it a clickMe_ method decorated by objc.IBAction, taking one argument, and then make it do something you can see, like print something. Then, make a global instance of it, and pass it as the owner parameter to NSNib.

At this point it would probably be good to explain a little about what the “file’s owner” is and how loading nibs works.

When you instantiate a Nib in AppKit, you are creating a collection of graphical objects, connected to an object in your program that you construct. In other words, if you had a program that displayed a Person, you’d have a Person.nib and a Person class, and each time you wanted to show a Person you’d instantiate the Nib again with the new Person as the owner of that Nib. In the interface builder, this is represented by the “file’s owner” placeholder.

I am explaining this because if you’re interested in reading this article, you’ve probably been interested enough in Mac programming to do something like the aforementioned Temperature Converter tutorial, but such tutorials almost universally just use the single default “main” nib that gets loaded when the application launches, so although they show you how to do many different things with UI elements, it’s not clear how the elements got there. This, here, is how you make new UI elements get there in the first place.

Back to our clicker example: now that we have a class with a method, we need to tell Xcode that the clicking the button should call that method. So what we’re going to tell Xcode is that we expect this Nib to be owned by an instance of Clicker. To do this, go to MyUI.xib and select the “File’s Owner” (the thing that looks like a transparent cube), to to the “Identity Inspector” (the tiny icon that looks like a driver’s license on the right) and type “Clicker” in the “Class” field at the top.

If you’ve properly added button.py to your project and declared that class (as an NSObject), it should automatically complete as you start typing the name:

Just-A-Button-11

Now you need to connect your clickMe action to the button you already created. If you’ve properly declared your method as an IBAction, you should see it in the list of “received actions” in the “Connections Inspector” of the File’s Owner (the tiny icon that looks like a right-pointing arrow in a circle):

Just-A-Button-12

Drag from the circle to the right of the clickMe: method there to the button you’ve created, and you should see the connection get formed:

Just-A-Button-13

If you save your xib at this point and re-run your python file, you should be able to click on the button and see something happen.

Finally, we want to be able to not just get inputs from the GUI, but also produce outputs. To do this, we want to populate an outlet on our Clicker class with a pointer to an object in the Nib. We can do this by declaring a variable as an objc.IBOutlet(); simply add a from objc import IBOutlet, and change Clicker to read:

1
2
3
4
5
class Clicker(NSObject):
    label = IBOutlet()
    @IBAction
    def clickMe_(self, sender):
        self.label.setStringValue_(u"\N{CHECK MARK}")

In case you’re wondering where setStringValue_ comes from, it’s a method on NSTextField, since labels are NSTextFields.

Then we can place a label into our xib; and we can see it is in fact an NSTextField in the Identity Inspector:

Just-A-Button-14

I’ve pre-filled mine out with a unicode “BALLOT X” character, for style points.

Then, we just need to make sure that the label attribute of Clicker actually points at this value; once again, select the File’s Owner, the Connections Inspector, and (if you declared your IBOutlet correctly), you should see a new “label” outlet. Drag the little circle to the right of that outlet, to the newly-created label object, and you should see the linkage in the same way the action was linked:

Just-A-Button-15

And there you have it! Now you have an application that can open a window, take input, and display output.

This is, as should be clear by now, not really the preferred way of making an application for OS X. There’s no app bundle, so there’s nothing to code-sign. You’ll get weird behavior in certain cases; for example, as you’ve probably already noticed, the window doesn’t come to the front when you launch the app like you might expect it to. But, if you’re a Python programmer, this should provide you with a quick scratch pad where you can test ideas, learn about how interface builder works, and glue a UI to existing Python code quickly before wrestling with complex integration and distribution issues.

Speaking of interfacing with existing Python code, of course you wouldn’t come to this blog and expect to get technical content without just a little bit of Twisted in it. So here’s how you hook up Twisted to an OS X GUI: instead of runEventLoop, you need to run your application like this:

1
2
3
4
5
6
7
# Before importing anything else...
from PyObjCTools.AppHelper import runEventLoop
from twisted.internet.cfreactor import install
reactor = install(runner=runEventLoop)

# ... then later, when you want to run it ...
reactor.run()

In your virtualenv, you’ll want to pip install twisted[osx_platform] to get all the goodies for OS X, including the GUI integration. Since all Twisted callbacks run on the main UI thread, you don’t need to know anything special to do stuff; you can call methods on whatever UI objects you have handy to make changes to them.

Finally, although I definitely don’t have room in this post to talk about all the edge cases here, I will address two particularly annoying ones; often if you’re writing a little app like this, you really want it to take keyboard focus, and by default this window will come up in the background. To fix that, you can do this right before starting the main loop:

1
2
3
4
from AppKit import NSApplication, NSApplicationActivationPolicyAccessory
app = NSApplication.sharedApplication()
app.setActivationPolicy_(NSApplicationActivationPolicyAccessory)
app.activateIgnoringOtherApps_(True)

And also, closing the window won’t quit the application, which might be pretty annoying if you want to get back to using your terminal, so a quick fix for that is:

1
2
3
4
class QuitWhenClosed(NSObject):
    def applicationShouldTerminateAfterLastWindowClosed_(self, app):
        return True
app.setDelegate_(QuitWhenClosed.alloc().init().retain())

(the “retain” is necessary because ObjC is not a garbage collected language, and app.delegate is a weak reference, so the QuitWhenClosed would be immediately freed (and there would be a later crash) if you didn’t hold it in a global variable or call retain() on it.)

You might have other problems with this technique, and this post definitely can’t fit solutions to all of them, but now that you can load a nib, create classes that interface builder can understand, and run a Twisted main loop, you should be able to use the instructions in other tutorials and resources relatively straightforwardly.

Happy hacking!


(Thanks to everyone who helped with this post. Of course my errors are entirely my own, but thanks especially to Ronald Oussoren for his tireless work on pyObjC and py2app over the years, as well as to Mark Eichin and Amber Brown for some proofreading and feedback before this was posted.)


  1. If it didn’t, we could get Xcode to produce the appropriate XML by simply writing appropriately-shaped classes in Objective C - or possibly Swift - and then never bothering to build them, since all Xcode cares about in this part of the process is that it can see the source. My understanding is that’s how it worked when PyObjC was first developed, and it might always become necessary again if Xcode ever stops supporting Python. I have no idea if that’s likely, but unfortunately it seems clear Python isn’t a very popular language for mac apps. 

Syndicated 2015-07-17 07:52:00 from Deciphering Glyph

Sorry I Unfollowed You

Since Alex Gaynor wrote his seminal thinkpiece on the subject, “I Hope Twitter Goes Away”, I’ve been wrestling to define my relationship to this often problematic product.

On the one hand, Twitter has provided me with delightful interactions with human beings who I would not otherwise have had the opportunity to meet or interact with. If you are the sort of person who likes following people, four suggestions I’d make on that front are Melissa 🔔, Gary Bernhardt, Eevee and Matt Blaze, all of whom have blogs but none of whom I would have discovered without Twitter.

Twitter has also allowed me to reach a larger audience with my writing than I otherwise would have been able to. Lots of people click on links to this blog from Twitter either from following me directly or from a retweet. (Thank you, retweeters, one and all.)

On the other hand, the effect of using Twitter on my productivity is like having a constant, low-grade headache. While Twitter has never been a particularly bad distraction as measured by hours spent on it (I keep metrics on that, and it’s rarely even in the top 10), I feel like consulting Twitter is something I do when I am stuck, or having to think about something hard. “I’ll just check Twitter” is an easy way to “take a break” right at the moment that I ought to be thinking harder, eliminating distractions, mustering my will to focus.

This has been particularly stark for me as I’ve been trying to get some real writing done over the last couple of weeks and have been consistently drawing a blank. Given that I have a deadline coming up on Wednesday and another next Monday, something had to give.

Or, as Joss Whedon put it, when he quit Twitter:

If I’m going to start writing again, I have to go to the quiet place, and this is the least quiet place I’ve ever been in my life.

I’m an introvert, and using Twitter is more like being at a gigantic, awkward party all the time than any other online space I’ve ever been in.

There’s an irony here. Mostly what people like that I put on Twitter (and yes, I’ve checked) are announcements that link to other things, accomplishments in other areas, like a blog post, or a feature in Twisted, but using Twitter itself is inimical to completing those things.

I’m loath to abandon the positive aspects of Twitter. Some people also use Twitter as a replacement for RSS, and I don’t want to break the way they choose to pay attention to the stuff that I do. And a few of my friends communicate exclusively through direct messages.

The really “good” thing about Twitter is discovery. It enables you to discover people, content, and, eugh, “brands” that appeal to you. I have discovered things that I enjoy many times. The fundamental problem I am facing, which is a little bit hard to admit to oneself, is that I have discovered enough. I have enough games to play, enough books and articles to read, enough podcasts to listen to, enough movies to watch, enough code to write, enough open source libraries to investigate, that I will be busy for years based on what I already know.

For me, using Twitter’s timeline at this point to “discover” more things is like being at a delicious buffet, being so full I’m nauseous, and stuffing my pockets with shrimp “just in case” I’m hungry “when I get home” - and then, of course, not going home.

Even disregarding my desire to produce useful content, if I just want to enjoy consuming content more deeply, I have to take the time to engage with it properly.

So here’s what I’m doing:

  1. I am turning on the “anyone can direct message me” feature. We’ll see how that goes; I may have to turn it off again later. As always, I’d prefer you send email (or text me, if it’s time-critical).
  2. I am unfollowing literally everyone, and will not follow people in the future. Checking my timeline was the main information junk-food I want to avoid.
  3. Since my timeline, rather than mentions and replies, was my main source of distraction, I’ll continue paying attention to mentions and replies (at least for now; I’ll have to see if that becomes a problem in the absence of a timeline).
  4. In order to avoid producing such information junk-food myself, I’m going to try to directly tweet less, and put more things into brief blog posts so I have enough room to express them. I won’t say “not at all”, but most of the things that I put on Twitter would really be better as longer, more thoughtful articles.

Please note that there’s nothing prescriptive here. I’m outlining what I’m doing in the hopes that others might recognize similar problems with themselves - if everyone used Twitter this way, there would hardly be a point to the site.

Also, if I’ve unfollowed you, that doesn’t mean I’m not interested in what you have to say. I already have a way of keeping in touch with people’s more fully-formed ideas: I use Blogtrottr to deliver relevant blog articles to my email. If I previously followed you and you think I might not be reading your blog already (in most cases I believe I already am), please feel free to drop me a line with an RSS link.

Syndicated 2015-06-09 00:41:00 from Deciphering Glyph

Separate your Fakes and your Inspectors

When you are writing unit tests, you will commonly need to write duplicate implementations of your dependencies to test against systems which do external communication or otherwise manipulate state that you can’t inspect. In other words, test fakes. However, a “test fake” is just one half of the component that you’re building: you’re also generally building a test inspector.

As an example, let’s consider the case of this record-writing interface that we may need to interact with.

1
2
3
4
5
6
class RecordWriter(object):
    def write_record(self, record):
        "..."

    def close(self):
        "..."

This is a pretty simple interface; it can write out a record, and it can be closed.

Faking it out is similarly easy:

1
2
3
4
5
class FakeRecordWriter(object):
    def write_record(self, record):
        pass
    def close(self):
        pass

But this fake record writer isn’t very useful. It’s a simple stub; if our application writes any interesting records out, we won’t know about it. If it closes the record writer, we won’t know.

The conventional way to correct this problem, of course, is to start tracking some state, so we can assert about it:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
class FakeRecordWriter(object):
    def __init__(self):
        self.records = []
        self.closed = False

    def write_record(self, record):
        if self.closed:
            raise IOError("cannot write; writer is closed")
        self.records.append(record)

    def close(self):
        if self.closed:
            raise IOError("cannot close; writer is closed")
        self.closed = True

This is a very common pattern in test code. However, it’s an antipattern.

We have exposed 2 additional, apparently public attributes to application code: .records and .closed. Our original RecordWriter interface didn’t have either of those. Since these attributes are public, someone working on the application code could easily, inadvertently access them. Although it’s unlikely that an application author would think that they could read records from a record writer by accessing .records, it’s plausible that they might add a check of .closed before calling .close(), to make sure they won’t get an exception. Such a mistake might happen because their IDE auto-suggested the completion, for example.

The resolution for this antipattern is to have a separate “fake” object, exposing only the public attributes that are also on the object being faked, and an “inspector” object, which exposes only the functionality useful to the test.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
class WriterState(object):
    def __init__(self):
        self.records = []
        self.closed = False

    def raise_if_closed(self):
        if self.closed:
            raise ValueError("already closed")


class _FakeRecordWriter(object):
    def __init__(self, writer_state):
        self._state = writer_state

    def write_record(self, record):
        self._state.raise_if_closed()
        self._state.records.append(record)

    def close(self):
        self._state.raise_if_closed()
        self._state.closed = True


def create_fake_writer():
    state = WriterState()
    return state, _FakeRecordWriter(state)

In this refactored example, we now have a top-level entry point of create_fake_writer, which always creates a pair of WriterState and thing-which-is-like-a-RecordWriter. The type of _FakeRecordWriter can now be private, because it’s no longer interesting on its own; it exposes nothing beyond the methods it’s trying to fake.

Whenever you’re writing test fakes, consider writing them like this, to ensure that you can hand application code the application-facing half of your fake, and test code the test-facing half of the fake, and not get them mixed up.

Syndicated 2015-05-09 06:52:00 from Deciphering Glyph

Not Funny

What?

Today’s “joke” from the PSF about PyCon Havana was not funny, and, speaking as a PSF Fellow, I do not endorse it.

What’s Not Funny?

Honestly I’m not sure where I could find a punch-line in this. I just don’t see much there.

But if I look for something that’s supposed to be “funny”, here’s what I see:

  1. Cuba is a backward country without sufficient technology to host a technical conference, and it is absurd and therefore “funny” that we could hold PyCon there.
  2. We are talking about PyCon US; despite the recent thaw in relations, decades of hostility that have torn families apart make it “funny” that US citizens would go to Cuba for a conference.

These things aren’t funny.

Some Non-Reasons I’m Writing This

A common objection when someone speaks up about a subject like this is that it’s “just a joke”. That anyone speaking up and saying that offensive things aren’t funny somehow dislikes the very concept of humor. I don’t know why people think that, but I guess I need to make it clear: I am not an enemy of joy. That is not why I’m saying something.

I’m also not Cuban, I have no Cuban relatives, and until this incident I didn’t even know I had friends of Cuban extraction, so I am not personally insulted by this. That means another common objection will crop up: some will ask if I’m just looking for an excuse to get offended, to write about taking offense and get attention for it.

So let me assure you, that personally, this is not the kind of attention that I want. I really didn’t want to write this post. It’s awkward. I really don’t want to be having these types of conversations. I want to get attention for the software I write, not for my opinions about tacky blog posts.

Why, Then?

I might not know many Cuban python programmers personally, but I’d love to meet some. I’d love to meet anyone who cares about programming. Meeting diverse people from all over the world and working with them on code has been one of the great joys of my life. I love the fact that the Python community facilitates that and tries hard to reach out to people and to make them feel welcome.

I am writing this because I know that, somewhere out there, there’s a Cuban programmer, or a kid who will grow up to be one, who might see that blog post, and think that the Python community, or the software industry, thinks that they’re a throw-away punch line. I want them to know that I don’t think they’re a punch line. I want them to know that the python community doesn’t think they’re a punch line. I want them to know that they are not a punch line, and I want them to pursue their interest in programming exactly as far as it takes them and not push them away.

These people are real, they are listening, and if you tell me to just “lighten up” you are saying that your enjoyment of a joke is more important than their membership in our community.

It’s Not Just Me

The PSF is paying attention. The chairman of the PSF has acknowledged the problematic nature of the “joke”. Several of my friends in the Python community spoke up before I did (here, here, here, here, here, here, here), and I am very grateful for their taking the community to task and keeping us true to ideals of inclusiveness and empathy.

That doesn’t excuse the public statement, made using official channels, which was in very poor taste. I am also very disappointed in certain people within the PSF1 who seem intent on doubling down on this mistake rather than trying to do something to correct it.


  1. names withheld to avoid a pile-on, but you know who you are and you should be ashamed. 

Syndicated 2015-04-02 05:47:00 from Deciphering Glyph

Headcanon

My Castle headcanon1 has always been that, when they finally catch up with Mal (oh, and they definitely do catch up with him; the idea that no faction within the Alliance would seek revenge for what he’s done is laughable) they decide that they can, in fact, “make people better”, and he is no exception. After the service he has done in exposing the corruption and cover-ups behind Miranda, they can’t just dispose of him, so they want to rehabilitate him and make him a productive, contributing member of alliance society.

They can’t simply re-format his brain directly, of course. It wouldn’t be compatible with his personality, and his underlying connectome would simply reject the overlaid neural matrix; it would degrade over time, and he would have to return for treatments far too regularly for it to be practical.

The most fitting neural re-programming they can give him, of course, would be to have him gradually acclimate to becoming a lawman. So “Richard Castle” begins as an anti-authoritarian man-child and acquiesces, bit by bit, to the necessity of becoming an agent of the enforcement of order.

My favorite thing about the current season is that, while it is already obvious that my interpretation is correct, this season has given Mal a glimmer of hope. Clearly the reprogramming isn’t working, and aspects of his real life are coming through.

They really can’t take the sky from him.

Syndicated 2015-03-22 08:16:00 from Deciphering Glyph

Deploying Python Applications with Docker - A Suggestion

Deploying python applications is much trickier than it should be.

Docker can simplify this, but even with Docker, there are a lot of nuances around how you package your python application, how you build it, how you pull in your python and non-python dependencies, and how you structure your images.

I would like to share with you a strategy that I have developed for deploying Python apps that deals with a number of these issues. I don’t want to claim that this is the only way to deploy Python apps, or even a particularly right way; in the rapidly evolving containerization ecosystem, new techniques pop up every day, and everyone’s application is different. However, I humbly submit that this process is a good default.

Rather than equivocate further about its abstract goodness, here are some properties of the following container construction idiom:

  1. It reduces build times from a naive “sudo setup.py install” by using Python wheels to cache repeatably built binary artifacts.
  2. It reduces container size by separating build containers from run containers.
  3. It is independent of other tooling, and should work fine with whatever configuration management or container orchestration system you want to use.
  4. It uses existing Python tooling of pip and virtualenv, and therefore doesn’t depend heavily on Docker. A lot of the same concepts apply if you have to build or deploy the same Python code into a non-containerized environment. You can also incrementally migrate towards containerization: if your deploy environment is not containerized, you can still build and test your wheels within a container and get the advantages of containerization there, as long as your base image matches the non-containerized environment you’re deploying to. This means you can quickly upgrade your build and test environments without having to upgrade the host environment on finicky continuous integration hosts, such as Jenkins or Buildbot.

To test these instructions, I used Docker 1.5.0 (via boot2docker, but hopefully that is an irrelevant detail). I also used an Ubuntu 14.04 base image (as you can see in the docker files) but hopefully the concepts should translate to other base images as well.

In order to show how to deploy a sample application, we’ll need a sample application to deploy; to keep it simple, here’s some “hello world” sample code using Klein:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
# deployme/__init__.py
from klein import run, route

@route('/')
def home(request):
    request.setHeader("content-type", "text/plain")
    return 'Hello, world!'

def main():
    run("", 8081)

And an accompanying setup.py:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
from setuptools import setup, find_packages

setup (
    name             = "DeployMe",
    version          = "0.1",
    description      = "Example application to be deployed.",
    packages         = find_packages(),
    install_requires = ["twisted>=15.0.0",
                        "klein>=15.0.0",
                        "treq>=15.0.0",
                        "service_identity>=14.0.0"],
    entry_points     = {'console_scripts':
                        ['run-the-app = deployme:main']}
)

Generating certificates is a bit tedious for a simple example like this one, but in a real-life application we are likely to face the deployment issue of native dependencies, so to demonstrate how to deal with that issue, that this setup.py depends on the service_identity module, which pulls in cryptography (which depends on OpenSSL) and its dependency cffi (which depends on libffi).

To get started telling Docker what to do, we’ll need a base image that we can use for both build and run images, to ensure that certain things match up; particularly the native libraries that are used to build against. This also speeds up subsquent builds, by giving a nice common point for caching.

In this base image, we’ll set up:

  1. a Python runtime (PyPy)
  2. the C libraries we need (the libffi6 and openssl ubuntu packages)
  3. a virtual environment in which to do our building and packaging
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
# base.docker
FROM ubuntu:trusty

RUN echo "deb http://ppa.launchpad.net/pypy/ppa/ubuntu trusty main" > \
    /etc/apt/sources.list.d/pypy-ppa.list

RUN apt-key adv --keyserver keyserver.ubuntu.com \
                --recv-keys 2862D0785AFACD8C65B23DB0251104D968854915
RUN apt-get update

RUN apt-get install -qyy \
    -o APT::Install-Recommends=false -o APT::Install-Suggests=false \
    python-virtualenv pypy libffi6 openssl

RUN virtualenv -p /usr/bin/pypy /appenv
RUN . /appenv/bin/activate; pip install pip==6.0.8

The apt options APT::Install-Recommends and APT::Install-Suggests are just there to prevent python-virtualenv from pulling in a whole C development toolchain with it; we’ll get to that stuff in the build container. In the run container, which is also based on this base container, we will just use virtualenv and pip for putting the already-built artifacts into the right place. Ubuntu expects that these are purely development tools, which is why it recommends installation of python development tools as well.

You might wonder “why bother with a virtualenv if I’m already in a container”? This is belt-and-suspenders isolation, but you can never have too much isolation.

It’s true that in many cases, perhaps even most, simply installing stuff into the system Python with Pip works fine; however, for more elaborate applications, you may end up wanting to invoke a tool provided by your base container that is implemented in Python, but which requires dependencies managed by the host. By putting things into a virtualenv regardless, we keep the things set up by the base image’s package system tidily separated from the things our application is building, which means that there should be no unforseen interactions, regardless of how complex the application’s usage of Python might be.

Next we need to build the base image, which is accomplished easily enough with a docker command like:

1
$ docker build -t deployme-base -f base.docker .;

Next, we need a container for building our application and its Python dependencies. The dockerfile for that is as follows:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
# build.docker
FROM deployme-base

RUN apt-get install -qy libffi-dev libssl-dev pypy-dev
RUN . /appenv/bin/activate; \
    pip install wheel

ENV WHEELHOUSE=/wheelhouse
ENV PIP_WHEEL_DIR=/wheelhouse
ENV PIP_FIND_LINKS=/wheelhouse

VOLUME /wheelhouse
VOLUME /application

ENTRYPOINT . /appenv/bin/activate; \
           cd /application; \
           pip wheel .

Breaking this down, we first have it pulling from the base image we just built. Then, we install the development libraries and headers for each of the C-level dependencies we have to work with, as well as PyPy’s development toolchain itself. Then, to get ready to build some wheels, we install the wheel package into the virtualenv we set up in the base image. Note that the wheel package is only necessary for building wheels; the functionality to install them is built in to pip.

Note that we then have two volumes: /wheelhouse, where the wheel output should go, and /application, where the application’s distribution (i.e. the directory containing setup.py) should go.

The entrypoint for this image is simply running “pip wheel” with the appropriate virtualenv activated. It runs against whatever is in the /application volume, so we could potentially build wheels for multiple different applications. In this example, I’m using pip wheel . which builds the current directory, but you may have a requirements.txt which pins all your dependencies, in which case you might want to use pip wheel -r requirements.txt instead.

At this point, we need to build the builder image, which can be accomplished with:

1
$ docker build -t deployme-builder -f build.docker .;

This builds a deployme-builder that we can use to build the wheels for the application. Since this is a prerequisite step for building the application container itself, you can go ahead and do that now. In order to do so, we must tell the builder to use the current directory as the application being built (the volume at /application) and to put the wheels into a wheelhouse directory (one called wheelhouse will do):

1
2
3
4
5
$ mkdir -p wheelhouse;
$ docker run --rm \
         -v "$(pwd)":/application \
         -v "$(pwd)"/wheelhouse:/wheelhouse \
         deployme-builder;

After running this, if you look in the wheelhouse directory, you should see a bunch of wheels built there, including one for the application being built:

1
2
3
4
5
6
$ ls wheelhouse
DeployMe-0.1-py2-none-any.whl
Twisted-15.0.0-pp27-none-linux_x86_64.whl
Werkzeug-0.10.1-py2-none-any.whl
cffi-0.9.0-py2-none-any.whl
# ...

At last, time to build the application container itself. The setup for that is very short, since most of the work has already been done for us in the production of the wheels:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
# run.docker
FROM deployme-base

ADD wheelhouse /wheelhouse
RUN . /appenv/bin/activate; \
    pip install --no-index -f wheelhouse DeployMe

EXPOSE 8081

ENTRYPOINT . /appenv/bin/activate; \
           run-the-app

During build, this dockerfile pulls from our shared base image, then adds the wheelhouse we just produced as a directory at /wheelhouse. The only shell command that needs to run in order to get the wheels installed is pip install TheApplicationYouJustBuilt, with two options: --no-index to tell pip “don’t bother downloading anything from PyPI, everything you need should be right here”, and, -f wheelhouse which tells it where “here” is.

The entrypoint for this one activates the virtualenv and invokes run-the-app, the setuptools entrypoint defined above in setup.py, which should be on the $PATH once that virtualenv is activated.

The application build is very simple, just

1
$ docker build -t deployme-run -f run.docker .;

to build the docker file.

Similarly, running the application is just like any other docker container:

1
$ docker run --rm -it -p 8081:8081 deployme-run

You can then hit port 8081 on your docker host to load the application.

The command-line for docker run here is just an example; for example, I’m passing --rm so that if you run this example just so that it won’t clutter up your container list. Your environment will have its own way to call docker run, how to get your VOLUMEs and EXPOSEd ports mapped, and discussing how to orchestrate your containers is out of scope for this post; you can pretty much run it however you like. Everything the image needs is built in at this point.

To review:

  1. have a common base container that contains all your non-Python (C libraries and utilities) dependencies. Avoid installing development tools here.
  2. use a virtualenv even though you’re in a container to avoid any surprises from the host Python environment
  3. have a “build” container that just makes the virtualenv and puts wheel and pip into it, and runs pip wheel
  4. run the build container with your application code in a volume as input and a wheelhouse volume as output
  5. create an application container by starting from the same base image and, once again not installing any dev tools, pip install all the wheels that you just built, turning off access to PyPI for that installation so it goes quickly and deterministically based on the wheels you’ve built.

While this sample application uses Twisted, it’s quite possible to apply this same process to just about any Python application you want to run inside Docker.

I’ve put a sample project up on Github which contain all the files referenced here, as well as “build” and “run” shell scripts that combine the necessary docker commandlines to go through the full process to build and run this sample app. While it defaults to the PyPy runtime (as most networked Python apps generally should these days, since performance is so much better than CPython), if you have an application with a hard CPython dependency, I’ve also made a branch and pull request on that project for CPython, and you can look at the relatively minor patch required to get it working for CPython as well.

Now that you have a container with an application in it that you might want to deploy, my previous write-up on a quick way to securely push stuff to a production service might be of interest.

(Once again, thanks to my employer, Rackspace, for sponsoring the time for me to write this post. Thanks also to Shawn Ashlee and Jesse Spears for helping me refine these ideas and listening to me rant about them. However, that expression of gratitude should not be taken as any kind of endorsement from any of these parties as to my technical suggestions or opinions here, as they are entirely my own.)

Syndicated 2015-03-06 22:58:00 from Deciphering Glyph

According To...?

I believe that web browsers must start including the ultimate issuer in an always-visible user interface element.

You are viewing this website at glyph.twistedmatrix.com. Hopefully securely.

We trust that the math in the cryptographic operations protects our data from prying eyes. However, trusting that the math says the content is authentic and secure is useless unless you know who your computer is talking to. The HTTPS/TLS system identifies your interlocutor by their domain name.

In other words, you trust that these words come from me because glyph.twistedmatrix.com is reasonably associated with me. If the lock on your web browser’s title bar was next to the name stuff-glyph-says.stealing-your-credit-card.example.com, presumably you might be more skeptical that the content was legitimate.

But... the cryptographic primitives require a trust root - somebody that you “already trust” - meaning someone that your browser already knows about at the time it makes the request - to tell you that this site is indeed glyph.twistedmatrix.com. So you read these words as if they’re the world according to Glyph, but according to whom is it according to me?

If you click on some obscure buttons (in Safari and Firefox you click on the little lock; in Chrome you click on the lock, then “Connection”) you should see that my identity as glyph.twistedmatrix.com has been verified by “StartCom Class 1 Primary Intermediate Server CA” who was in turn verified by “StartCom Certification Authority”.

But if you do this, it only tells you about this one time. You could click on a link, and the issuer might change. It might be different for just one script on the page, and there’s basically no way to find out. There are more than 50 different organizations which could certify that could tell your browser to trust that this content is from me, several of whom have already been compromised. If you’re concerned about government surveillance, this list includes the governments of Hong Kong, Japan, France, the Netherlands, Turkey, as well as many multinational corporations vulnerable to secret warrants from the USA.

Sometimes it’s perfectly valid to trust these issuers. If I’m visiting a website describing some social services provided to French citizens, it would of course be reasonable for that to be trusted according to the government of France. But if you’re reading an article on my website about secure communications technology, probably it shouldn’t be glyph.twistedmatrix.com brought to you by the China Internet Network Information Center.

Information security is all about the user having some expectation and then a suite of technology ensuring that that expectation is correctly met. If the user’s expectation of the system’s behavior is incorrect, then all the technological marvels in the world making sure that behavior is faithfully executed will not help their actual security at all. Without knowing the issuer though, it’s not clear to me what the user’s expectation is supposed to be about the lock icon.

The security authority system suffers from being a market for silver bullets. Secure websites are effectively resellers of the security offered to them by their certificate issuers; however, the customers are practically unable to even see the trade mark - the issuer name - of the certificate authority ultimately responsible for the integrity and confidentiality of their communications, so they have no information at all. The website itself also has next to no information because the certificate authority themselves are under no regulatory obligation to disclose or verify their security practices.

Without seeing the issuer, there’s no way for “issuer reputation” to be a selling point, which means there’s no market motivation for issuers to do a really good job securing their infrastructure. There’s no way for average users to notice if they are the victims of a targetted surveillance attack.

So please, browser vendors, consider making this information available to the general public so we can all start making informed decisions about who to trust.

Syndicated 2015-02-13 20:52:00 from Deciphering Glyph

Security as Stencil

Image Credit: Horia Varlan

On the Internet, it’s important to secure all of your communications.

There are a number of applications which purport to give you “secure chat”, “secure email”, or “secure phone calls”.

The problem with these applications is that they advertise their presence. Since “insecure chat”, “insecure email” and “insecure phone calls” all have a particular, detectable signature, an interested observer may easily detect your supposedly “secure” communication. Not only that, but the places that you go to obtain them are suspicious in their own right. In order to visit Whisper Systems, you have to be looking for “secure” communications.

This allows the adversary to use “security” technologies such as encryption as a sort of stencil, to outline and highlight the communication that they really want to be capturing. In the case of the NSA, this dumps anyone who would like to have a serious private conversation with a friend into the same bucket, from the perspective of the authorities, as a conspiracy of psychopaths trying to commit mass murder.

The Snowden documents already demonstrate that the NSA does exactly this; if you send a normal email, they will probably lose interest and ignore it after a little while, whereas if you send a “secure” email, they will store it forever and keep trying to crack it to see what you’re hiding.

If you’re running a supposedly innocuous online service or writing a supposedly harmless application, the hassle associated with setting up TLS certificates and encryption keys may seem like a pointless distraction. It isn’t.

For one thing, if you have anywhere that user-created content enters your service, you don’t know what they are going to be using it to communicate. Maybe you’re just writing an online game but users will use your game for something as personal as courtship. Can we agree that the state security services shouldn’t be involved in that?. Even if you were specifically writing an app for dating, you might not anticipate that the police will use it to show up and arrest your users so that they will be savagely beaten in jail.

The technology problems that “secure” services are working on are all important. But we can’t simply develop a good “secure” technology, consider it a niche product, and leave it at that. Those of us who are software development professionals need to build security into every product, because users expect it. Users expect it because we are, in a million implicit ways, telling them that they have it. If we put a “share with your friend!” button into a user interface, that’s a claim: we’re claiming that the data the user indicates is being shared only with their friend. Would we want to put in a button that says “share with your friend, and with us, and with the state security apparatus, and with any criminal who can break in and steal our database!”? Obviously not. So let’s stop making the “share with your friend!” button actually do that.

Those of us who understand the importance of security and are in the business of creating secure software must, therefore, take on the Sisyphean task of not only creating good security, but of competing with the insecure software on its own turf, so that people actually use it. “Slightly worse to use than your regular email program, but secure” is not good enough. (Not to mention the fact that existing security solutions are more than “slightly” worse to use). Secure stuff has to be as good as or better than its insecure competitors.

I know that this is a monumental undertaking. I have personally tried and failed to do something like this more than once. As the Rabbi Tarfon put it, though:

It is not incumbent upon you to complete the work, but neither are you at liberty to desist from it.

Syndicated 2015-01-25 00:16:00 from Deciphering Glyph

8 older entries...

New Advogato Features

New HTML Parser: The long-awaited libxml2 based HTML parser code is live. It needs further work but already handles most markup better than the original parser.

Keep up with the latest Advogato features by reading the Advogato status blog.

If you're a C programmer with some spare time, take a look at the mod_virgule project page and help us with one of the tasks on the ToDo list!