Advogato blog for mikal

We all know that the LCA2014 CFP is open, right?

Mon, 17 Jun 2013 10:14:18 GMT

I just want to make sure that everyone knows that the LCA2014 call for proposals is open. There are two calls this time around -- a call for proposals and a call for miniconfs. The call for proposals closes on 6 July, so you don't have heaps of time left to submit something.

So, if you're interested in speaking at linux.conf.au 2014, in Perth between 6 and 10 January 2014 you should hit up those CFPs now!

Tags for this post: conference lca2014 cfp
Related posts: LCA 2006: CFP closes today; Got Something to Say? The LCA 2013 CFP Opens Soon!; Call for papers opens soon

Comment

Merged in Havana: fixed ip listing for single hosts

Fri, 26 Apr 2013 08:08:30 GMT

Nova has supported listing the fixed ips for a single host for a while. Well, except for that time we broke it by removing the database call it used and not noticing. My change to fix that situation has just landed, so this should now work again. To list the fixed ips used on a host, do something like:

nova-manage fixed list hostname

I will propose a backport to grizzly for this now.

Tags for this post: openstack havana fixed_ip nova rackspace
Related posts: Upgrade problems with the new Fixed IP quota; Merged in Havana: configurable iptables drop actions in nova; Michael's surprisingly unreliable predictions for the Havana Nova release; Havana Nova PTL elections; Faster pip installs; Some quick operational notes for users of loop and nbd devices; Further adventures with base images in OpenStack; OpenStack at linux.conf.au 2013; Moving on; Image handlers (in essex); Openstack compute node cleanup

Comment

Michael's surprisingly unreliable predictions for the Havana Nova release

Sat, 20 Apr 2013 07:08:49 GMT

I should start out by saying that because OpenStack is an open source project, it is hard to know exactly what will land in Havana -- the developers are volunteers, and sometimes things get in the way of them doing the work they intended. However, these are the notes I wrote up on the high points of the summit for me -- I didn't see all the same sessions as other nova developers, so hopefully others will pitch in with their notes as well.

Scheduler

The scheduler seems to be a point of planned work for a lot of people in this release, with talk about having more scheduling code in the common library, and of adding new filter types. There is definite interest in being able to schedule by methods we don't currently support -- things like rack or PDU diversity, or trying to collocate a tenants machines together. HP is also interested in being able to sell dedicated machines to tenants -- in other words, they would guarantee that only one tenants instances appeared on a machine in return for a fee. At the moment this requires setting up a host aggregate for the tenant.

Feeding additional data into scheduling decisions

There is also interest in being able to feed more scheduling information to the nova-scheduler. For example, ceilometer intends to start collecting monitoring data from nova-compute nodes, and perhaps it might inform nova-scheduler that a machine is running hot or has a degraded RAID array. This might also be the source of PDU or CRAC failure information which might affect scheduling decisions -- these later two examples are interesting because they are information where it doesn't make sense to get it from the compute node, the correct location for this information is a data center wide system, not an individual machine. There is concern about nova-scheduler depending on other systems, so these updates from ceilometer will probably be advisory updates, with nova-scheduler degrading gracefully if they are not present or are stale.

Mothballing

This was almost instantly renamed to "shelving", but "swallow / spew" was also considered. This is a request that Rackspace sees from customers -- basically the ability to stop a virtual machine, but keep the UUID and IP addresses associated with the machine as well as the block device mapping. The proposal is to implement this as a snapshot of the machine, and a new machine state. The local disk files for the instance might get deleted if the resources are needed. This would feel like a reboot of an instance to a user.

This is of interest for workloads like "Black Friday" web servers. You could bring a whole bunch up, configure security groups, load balancers, and the applications on the instances and then shelve the instance. When you need the instance to handle load, you'd then unshelve the instance and once it was booted it would just magically start serving. Expect to see shelves instances be cheaper than a running instance, but not free. This is mostly because IP addresses are scarce. Restarting a shelved instance might take a while if the snapshot has to be fetched to a compute node. If you need a more "instant on" bursting capacity, then just leave instances idling and pay full price.

Deferred instance file delete

This is a nice to have requirement for shelving instances, but it is useful for other things as well. This is the ability to delay the deletion of instance files when an instance is torn down. This might end up being expressed as "keep these files for at least X days, unless you are tight on disk resources". I can see other reasons this would be useful -- for example helping support people rescue data from instances users tore down and now want back. It also defers the disk IO from deleting the files until its absolutely necessary. We could also perhaps detect times when the disks are "relatively idle" and use those to clean up file systems.

DNS in nova-network

Expect to see the current DNS driver removed, as no one uses it as best as we can tell. This will be replaced with a simpler drive in nova-compute and the recommendation that deployers use quantum DNS if possible.

Quantum

There is continued work of making quantum the default networking engine for nova. There are still some missing features, but the list of absolutely blocking features is getting smaller. A lot of discussion centered around how to live upgrade clouds from nova-network to quantum. This is not an easy problem, but smart people are looking at it. The solution might involve moving compute nodes over to quantum, and then live migrating instances over to those compute nodes. However, we currently only support one network driver at a time in nova, so we will need to change some code here.

Long running periodic tasks

There will be a refactor of the periodic task code in nova this release to move periodic tasks which incur a lot of blocking IO into separate processes. These processes will be launched by nova-compute, and not be cron jobs or something like that. Most of the discussion was around how to do this safely (eventlet makes it exciting), which is nice in that it indicates some level of consensus that this is needed. The plan for now is to do this in nova-compute, but leave other nova components for later releases.

Libvirt changes

Libvirt is the compute driver I work on, so it's the only one I want to comment on here. The other drivers are doing interesting things as well, I just don't want to get details wrong by not understanding their efforts.

First off, there should be some work done on better console logging in Havana. At the moment we use an unbounded file on disk. This will hopefully become a Unix domain socket managing a ring buffer of some form. The Unix domain socket leaves the option open of later making this serial console interactive, but that's not an immediate goal.

There was a lot of talk about LXC support, and how we need to support file system attachments as well as block devices. There is also some cleanup that can be done for the LXC support in the libvirt to make the code cleaner, but it is not clear who will work on this.

imagebackend.py will probably get refactored, but in ways that don’t make a big difference to users but make it easier to code against (and therefore more reliable). I'm including it here just because I'm excited about that refactor making this code easier to understand.

There was a lot of talk about live migration and the requirement for ssh between compute nodes. Operators don't love that compute nodes can talk to each other, but expect Havana to include some sort of on demand ssh key management, and a later release to proxy that traffic through something like nova-conductor.

Incremental backups are of interest to deployers as well, but there is concern that glance needs more support for chains of images before we can do that.

Conclusion The summit was fantastic once again, and the Foundation did an awesome job of hosting it. It was however a pretty tiring experience, and I'm sure I got some stuff here wrong, or missed things that others would consider important. It would be cool for other developers to write up summaries of what they saw at the summit as well.

Tags for this post: openstack havana rackspace summit nova summary prediction
Related posts: Merged in Havana: configurable iptables drop actions in nova; Havana Nova PTL elections; Upgrade problems with the new Fixed IP quota; Faster pip installs; Some quick operational notes for users of loop and nbd devices; Further adventures with base images in OpenStack; OpenStack at linux.conf.au 2013; Moving on; Image handlers (in essex); Openstack compute node cleanup

Comment

Getting started with OpenStack development

Tue, 16 Apr 2013 22:08:40 GMT

I just gave my presentation at the Havana Conference about how to get started with OpenStack development. A few people asked for my slide deck, so I am posting it here. The talk was taped, and I am sure some more formal release will happen in the future, but I wanted to get this out there for the people who had asked for it.

Tags for this post: openstack development guide howto conference python
Related posts: Hackathons again; Implementing SCP with paramiko; Breaking: Zap2IT turning off guide data for the US?; The new way of converting JPEGs to MPEGs; Packet capture in python; Top resale value cars; Let it be known I am open to bribes; Who writes Linux?; mbot: new hotness in Google Talk bots; Implementing parental filters with MythTV; Calculating a SSH host key with paramiko; Twisted conch; Example 2.1 from Dive Into Python; On syncing with Google Contacts; I'm liking python too, thanks for asking; SSL, X509, ASN.1 and certificate validity dates; Python effective TLD library bug fix; Multiple file support with scp; paramiko exec_command timeout; Killing a blocking thread in python?; Linux kernel cset patches

Comment

Faster pip installs

Mon, 8 Apr 2013 05:08:13 GMT

Last week with the help of the lovely openstack-infra people, I discovered that you can have a local cache of pip downloads. This speeds up rebuilding test environments when you need to jump between branches with different dependencies. Its as simple as chucking something like:

export PIP_DOWNLOAD_CACHE=~/cache/pip

Tags for this post: openstack tips pip cache

Comment

Merged in Havana: configurable iptables drop actions in nova

Sun, 31 Mar 2013 05:08:30 GMT

LaunchPad bug 1013893 asked nicely if the drop action for iptables rules created by nova-network could be configured. The idea here is that you might want to do something other than a plain old drop -- for example logging before dropping. This has now been implemented in Havana.

To configure the drop action, set the iptables_drop_action to the name of an already existing iptables target. Creating this target is not managed by nova, and you'll need to do it on every compute node. When iptables creates or deletes rules on compute nodes it will now use this new target. There's a bit of an upgrade problem here in that this will stop nova from deleting rules which use the old hard coded drop target. However, if an instance is torn down then all of its tables are torn down as well and rules will be deleted correctly, so this is only a problem if a security group is changed while the instance is running.

It occurs to me that we can do better here, so I've sent off this review to handle the case where a rule is being removed and used the default drop action.

For safety, I would recommend only using this flag on new compute nodes that have no instances running in order to make this simple.

Tags for this post: openstack havana nova iptables
Related posts: Further adventures with base images in OpenStack; Havana Nova PTL elections; Upgrade problems with the new Fixed IP quota; Openstack compute node cleanup

Comment

Upgrade problems with the new Fixed IP quota

Sun, 31 Mar 2013 00:08:28 GMT

In the last few weeks a new quota has been added to Nova covering Fixed IPs. This was done in response to LaunchPad bug 1125468, which was disclosed as CVE 2013-1838.

To be honest I think there are some things the vulnerability management team learned the hard way with this disclosure. For example, we didn't realize that we needed to update python-novaclient to allow users to set the quota, or that adding a quota would require changes in Horizon. Both of these errors have been corrected.

More importanly, the default value of the new quota was set to 10. I made this decision based on the default value of the instances quota coupled with a desire to protect deployments from denial of service. However, this decision combined with a failure to explicitly call out the new quota in the release notes for the Folsom stable release have resulted in some deployers experiencing upgrade problems. This was drawn to our attention by LaunchPad bug 1161190.

We have therefore moved to set the default quota for fixed IPs to unlimited. If you want to protect yourself from a potential DoS, then you should seriously consider changing this default value in your deployment. This can be done with the quota_fixed_ips flag. The code reviews implementing this change are either merged, or under review depending on the release. At the time of writing this Havana and Grizzly have a fix merged, with Folsom and Essex still under review.

I think this experience also reinforces the importance of testing all upgrades in a lab environment before doing them in production.

Sorry for any inconvenience caused.

Tags for this post: openstack nova quota fixed_ip vmt cve denial_of_service
Related posts: Further adventures with base images in OpenStack; Havana Nova PTL elections; Openstack compute node cleanup

Comment

Havana Nova PTL elections

Wed, 13 Mar 2013 16:09:30 GMT

This is just a quick reminder that there are only a couple more days to vote in the Nova PTL elections for the Havana cycle. If you're eligible to vote, you should have a voting URL in your email.

The candidates:

Russell Bryant -- announcement
Michael Still -- announcement

The incumbent PTL, Vishvananda Ishaya, has chosen not to run.

Rackspace is hiring OpenStack developers, let me know if you want to know more.

Tags for this post: openstack nova ptl election havana
Related posts: Further adventures with base images in OpenStack; Openstack compute node cleanup

Comment

Roanoke, Virginia

Tue, 12 Mar 2013 21:07:33 GMT

This is by far the smallest US town I've been to, so I found it quite exciting. I think I need to come back in summer when there are some leaves as well.

See more thumbnails

Tags for this post: events pictures 20130310 photo

Comment

Redshirts

Thu, 14 Feb 2013 10:13:02 GMT

ISBN: 9780765334797
LibraryThing

I was super excited to get my hands on the latest John Scalzi book because I've really liked his previous stuff. Unfortunately while this book is fun I feel that the underlying concept is pretty weak... Its more of a toy boy than something which makes you think, which is a disappointment to me.

Don't get me wrong, the overall execution is good, but the book feels lazily plotted, much like a badly done Harry Harrison does. So, if that sort of thing annoys you, give this one a miss.

Tags for this post: book john_scalzi exploration comedy space
Related posts: Dirk Gentley's Holistic Detective Agency; So Long, and Thanks For All the Fish ; The Restaurant at the End of the Universe ; Military satellites fly north-south, commercial satellites fly east-west?; The Man in the Rubber Mask; Bill The Galactic Hero ; Colony; Bill the Galactic Hero on the Planet of Bottled Brains; Ohhh, space junk; Russian scavengers life off abandoned space junk; The Light Fantastic; Starbound; The Hitchhikers Guide to the Galaxy ; Better Than Life; Bill the Galactic Hero Series; Marsbound; And Another Thing; Bingo!; Have I mentioned that I'm a big Smithsonian fan?; Life, the Universe and Everything ; Red Dwarf

Comment