Graphing Jenkins Statistics

Like many people, we use Jenkins at work as our continuous integration server and we require that all changes that are committed go through being built in CI before they can get deployed.  Yesterday, someone asked if we could add another jenkins slave to try to reduce the amount of time spent waiting on builds.  While the slaves are fully puppetized and so it’s not much work to bring an additional slave online, my own anecdotal experience made me think that we weren’t really held up often in a way that additional slaves would help.  I had a vague memory of some graphs within jenkins so eventually found them but didn’t really find them that enlightening.  The scale is funky, it’s a weird exponential moving average and I just didn’t find it that easy to get any insight from them.

So last night, I sat down and wrote a quick little script to run via cron and pull some statistics and throw them into graphite.  Already with less than a day of data, I’m better able to tell that we end up with a few periods of about ten minutes where having more executors could help that are correlated with when someone does a commit to one of the projects at the base of our dependency tree.  So that gives us a lot better idea of whether or not the cost of an additional machine is worth the few minutes that we’d be able to save in those cases.

Since it didn’t look like anyone else had done anything along these lines yet, I put the code up on github.  There are a lot more stats that could be pulled out via the jenkins api, this is really just a starting point for what I needed today.

Syndicated 2012-01-13 02:12:17 from Jeremy's Thoughts

Monitoring Driven DevOps

Wrote up a nice post that maps pretty well to the Ignite talk I gave at Velocity about using using monitoring to help drive your infrastructure development.

Go check it out over on the HubSpot dev blog

Syndicated 2011-06-28 00:12:08 from Jeremy's Thoughts

Velocity 2011

I spent last week out in California for the O’Reilly Velocity Conference.  It was in Santa Clara, which I hadn’t been to and frankly, I would be perfectly happy to not return.  Parts of California are nice, Santa Clara is an office building wasteland.  No good food options, nothing really going on, etc.  But I was there for a conference and not for other stuff, so it sufficed.

The conference was actually very good.  It has been a few years since I’ve been to a conference between grad school, my daughter being born, and being at a startup where conferences weren’t the priority.  But it was good to get back to it.  Had a lot of good hallway conversations with people about things that are relevant to us and saw a lot of good presentations.  And Velocity is especially relevant to me at this point as it was all about various web performance and operations stuff.  Where, unsurprisingly, there’s a lot of cool stuff going on.

I mostly kept to the more operations-y tracks just because they map better to what I’m currently working on.  I’ve come away with a bunch of things to look into and posted a whole bunch of choice quotes over on Twitter, but a few takeaways boiled down for here would include

• If you’re using a public cloud provider, plan for things to fail.  Build your systems expecting it and you’ll have less pain.
• HubSpot is doing an awesome job with post-mortems.  DanM actually posted a great blog post over on our dev blog about things we’ve learned from doing a lot of them.
• Everyone complains and focuses on javascript performance but that’s misguided.  The bottleneck is the DOM.  Interestingly, none of the browser guys talked about that apparently
• DevOps has mostly been about putting developers into ops (hi!) but also needs to be about putting ops into dev
• Web performance has been very successful in tying itself to business metrics.  Weirdly, operations has overall been less successful at that
• There’s a lot of work going on to help with debugging and working on webapps for mobile platforms.  Very cool.

None of those are particularly earth shattering revelations, but still good to see/hear.

Also, on Tuesday night I did a talk for the Ignite track.  So 5 minutes, 20 slides, auto-advancing.  My topic was “Just Too Late” and was largely around some things I’ve discovered transitioning into a role where I’m doing more ops stuff and the fact that I feel like I get to things too late.  But then turning it around and showing that’s not really so.  Stay tuned for a longer blog post on the topic.  But the talk went really well.  It was fun, a lot of positive feedback and was good for me to get back to it.  Looking forward to submitting some (full-length) proposals for talks for some conferences later this year.

I also had a few thoughts on the way conferences have changed since I last went to one

• Twitter really is a pretty big game changer.  Lots of conversation on twitter during the conference about which sessions were good, useful tidbits from sessions, etc.  I actually felt that the experience was pretty strongly enhanced by it
• Conference wireless still sucks.  But you can get decent data now for devices and avoid the use of the conference wireless entirely.  This made it easier to stay on twitter during the conference
• An iPad (or other tablet) is a pretty perfect device for looking at stuff during a conference.  It sits on your lap so you can just check it sporadically, the battery lasts all day, you can get data from a cellular provider, and it’s reasonably fast.

Anyway, good time was had.  Thanks to all the people that I met and chatted up.  And hopefully it won’t be as long before I make it to another conference

Syndicated 2011-06-20 01:33:23 from Jeremy's Thoughts

My new role

I’m still at HubSpot but my role within the company has changed a bit over the past few months.  Related to the article that Yoav wrote which was posted on onStartups today about how we’re trying to better empower our engineers and teams to really own things, I’ve shifted my focus some.

Instead of working on the product which is front and center to all of our customers or even working on the free tools at grader.com that millions of people use, I’m now instead focused quite a bit on various infrastructure related things for us. Obviously, I’ve done some of that all along, but at this point, it’s my primary job.

It’s a lot of fun. We are heavy users of EC2 and some of the other Amazon services. We also are using Rackspace Cloud some. And I wouldn’t be surprised if we add another provider in the future. So there is a challenge in making all of these environments look the same for the rest of our dev team as well as our on call folks.  We’re also working to make it so that we can easily continue to scale out as our compute needs increase.  All the sorts of things that I’ve spent some time thinking about over the years, but there’s no theoretical here — we’re really deploying, managing and everything else a pretty large distributed system. We are using a fair bit of open source stuff in addition to building some stuff ourselves.  The first thing was obviously ami-creator but there’s more to come almost certainly. In addition, we’ll probably be doing some work and submitting some patches to improve some of the tools and things that we use as it makes sense to do so.

And as we we are growing like crazy, I’m looking to hire some people to join my team to help us get even more things done. If I were writing a job description it would probably include bits and pieces like Linux administration, python, puppet, probably devops (as it’s something that’s in mind), cloud automation (… even though I still hate the word cloud), release and build tooling, monitoring, and more. Sound interesting? Drop me a line and let’s talk.

Syndicated 2011-02-03 15:30:59 from Jeremy's Thoughts

2010, The Quick Look Back

I could give some excuse about being busy, but everybody does that.  Let’s do the rundown of big events of 2010

1. Finished and submitted my thesis to complete the work necessary to finish up the SDM program and graduate in January.
2. My daughter was born in February.  There are plenty of pictures on Facebook if you know me and are on there (mostly taken by my wife)
3. I’ve continued working at HubSpot where there’s no shortage of work to be done and it seems like always something big going on or coming up
4. Picked up cyclocross racing in the fall.  Read all about it. Since apparently I managed to blog about that.

If you wanted more details, well, there was a lot more on twitter.  But hopefully going to get back into the habit of some blogging again for the new year.  Then again, I’ve said that before.

Syndicated 2011-01-13 03:03:51 from Jeremy's Thoughts