Older blog entries for skvidal (starting at number 441)

ansible and cloud instances

A few days ago I posted about using ansible to provision persistent ip instances in a public or private cloud.

Today I finished up the next piece I needed to make this work for copr builders w/transient ips/instances.

I needed a way to create a new instance, provision it as if it was one of the normal inventory of our systems, and return back the ip of that was given to it via eucalyptus (or ec2) automatically. And when I was done with it – I didn’t want any record of it preserved in our inventory system. I wanted it gone.

The problem was ansible wants to act only on the hosts it knows about in its inventory. This is quite understandable. But since I’m not specifying an ip to this host and I don’t know it in advance I wanted a way, in a single playbook to do this.

So I wrote add_host an action_plugin for ansible. These let you interact with the internal state of the inventory/playbook/etc while executing a playbook.

All it does is to add the host to the in-memory inventory the ansible playbook object has. And it adds that host to a group you specify. This is so in the second play in the playbook you can say ‘oh operate on hosts in this special group name’ and that will be the only host in that group.

I’ve sent in a pull request for it but it’s not been accepted quite yet. However, if you want to try it out you can just toss it into your  action_plugins dir in ansible and call it.

Here’s an example playbook. It’s very similar to the one for creating a persistent instance. In the fedora public ansible repo we are, in fact, importing the same set of tasks from another file to set them up the same.

It just means if anyone wants an instance in our private cloud running f17 or el6 it is incredibly trivial to make one available for you.

 

 

Syndicated 2012-10-31 05:52:17 from journal/notes

handy ansible action for adding root keys to cloud instances

You’ve just spun up a new instance and you need to give additional people access to the system as root. You have a common IDMS that houses ssh pub keys for your users. You want to trivially specify a list of those users and have their keys show up in root’s .ssh/authorized_keys file.

Here’s what you do:

 

- name: add root keys for other allowed users
action: authorized_key user=root key=’$PIPE(/path/to/script/for/keys ${root_auth_users})’
only_if: is_set(‘${root_auth_users}’}

 

In our infrastructure FAS houses all the pubkeys. So Toshio wrote this script: http://infrastructure.fedoraproject.org/infra/ansible/scripts/auth-keys-from-fas

So if you define a hostvar in your ansible inventory for this host – then the above will automatically populate your root authorized_keys with the right pub keys.

Kinda awesome, I think.

Syndicated 2012-10-26 17:42:33 from journal/notes

creating and provisioning euca/ec2/cloud* instances using ansible in one step

Since we’ve been working more and more with our private cloud setups on eucalyptus and openstack I’ve been trying to come up with a semi-sane way of having persistent hosts in the cloudlets.

So that if we shut things down or an instance dies – it can be brought back up automatically without someone having to explicitly say ‘go start this up’. I’ve also been working on setting up ansible such that we can run it via cron on our central command host to maintain our systems.

So, of course, I’ve combined those two things into one task

We want certain cloud instances to:

1. always exist
2. have the same ip each time they come up.

These are for things like the build master for COPRs/PPAs or for the jenkins-master server or for any number of random services where they need a persistent ip throughout recreations. For the ips we’re using euca-associate-address after allocating a number to our account.

So I started on a mechanism to create those instances and give me results I could use in ansible for other actions. I came up with ec2_create which lets you do that from a local_action call in ansible.

Once you have that module in your ansible library path you just need to:

• add the host to your ansible inventory file
• take  this playbook example
• modify it for your host
• run it like: ansible-playbook thatplaybook.yml

you’ll need euca2ools installed along with the deps for ansible.

Here’s what the playbook does:

1. it looks to see if the host is running (by a simple nc connect to port 22)
2. if it is not running then it will run ec2_create with your args
3. then associate the ip
4. wait for the host to get the ip and become available
5. then it goes on to the next play which is normal ansible provisioning

which is perfect, for the next time you run – steps 2-4 won’t happen – it just goes directly to provisioning. Since ansible playbooks are idempotent you can run them as many times as you want w/o maligning your server.

Now the example playbook is all merged together – but if you use host_vars in your ansible inventory and you setup the check/create tasks as an includable task file – then the playbook for creation and provisioning of any new cloud instance you want becomes VERY short.

More to come as I finish it.

Syndicated 2012-10-25 22:33:37 from journal/notes

ansible and cloud notes

I’ve been working with ansible a lot recently. Fedora has a new public repo:
http://infrastructure.fedoraproject.org/infra/ansible/
and
http://infrastructure.fedoraproject.org/cgit/ansible.git/tree/

That’s just the beginnings of our repo for managing systems. There’s a number of details to work out, though and I have to merge over all the non-private things from our builder repo. I’ve started that process it’s just a matter of making reasonable decisions on where the tasks should live.

Recently I’ve been looking at:
https://github.com/ansible/ansible-provisioning

and figuring out how a euca2ools-based local_action module would work. I can spin up new instances, if they don’t already exist, and then provision them then I can create an entire network from more or less nothing.

Also working out how best to use euca-allocate-address and and associate-address in our infrastructure to provide some persistent and consistent servers/apps in the cloud. I think I have that bit hashed out but we’re discussing some tracking for cloud instances so we can sensibly shut them down when we need to w/o making everyone angry

I’m positive I can do all these things with ansible and euca/openstack. The question now is how long it will take to make them all happen. Anyone interested in helping out, come by #fedora-admin and talk to me.

 

 

Syndicated 2012-10-18 15:42:14 from journal/notes

What I want from schwab:

What I want from schwab:

I had told my partner that schwab had always treated me well. That
they had never made me jump through any ridiculous hoops. That I
could do everything via the web. No need for phone calls.

I told her this b/c I thought it was true. I told her this b/c
she’s deaf and it means a lot to her to be able to do things
entirely on her own w/o having to involve anyone else.

So it was a bit shocking when she was asked to call schwab to verify
the deposit amounts and complete the moneylink setup to your
brokerage service.

Worse yet, when I called to deal with the problem, with her at my
side, I was told no one could speak with me, despite having all
the requisite security information. Furthermore, I was told that
if she wanted to complete the process she would need to call in
via TDD or Relay operator.

This was all done in the name of security. Let’s explain a
bit about how security works.

A person is asked a series of questions that only they should know the
answers to. As a result that person is authenticated as actual. Whether
you ask these questions over the web, in person or on the phone, it
provides you with proof that the person you were speaking with is
authentic. Yes, sometimes that data is compromised by people intending
to defraud but if the person presents all the valid information,
it is impossible to know that.

This is the very basics of how it works:
Question, Answer, Authenticated.

In this case the questions were:
1. ssn
2. mother’s maiden name
3. account number
4. the precise amounts deposited into the other account
to moneylink with

Schwab requires that this information be presented ONLY over the phone.
They claim that this is to make it more secure. How or why a telephone
conversation is more secure is never offered. But I will let it go
for the moment their claim that it is. For a hearing person.

For a deaf person who is forced to use a Relay operator it means
connecting to a service, having them dial another number, then giving
all of the above information to a random person with whom you have
no contractual nor social relationship of any kind and virtually no
way to track them down, especially if you use the service more than
occasionally.

Please explain to me how that is MORE secure than having my partner
submit the information over an encrypted network connection?

If this were all done over the web at the end of the day the same
data exchange results but instead of making my partner angry
and resentful, you’ve made her feel happy. And you’ve not made me
into a liar when I told her that schwab wouldn’t make her do all that.

I want this corrected.

Here’s how you correct it:
1. Make the service entirely web-based and web-driven.

2. Provide a mechanism on your website where if a customer
needs to interact with a representative they can do so
in an encrypted chat window.

3. If you are going to try to force people to use the TTY and
or relay operators. Test it out for yourselves and make
your security audit and customer service people use it
with their own information. See how much trust they feel
in that system.

Syndicated 2012-10-12 13:13:29 from journal/notes

Ansible and Fedora Infrastructure

I’ve been getting some questions about why and how Fedora Infrastructure is trying out ansible for systems mgmt. So, I thought I’d write up the answers I’m regularly replying with.

https://fedoraproject.org/wiki/User:Skvidal/Ansible

Feel free to add comments, etc there.

Syndicated 2012-08-16 16:11:47 from journal/notes

func future and maintainer

Over the last year or so development on func has slowed and stopped. I’ve
been working on other projects and while I’m still USING func I’m not
working on it anymore. I asked about new maintainers to all the folks with
commit access and Steve Salevan stepped up! He’s been working on func for
a VERY long time and his new position has opened up opportunities for him
to enhance it further. I am happy that someone else will pick up func and
make it better and I’m also happy it is someone who has worked on it for
so long.

Thank you for being patient while things slumbered.

Syndicated 2012-08-03 15:33:55 from journal/notes

coprs/buildsystems

If you’ve been following my blog you’ve probably seen most of what I’m working on. However, at the request of my manager (hi Tom!) I wanted to collect what I’ve been working on and why. So I decided to put the pieces together on the fedora wiki:

https://fedoraproject.org/wiki/User:Skvidal/BuildSystem

If you’d like to help or to discuss email me or discuss on the fedora buildsys list.

Syndicated 2012-07-26 23:22:01 from journal/notes

ansible and builder creation/provisioning etc

I posted this to the ansible list but I thought I’d point it out here, too:

http://fedorapeople.org/cgit/skvidal/public_git/scripts.git/tree/ansible/start-prov-boot.py

For fedora infrastructure we wanted to make our builders trivial to refresh and reinstall. They hadn’t been that easy to do before and ideally I wanted something I could have done without any user intervention at all.

However, I did not want to just instantiate a pre-existing image for multiple reasons:

1. b/c it includes an implicit amount of staleness in the img – and ongoing maintenance

2. b/c it includes a certain amount of dirtiness b/c of the sysprep scripts are not as clean as a brand-new kickstart is

3. in any case I wanted something that could work in other places and with other virt tools.

The above is a first cut – but it mimics the vm kickstarting process we use for a lot of hosts in fedora infrastructure. However, it does it all using ansible to flow-control the whole thing.

I’m positive someone will mention some other mechanism I could have used to do the same thing. I’m not very interested, to be honest. I wanted something that:

1. didn’t require me to do a bunch of special magic in the kickstart %post to be able to molest/handle the instance  (in this case I just need to suck down an authorized_keys file)

2. didn’t require any special software installed on the instance (in this case ansible only needs a running sshd – which I cannot imagine installing a system w/o)

3. was relatively easy to work with for the future.

4. something that was so loosely tied to any specific instance-creation program or platform that I could easily swap out one for the other or make it modular w/o a massive restructuring.

5. was in python

anyway – it worked very nicely to spin up the 35 buildvm instances for the current koji rebuild and if nothing else, I’m quite pleased with that.

 

Syndicated 2012-07-19 19:55:55 from journal/notes

mockremote

I’ve been working with ansible and mock to make a simpler tool for building some pkgs on a remote system. I started on ftbfs but I wanted something for a less involved build process. Essentially a ‘build these pkgs over there and fetch back the results as you go’ command. So this is mockremote.

mockremote:
 mockremote will build a set of packages on a remote machine using mockchain. 
If you specify more than one package it will build them one at a time on the remote machine
allowing for each previous result to be used as a buildreq for the one before. Just like
mockchain it will will optionally build them recursively until it can no longer achieve
a successful build.
It retrieves the results from each build via rsync after each pkg completes. This ensures
that in the event of a remote system crash you lose the least number of pkgs.
If you restart the build, pointing to the same destdir it will only build those pkgs which
did not complete before.

 - requires mock, mockchain on remote system
 - requires the user you specify for the remote system be able to run mock/mockchain 
   (defaults to user mockbuilder)
 - requires ansible and rsync locally
 - requires you have ssh keys for the host on which you wish to build

 mockremote -b hostname -u username -r mock_chroot -a url pkg pkg pkg

It’s a pretty straightforward command and works nicely for building 
things in a set when your remote system is in a better place or has 
better bandwidth or just better cpu/ram combos to build a package
you want built.

I know what you’re thinking: “But can’t I do all this with ssh and running mockchain on the remote host by myself?”

and the answer to that is – yes – you can – this command just means you don’t HAVE to, which is nice.

Syndicated 2012-06-26 15:52:42 from journal/notes