Python Engineer Beginner Mistakes
I am writing Python code since 5 years, and I have worked with around 5-6 entry-level Python programmers. There is a pattern I can see in their programming practices. My hope is by reading this article there will be lesser such mistakes. I have worked on OpenStack for over three years, which is written in Python and has a very high code quality, so I feel I am qualified enough to write this post.Kubernetes in 30 minutes with minikube on Mac
Below are steps to create an express setup of Kubernetes on your Mac for quick use. Minikube is a small setup by Kubernetes guys, which will spawn a virtual machine and have a tiny (but fully functional) Kubernetes cluster inside the VM. kubectl (pronounced ‘kube (like ‘tube’) control`) is the command line client you'll use to connect to the Kubernetes cluster (which runs inside the VM created by minikube, in case you forgot :) )Make Vagrant Synced folders on Ubuntu Xenial guest work on Mac
Today I installed latest Vagrant, version 1.8.5. I was waiting for a newer release because 1.8.1 and previous versions don't work well with host-only networks for Ubuntu Xenial (16.04) guests on my Mac running El Capitan. But I faced another issue now: mount: unknown filesystem type 'vboxsf' This is while specifying a synced folder in my Vagrantfile with this line: config.vm.synced_folder("/Users/apple/src/myutils", "/home/ubuntu/myutils") where myutils is where I keep all my commandline shortcuts, tricks, and other shortcut-ish stuff.You dont need Vim swap files. And how to get rid of them
Estimated reading time: 5 minutes. Almost all IDEs today have ‘autosave’ feature. That is, you don't need to explicitly save a file. The file is automatically saved as you type, so that even if your computer crashes, you don't lose data. This makes me wonder why Vim's default behavior is of using swap files. Swap files are annoying. I've seen all Vim developers, including me, struggle with swap files at one point in their life.Everything you need to know about Tmux copy paste - Ubuntu
Copying from a Tmux session is something every Tmux user struggled with once. I'm listing out all the stuff I learnt in this blog. I tested everything on Ubuntu 14.04 Trusty Tahr, which runs Tmux version 1.8 (I have a few commands included for Tmux version 2+ also) To check your Tmux version, run tmux -V. If you have a Mac, see here. Know about copy buffers When you do a CTRL`+`c, the stuff you copy is stored in your computer's buffer, called ‘clipboard’ from where you can paste anywhere by doing a CTRL+'v’.Everything you need to know about Tmux copy paste
Copying from a Tmux session is something every Tmux user struggled with once. I'm listing out all the stuff I learnt in this blog. I tested everything on Mac OS X El Capitan. For Ubuntu/Linux guide, see here. Mac OS X El Capitan runs Tmux version 2.1. To check your Tmux version, run tmux -V. Know about your terminal The default terminal which comes with Mac is pretty limiting. Instead of listing the limitations (and workarounds around it), I'll just say that before you proceed, please install iTerm 2 and start using that immediately.Moving from Pelican to Hugo
I was looking for a better theme for my blog which is based on Pelican static site generator in Python, but then I happened to look at this blog written in Hugo - a Go counterpart of Pelican. The thing which caught my eye was the posts page, which is exactly how I wanted my first page to be. I've always wanted something like Paul Graham's essays page, but couldn't find anything close to it until I found this site.About
Hello! I am Rushi Agrawal. I work as a software engineer at Amazon. I graduated from IIT Hyderabad in 2012 with a Bachelors degree in Computer Science and Engineering. Before that, I did my schooling from my hometown Shegaon near Nagpur. Contact me via email at rushi.agr@gmail.com, or via phone at +91 99-4518-4519. Thank you for your time.Tmux session shortcuts
Tmux is awesome. But the session management commands are way too long for my liking. Listing all sessions is tmux list-sessions, attaching to a session mysession is tmux attach -t mysession, etc. So I created a few functions and aliases, which can be found here. The general idea is, all commands start with mx, which is basically a shortcut for ‘tMuX’. So mxl is to ‘l'ist tmux sessions, mxa is to ‘a'ttach to a tmux session, etc.A different approach to AWS CLI
For a user of Amazon Web Services, quickly viewing a dashboard of all resources is not straightforward. If using a web browser, the session won't persist for more than a day and one will have to re-authenticate every day. If using AWS CLI, then he/she might not remember the exact command (was it describe-key-pair or describe-keypair or describe-keypairs?) but even if they do, the output is a not-so-readable JSON. For viewing virtual machines, you might not want to see hundreds of lines of detail.Build VM Images using Diskimage-builder
OpenStack has this nice tool diskimage-builderto create virtual machine images without the need of a cloud. These vm images can then be uploaded to cloud (e.g. in Glance for OpenStack cloud), and they become immediately usable. You can also create VMs in virtualbox from these images (though I don't remember exact steps to make the image work with VirtualBox. Do let me know if you get the VM working with VirtualBox/Vagrant)Better way to view MySQL tables
The problem You are trying to print a MySQL table with a large number of columns, with SELECT * command. You type SELECT * FROM mysql.user LIMIT 1, and your terminal becomes this. If you wanted to view more than one row, you're in a trouble :) The solution Run MySQL with this option: mysql --pager="less --chop-long-lines --quit-if-one-screen --no-init' This will make your table display only the rows it can in the current screen, something like this.Quick JustDial scraper
So my friend asked me to scrape data from JustDial and give it to him in an excel sheet. I thought let's give it a try. He needed name of firm, address and phone number of any JustDial URL he wants to scrape. After effectively around 4 hours of work, the below script was created. Note that the script is dirty. You need to edit the jd_url to search any other URL.OpenStack Keystone with Cassandra Database in DevStack
Using Cassandra as backing database for OpenStack Keystone was our solution to multi-region deployment problem of OpenStack cloud. This blog post is not to discuss the problem. We are talking about how to have a development environment to play around with Keystone working with a dev Cassandra deployment. “Just run this script and you're all set!” I've put together all commands into a script which can be found here: https://raw.githubusercontent.com/rushiagr/keystone-cassandra/master/keystone-cassandra.sh If you have a fresh Ubuntu VM, just copy this script into that machine and execute it.Apache mod_wsgi parallelization notes
This is my notes on ‘Processes and Threading’ doc by the author of mod_wsgi module of Apache. This blog post just serves as a ‘quick refresher’, and is only useful if you have read the original document but it's been too long since you read it :) Apache with mod_wsgi A Python application can run with multiple processes as well as multiple threads with mod_wsgi. Prefork multiprocessing module Apache creates multiple processes, and each request is handled by one process.Docker quick start notes
Update *: Blog updated to use latest way to install docker. After reading about docker and containers, I thought let's try them out. Here are my notes. Obviously all of them are taken from Internet. Maybe this collection here will help someone start with docker faster than spending time searching all over the internet. It assumes your base OS is Ubuntu 14.04 Trusty Tahr (when was the last time you saw the codename spelled ‘Trusty Tahr’ and not ‘Trusty’?Cache APT packages with Squid proxy
TL;DR: Know how to install and set up Squid proxy, so that you can cache packages, and hence save bandwidth if you want to install those packages again and again. Also works if you are already behind a squid proxy. Problem: Repetitive download. Slow. If you deal with virtual machines a lot, you might know the pain of managing packages on each one of them. Every time I had to create a new VM, I would run apt-get update (to get information about all the latest packages available for my Ubuntu system), apt-get dist-upgrade (to install latest versions of all packages already installed), and also install some packages not present in stock Ubuntu image, like git (yes, it's 2015 and Ubuntu still doesn't come pre-installed with git), ipython, bwm-ng and some others.Puppet installation from modules
A quick example of how to use Puppet to install and manage MySQL. We'll download required Puppet modules from their git repositories. Again, everything is tried on Ubuntu (14.04). Make sure hostname -f shows your FQDN. Then install puppet sudo apt-get install puppet We'll use git submodules to manage different git repositories. But first, create our own repository mkdir puppet-mysql cd puppet-mysql git init Install Puppet modules stdlib and mysql into directory modules as git submodules.iSCSI administration on Ubuntu - Quick Start
This post get's you started with iSCSI administration on an Ubuntu machine. Although I have used Ubuntu Trusty (14.04) version, it should work with Precise (12.04) too, with the latest packages. Prerequisites Make sure you have atleast a little idea of what these terms mean: iSCSI, LUN, IQN, initiator, target and portal. Google and wikipedia are your friends. A quick summary: There are two parts of iSCSI communication - initiator and target.OpenStack Unit Testing Nuggets
A small post about little things I found out while running unit tests in OpenStack. Unit-testing setup Everybody knows ./run_tests.sh is used to run the unit tests of an OpenStack project. But, you require to install dependencies before doing it. And installing dependencies might not always succeed. So make sure you install these packages before running pip install -r requirements.txt: sudo apt-get install build-essential libssl-dev libffi-dev \ python-dev libxslt1-dev libpq-dev python-mysqldb \ libmysqlclient-dev libvirt-dev Atleast cinder and nova dependencies will get installed properly after this.Amazon EC2 API with OpenStack - Developer Quick Start
OpenStack has support for EC2 API, that is, you can fire Amazon's API against an OpenStack cloud and it will still work. This article gets you started with using them locally against DevStack. It is more of a starter guide to a novice. Fire a DevStack with it's default settings. See this post for more information on it. git clone http://github.com/openstack-dev/devstack cd devstack/ ./stack.sh Source openrc source openrc View all EC2 credentials available for the current user (here, demo user in demo tenant)DevStack behind proxy
I have now seen some people struggling to get DevStack working behind proxy. Some, thinking it is a bug in DevStack, have actually posted patches for it too! Here, I'll tell you the simple way to get stack.sh complete succesfully from behind a proxy. By default, devstack will clone from the ‘actual’ OpenStack git repositories, residing at git://git.openstack.org. Some people might face a problem with it, as DevStack uses git protocol to clone the repo.Mounting QCOW2 images
Isn't it fun that even before you start a VM out of an image, you can add files to that image, see and edit the directory and file structure of that VM? I wanted to boot a VM out of a disk-image, but how will I know out of the 256 available IPs for that VM, which one actually got assigned? I tried vnc console, but the connection was terribly flaky.Open Source Puppet - Quick Start
This post aims to be your quickest guide to get started with Puppet. We'll be using the open source version of Puppet. An hour of spare time and two Ubuntu machines (physical or virtual doesn't matter) is all that is needed. Quick Introduction Lets say you want to install and run apache server on one of the machines in your lab. On another, you want to create a new user. On a third machine, you want to install MySQL, and allow access to this machine only from the first server.OpenStack in an hour with DevStack
So you found out a cool new technology “OpenStack” and want to try it real quick? Or probably you are hired in a company for your Python skills and now you are supposed to work on OpenStack in the shortest possible time? Fear not, it is not that hard to get started. DevStack is your friend-in-need. No, don't click that hyperlink just yet :)
MySQL Cheat Sheet
Databases are so important, yet almost all the time I need to work with it, I find that I have already forgotten all the syntax! So here I am writing down a quick cheat sheet to get me up and running when I'm waking up from slumber. I hope this will help atleast one other guy on this planet. I use mostly Ubuntu, so some of the commands might be Ubuntu specific.Playing around with Cinder multi-backend
With Grizzly release, Cinder got equipped with another major feature – multi-backends with filter scheduler. So now you can have more than one storage boxes for block storage and manage them with one Cinder deployment. Here, I'm going to test it out using our favourite method – trying it out on DevStack!
Quick Start: Linux Logical Volume Manager
While installing the latest Ubuntu OS on your computer, you will see that you can install the OS using LVM (Logical Volume Manager) utility. Ever wonder what is it? LVM (Logical Volume Manager) is that fantastic utility for storage administration. It provide the users with abilities which were not possible with raw disks. The storage is now ‘virtualized’. You can now easily create, move and extend volumes (for now, think of it as disk partitions) without bothering about data corruption.Installing OpenID plugin for MediaWiki
This post is about setting up your wiki such that their users access the wiki only via an OpenID provider login (e.g. Google or Facebook login). This post assumes MediaWiki is already installed.
Linux Screens in DevStack
This blog article explains handling screens within OpenStack running as a DevStack setup. Some useful generic screen commands are also provided at the end.
Cinder on DevStack - Quick Start
This blog post covers all the important commands for OpenStack Cinder - The block storage project. Although this guide is written based on DevStack, it will work with actual deployment of OpenStack cloud also (except the DevStack/Linux specific bits).
OpenStack Tempest quick start with DevStack
UPDATE(April 2014): Now when you set up DevStack, Tempest is automatically set up for you.
This quick start guide explains setting up Tempest with a DevStack environment for Grizzly release. Most of the information here is taken from here, the only difference being this blog post is more recent, and is based on Grizzly.
Git(hub) Cheat Sheet
I am posting my github cheat sheet here. I started writing it as soon as I started learning Github and Git. So, some of the content here can appear quite naive. I will try to keep this blog post as updated as I can, and if you find any suggestion, please comment!
This post is just for a reference of commands. This post will be a bad way to learn how git works. A basic understanding of git is assumed.