A Docker Based Development Environment

At jnaapti, we have been using containerization technologies since almost the very early days. In 2011, when I was evaluating a solution to provide light-weight containers to our learners in the Virtual Coach, I was told that the only mature solution was to use Virtualization solutions. But Virtualization was too slow (in terms of boot up time) and I didn’t have enough resources to keep stand-by nodes running all the time. So after some evaluation, I decided to use LXC and it served its purpose. However, there were several features missing and I was in the verge of building a few of them myself.

So it’s not surprising that when I discovered Docker, I just fell in love with it. One of the first things we did was we moved our LXC based learner containers to Docker. We then slowly started to migrate portions of our infrastructure to Docker. We achieved full Docker migration last November, and then also moved our staging/testing systems in the cloud to Docker.

The last in the list was to migrate our development environments. The initial migration wasn’t too hard, because of 2 things:

  1. We already had Docker in production – so it was a matter of working off our production Dockerfiles
  2. We were already using KVM in our development – so we already had a clear idea about what containers our system should be running.

Why the move away from KVM based development environments? Simple! KVM’s disk usage is too much to suit our requirements. We had our development images of 5G each and you have 10’s of such images and are soon out of disk space. I am not sure how many Docker fans will approve some of the things I discuss here but I believe that this is a much leaner solution than using Virtualization and I don’t see any issues in the way I am doing it.

So with the migration from KVM to Docker, there were a few additional things that I wanted to handle:

  1. Can we use Desktop tools like text-editors with data in Docker? Imagine I am writing a NodeJS application. I want to use Atom installed in my host to write code. However, I want to run NodeJS inside the container.
  2. It is very easy to work with command line utilities (which don’t need X) in containers, but how about Desktop utilities like Eclipse? Can we run this in a Docker container and still have the same user experience as a regular app? What are some best practices to do so?
  3. Is it possible to expose devices into Docker containers – this is required for eg, if we are doing Android development and want to debug our app in an actual Android device

The first 2 were rather easy and I sailed through it. The third one, I struggled a little, but I finally made some head-way.

This post attempts to capture some of my learnings in this entire process in case you want to build a similar environment. So let me answer these questions:

Using Desktop tools with Docker containers

This one is easy. A practice we follow is all the code that we write is inside a mounted volume. Containers are used to run processes in a contained fashion, but the processes are manipulating files that are in our host (and not in the underlying diff file system). We can as well remove our containers and we don’t lose anything.

Here is a sample run to demonstrate this:

Start a Docker container that has NodeJS installed in it. Make sure that this container has access to a local host directory (in this case /home/gautham/Desktop/node-data):

gautham@ananya:~|? docker run -d -P -v /home/gautham/Desktop/node-data:/data --name "node-example" ananya-nodejs:0.0.1
820e105db5061b380e6117e42a0cabad5f00c54e54f5016aefc18399e2a2eb25

Check if the container is running

gautham@ananya:~|? docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
820e105db506 ananya-nodejs:0.0.1 "/usr/sbin/sshd -D" 9 seconds ago Up 8 seconds 0.0.0.0:49155->22/tcp node-example

Using Atom to edit files in a volume shared with a Docker container

Inside the container (which is accessible via SSH):
gautham@ananya:~|? ssh -p 49155 ubuntu@localhost
ubuntu@localhost's password:
Last login: Tue Apr 14 14:13:12 2015 from 172.17.42.1
ubuntu@820e105db506:~$ cd /data/
ubuntu@820e105db506:/data$ ls
hello.js
ubuntu@820e105db506:/data$ node hello.js
Hello World!

Using Desktop tools inside Docker containers

This one initially seemed a little difficult, but I figured out soon.

Create an image that has lxde-core package installed in it.

Now, there are 2 options:

Connect to lxde running inside the Docker container

Run these commands in host:

docker run -d -P ananya-desktop:0.0.1
sudo su
xinit -- :1 &

This will now switch you to a different terminal (accessible at Ctrl+Alt+F8). You will also see a white terminal. Type the following in this terminal to start LXDE:

docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
38280dd66b98 ananya-android:0.0.1 "/usr/sbin/sshd -D" 4 minutes ago Up 4 minutes 0.0.0.0:49156->22/tcp, 0.0.0.0:49157->5901/tcp hopeful_lumiere
ssh -X -p 49156 ubuntu@localhost

In the Docker container run

startlxde

You should now see a full fledged Desktop running like this!

LXDE running in a Docker container

So your host is running at Ctrl+Alt+F7 and your Docker container is at Ctrl+Alt+F8. Use this option in case you are running many Desktop applications and you want to be totally isolated from the host when working with the applications in the container (i.e you are not using any host applications in conjunction with the applications in the Docker container).

Only run the application that you are interested in

I found this option to be better in some ways. I have my Android Studio setup using this option now.

gautham@ananya:~|? docker run -d -P ananya-desktop:0.0.1
gautham@ananya:~|? ssh -X -p 49156 ubuntu@localhost
ubuntu@localhost's password:
Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.16.0-34-generic x86_64)

* Documentation: https://help.ubuntu.com/
Last login: Tue Apr 14 14:25:47 2015 from 172.17.42.1
ubuntu@38280dd66b98:~$ cd android-studio/
ubuntu@38280dd66b98:~/android-studio$ ls
ubuntu@38280dd66b98:~/android-studio$ bin/studio.sh

And lo and behold!

Anroid Studio Running in a Docker Container

Accessing devices within Docker container

A final requirement was whether we can get Docker to detect USB devices. I found that if you pass a –privileged flag and mount the /dev/ device appropriately, you can then access it in the Docker container. I was able to successfully use adb along with my Docker container.

docker run --privileged -v /data:/data -v /dev/bus/usb:/dev/bus/usb -d -P ananya-android:0.0.1

Docker has been a boon and although there are several area of improvement, I see that it has a future. It has become an indispensable tool in our software tools arsenal in jnaapti.

Cross-posted here: http://jnaapti.com/blog/2015/04/14/a-docker-based-development-environment/

The BullsNCows Application in AngularJS

About 5 years back, when I was still dabbling with client-side technologies for the first time, I had written the BullsNCows game in pure Javascript.

My knowledge of Javascript and client side technologies has evolved quite a bit since then. So, recently when I was looking at this code I thought, “This code could be better if we use AngularJS“.

So today I thought of rewriting this in AngularJS and using Bootstrap for styling. Here is the code in Github in case you want to play with it and here is the game in case you want to play.

bullsncows

So what’s changed?

  • All the code related to styling has been moved to Bootstrap. The alternate table coloring seems so much easier now!
  • The update of the table is done with AngularJS so I don’t need to do complex DOM manipulation – it is a straight-forward binding to the tr element of the table.
  • The data handling and algorithm to compute bulls and cows is slightly improved. It used to be a O(n^2) algorithm. It’s now linear.
  • Elimination of globals – except the app object nothing is global now.

Jnaapti – One Year On…

Time flies!

It has been a year since I started Jnaapti and it is time to take a step back and reflect on how things have fared during the last one year. I was reading through the mails and the Jnaapti weekly reports and was wondering if there is anything that I have to add to the 6 month report that I wrote back in December or if there is anything that I don’t agree to among the things I mentioned in that list.

So here goes:

  • Naming: Somehow, I don’t recall reading this anywhere, or atleast, didn’t think it was important. But, just like you do usability testing of your product, I think it is important to get the name tested with a few people before you christen your company. While choosing an Indian name, also remember to include people from various parts of India and abroad.
    The name Jnaapti is a Sanskrit word which means to “acquire knowledge”. I had a few things in mind when choosing the name:

    1. I wanted an Indian name – because the target segment was Indian to begin with
    2. I didn’t want to have a problem with branding, in acquiring domains etc
    3. Finally, I wanted the name to be generic enough to be used until the vision of the company has been achieved and shouldn’t require a change as I pivot.

    Turns out, people don’t get it.

    Here are some suggestions to people about naming a company:

    • Make sure the word is such that it can be pronounced by anyone and is easy to spell over the phone. You will loose out on a lot of branding if people search for the wrong keywords.
    • How would people come across your name? Would you expect people to search for it – if not, the logo along with the name should be catchy.
    • What are the closest other names to the name of your company? Is it easy to mis-spell it? If people search for it now, will Google suggest a more popular alternative? Do you expect to override it anytime soon?
    • Is your audience global or local? If global, go for something that is not specific to your geography or atleast easy for people outside your geography to pronounce.
    • Stay with the chosen word for atleast 3-4 days. Imagine yourself using it in conversations and see if you are comfortable with the word even after 3-4 days.

    Personally, I prefer short, single words for company names.

  • Pivoting is not free: Don’t get me wrong. I am a big fan of lean principles and I still religiously follow a lot of them. And I do understand that pivoting is not about abandoning one idea and moving onto the other or changing your vision as if caught in a tornado but I guess it is possible that people think pivoting comes without a cost when in fact the reality is not so. In short, pivoting has the same issues as knowing when to shut down your startup – should we persevere or should we rethink?
    When you have validated a part of your business model (eg: the demand, the problem/solution fit or say portions of your product) and decide to pivot another aspect of it, there is a good chance of temporary disruption in the flow (read target segment, getting users, revenue streams etc). It’s quite possible that you had been generating revenues due to your efforts, but now you don’t see it to be sustainable, so you decide to pivot. This may mean, you have to re-design your site/app experience, come up with a new marketing message and start over on the marketing efforts, and you need to be prepared to bear the cost. This may be tough, especially when you are bootstrapping the company. But then, you don’t have a choice because the whole idea of pivoting is to reduce the likelihood of reaching a destination that you didn’t intend to reach in the first place.
    Between validation and scale is an important step – “validating the scale”. As described in the lean stack, there are 4 stages when it comes to building a product – Understand Problem, Define Solution, Validate Qualitatively, Verify Quantitatively. However the MVP that works in Stage 3 need not work in Stage 4 and may require a strategy pivot, so be careful in Stage 3 to ensure that the same solution holds when you get to Stage 4. This is easier said than done.
    Recognize failures early – atleast have a weekly review of how things are going with regard to the company’s directions and make sure you make corrections in your path. Get it validated by people who can guide you.
  • Follow the ant philosophy: Learn from the horse’s mouth as he has described it well.
    Perseverance is important. Sometimes it feels like you are putting in a lot of effort, but things don’t seem to work, but sometimes the exact opposite is true. I have seen this with marketing efforts. It takes time for marketing to reach fruition and it is important not to lose your patience when the going gets tough.
  • Have a strong social network or add someone to your team who does: I can’t say this enough and this builds on the 2 things I mentioned in my previous post – “Get out of the building” and “The strange story of co-incidences and chance meetings”. It always helps to have a strong social network (both online and offline). The number of new people I have met (online/offline) has crossed 200 since the time I started Jnaapti.
  • Self discipline revisited: If you have decided on a rhythm, stick with it. Get good at reducing interruptions and handling unavoidable interruptions well. When meeting clients, time is not under your control, so get good at squeezing tasks into available slots between unavoidable interruptions. Time discipline is extremely important especially when you are working alone. Maintain a log of where you are spending your time and optimize what you can optimize. Get good at continual automation not just with engineering processes but with life! Make a note of key metrics that you want to track on a daily/weekly basis.
  • Be prepared to learn: This may seem obvious, but sometimes you may not realize the effort involved. Running a one-person startup has meant that I had to learn how businesses function, understand sales and marketing, negotiation, valuation, funding, recruitment, building and maintaining relationships, understand lead generation, cost of customer acquisition, designing usable products, build a great user experience and a host of other things in addition to engineering. I like it and is one of the reasons I am doing this, but then, not everyone does.

When I started this journey, it was supposed to be a one year stride. The fact that it is Jun 2012 and I haven’t lost confidence means that this is going somewhere and I intend to continue along this path into the near future…

Jnaapti – A 6 month Report

This post is not quite about Jnaapti‘s vision and why I am doing it, but more about things that I have learnt since I founded Jnaapti.

I can’t believe it has been more than 6 months already and what an experience it has been! I can easily blog a few hundred pages about my experiences, but here is the MVP (Minimum Valuable Post) – the key things that other wantrepreneurs and newbie entrepreneurs can learn from:

  • The art of self-discipline: If there is one thing that I should pick among the various things that I learnt in the last 6 months, I would pick being self-disciplined and maintaining a rhythm in the face of distractions that are not under your control. Time management is extremely important. When you are handling all aspects of business, there is a good chance that you need to meet someone at their convenience, or you are asked to provide a service (in my case training) and these are not under your control. So you need to get really good at handling these, and getting back to your work as soon as this is complete. For productivity freaks, I use a combination of Pomodoro technique, Hamster, Workrave, this hack and a simple Libreoffice document to manage all my data. I wish it could have been simpler and someday, I hope to continue my efforts in building productivity tools.
  • Get out of the building: I can’t stress this enough. Nearly everytime I got out of the building and started speaking to people, there was some new insight or a business deal that happened. A lot of the times, it seems purely co-incidental (read the next point). The initial couple of weeks were very haphazard and it just seemed like a humongous project that was lying in front of me. While I did have a huge list of tasks/ideas to work on, I didn’t know where to begin. What should I build now? No clear answers. As I started hunting for answers, I ran into the Lean Startup concepts and fell in love with Ash Maurya, Eric Ries and Steve Blank‘s philosophies. I followed the lean canvas way of capturing the business model religiously and it has worked out very well. If anything, it helped reduce noise, cut to the chase and build something of value “now”. The idea of prioritizing validation and learning before scale has worked in my favor. The first cut of Jnaapti’s offering didn’t even have a software component and was entirely conducted over email – because I realized that the MVP didn’t need a software component. I concentrated less on the website messaging, since most of my contacts were through my personal network.
  • The strange story of co-incidences and chance meetings: Nearly every business contract I got seems like a pure chance meeting and the more this happens the more you believe in what is written here. This kind of also proves that there is a critical mass of startups in Bangalore now, and there is a good chance that you will bump into someone who is a startup founder or early employee when you visit coffee shops or technical events. This has worked out so well that I have made it a point to meet, on an average one or two new people every week. Till date, I have met more than 80 new people (including Jnaapti Learners), and that’s an average of 10 people per month!
  • My experiments with pricing: An early learning for me was about how you need to get pricing right and it’s almost always never. The inital couple of months I was demanding way too less for my services and slowly as I found out the value of my services, I tweaked it to a point which seems reasonable both for me and my clients. But that said, every new assignment is a new negotiation.
  • Understanding waste reduction: When you are the only person working, and you have the 3 aspects of an organization in front of you – business, product and engineering – every moment you spend, you need to think twice. Is this the best use of my time? It helps a lot to reflect on how you spent your time and look at what you could have rather NOT done. This is the art of waste reduction.
    On a similar note, I have started maintaining what are called “Implementation reports” for Jnaapti’s product development phases. An implementation report is an assessment of

    • how a specific phase fared
    • what features we set out to build
    • what was our reasoning about why they should be built
    • how are the features being used now
    • if they are not beng used, why so
    • can its usage be improved
    • should it be chucked
    • how can we ensure that we don’t have such waste in future
  • Getting to ramen profitability: There is one thing I can tell you. We, as software engineers in India, don’t value money (I am speaking generally). We get an awesome package (stop cribbing you guys) right out of college, and then we get into this spiral of work-get paid-work harder-get promoted-ask for a raise. And when things don’t work in our favor, we start complaining or switch our jobs looking for a better role or more pay. If you want to value money better, quit your job to know what others go through. In the face of inflation, the rupee v/s dollar prices and rising infrastructure costs it has been tough to keep up, but then you got to pay a price. I am close to achieveing ramen profitability and I hope things will be better next year. And while achieving ramen profitability is important, it is equally important to ensure that you achieve profitability via the business you are in and not via some job that doesn’t help you validate or learn more about your business. And yeah, if you want a piece of advice, to get to ramen profitability soon, just invest your savings in some place that is not easy to suck from!

I have taken enough breaks and I never felt that this was too demanding (although I do put in long hours once in a while). Needless to say, I am loving the freedom and the flexibility that I have and I wish I can continue this forever! Overall, it has been an amazing 6 months!

Moving your WordPress blog from Apache to Cherokee in 30 minutes

In my post on VPS Hosting experiences, I had mentioned that inspite of doing server configuration tweaking, I found that load times were gradually increasing and I was experimenting with an alternative server named Cherokee.

The whole migration took less than half a day – including learning Cherokee, trying out locally and then using it in my blog. So what are the steps I followed to move to Cherokee?

I use Ubuntu 10.04 LTS as my dev system as well as on the production server – so one of the things that I am confident about is that, if something works in my dev environment, it is bound to work in the production setup, with minimal pains during deployment. So I wanted to first try out the entire setup – make sure everything is fine, and then replicate the setup on the production server.

I started by installing Cherokee from the PPA and also php5-cgi:

add-apt-repository ppa:cherokee-webserver/ppa
apt-get update
apt-get install cherokee
apt-get install php5-cgi

Continue reading Moving your WordPress blog from Apache to Cherokee in 30 minutes

VPS Hosting Experiences

So after the frustrating experiences with my shared hosting provider, I decided to move to VPS hosting once and for all. I knew that this would mean, spending more money, and having to spend more time and energy tweaking configurations and monitoring the site, I thought it will be worth the effort and price for the flexibility that I would get from it.

So sometime in late December, I made the move. After looking around and asking a few people, I finally decided to go with VPS.net. The movement from shared hosting to VPS was a breeze and I was up and running in under 2 hours. The experience with VPS.net until now has been pleasant.

Meanwhile, I am closely monitoring Google Webmaster Central and there are some very interesting observations and that is what I wanted to share here.

Google Webmaster Central data for buzypi.in
Google Webmaster Central data for buzypi.in
  • Gzipped Content
    The first observation is how, when I moved from shared hosting to VPS, the data download size reduced drastically with no significant change in the number of pages crawled per day. This is because I use GZIP encoding, while my shared host did not (when you pay for bandwidth there is no incentive to reduce the size, now is there?!)
  • Improvement in load times
    The second observation is how the time to download also reduce drastically when I moved to VPS hosting. This was expected. While my server now has only my services running, I am not sure how many umpteen other websites were being served on my shared host.
  • Server configuration tweaking
    Towards mid Jan, the load times started increasing. This is because I had a few other services hosted on the same machine and the server started thrashing. The biggest issue with most VPS providers is that they are very lenient on bandwidth and storage, but very stingy when it comes to memory. So I had 2 choices – either I upgrade my configuration and pay nearly twice the price, or I start playing with the Apache and PHP configurations and see if I can squeeze out more performance from the system. I decided to go for the latter. I cut down on the services hosted, disabled unnecessary modules, played with threads and child processes, and tweaking PHP configurations. But no matter what I did, the load times stayed up there, or worse, continued to increase and there was nothing I could think of.
    Recently a friend of mine asked me to give Cherokee a try. Cherokee is considered to be blazingly fast and very lightweight compared to Apache. So I have moved my blog to Cherokee now and hope to monitor the performance closely over the next few days.
  • Google on steroids
    Another observation is how Google suddenly decided to give my site a real test – and decided to download virtually all the pages possible in a single day – this happened a couple of days back and I am yet to discover why this happened. What I am happy about is that the load times were decent when this happened.
  • Load times and Google Ranking
    I can confirm that there is some corelation between page load times and rankings in Google. In December, when my site was taking as many as 3 seconds to load (Google said my site was slower than 94% of the sites in the world!) – some of the keywords for which my posts used to appear in the first page moved to the second or third pages. It was only in January did I see them come back to their original positions.

Overall, it has been a good experience – you learn a lot when you moved to VPS!

Google Docs, ODF and Data Portability

Consider the code below to display a line of text in HTML:


<style>
.paragraph-text {font-family: Arial; font-size: 11pt; font-weight: normal; text-decoration: none}
</style>
...
<p><span class="paragraph-text">Here is a test line</span></p>

Now let’s say, we see some developer write it this way:


<style>
.T1_1 {font-family: Arial; font-size: 11pt; font-weight: normal; text-decoration: none}
.T1_2 {font-family: Arial; font-size: 11pt; font-weight: normal; text-decoration: none}
.T1_3 {font-family: Arial; font-size: 11pt; font-weight: normal; text-decoration: none}
.T1_4 {font-family: Arial; font-size: 11pt; font-weight: normal; text-decoration: none}
.T1_5 {font-family: Arial; font-size: 11pt; font-weight: normal; text-decoration: none}
.T1_6 {font-family: Arial; font-size: 11pt; font-weight: normal; text-decoration: none}
.T1_7 {font-family: Arial; font-size: 11pt; font-weight: normal; text-decoration: none}
.T1_8 {font-family: Arial; font-size: 11pt; font-weight: normal; text-decoration: none}
.T1_9 {font-family: Arial; font-size: 11pt; font-weight: normal; text-decoration: none}
</style>
...
<p class="P1">
<span class="T1_1">Here</span>
<span class="T1_2"> </span>
<span class="T1_3">is</span>
<span class="T1_4"> </span>
<span class="T1_5">a</span>
<span class="T1_6"> </span>
<span class="T1_7">test</span>
<span class="T1_8"> </span>
<span class="T1_9">line</span>
</p>

What would you say of the quality of the markup above?

Continue reading Google Docs, ODF and Data Portability

Fun with X – hacks in Ubuntu

I have always wanted to have more control over my GUI windows so that I can control them and monitor them via commands. So when I looked around for tools to help me do that I found a couple in Ubuntu that help me do some pretty nifty stuff.

So here are a couple of quick hacks:

  • Monitoring what you do on a daily basis:
    Productivity geeks will love this! This is a script which can help you know what the active window title is and print it into a file. Just cron this script to run once a minute and you will have a pretty good picture of what you do on a daily basis. Want to know how much time you spend on Facebook? Try this script!


    printf "`date "+%D %T"`: " >> active-window-log
    active=`xprop -root | grep -P "^_NET_ACTIVE_WINDOW" | awk '{print $NF}'`
    title=`xwininfo -id $active | grep -P "^xwin"` >> active-window-log

    Try this script for a day and you will be amazed by the insights you get by looking at the output log.

  • Monitoring a terminal for changes: How many times has it so happened that you had a terminal which was running a script and you were waiting for it to complete, but then you didn’t know how long it would take before it printed the next line of output? Wouldn’t it be great if you could monitor it?

    So here is the command that you can use to monitor changes to a specific terminal:

    window_id=`xwininfo | grep "Window id" | sed -e 's/.*Window id: //g' -e 's/ .*//g'`;
    xwd -id "$window_id" > /tmp/initial;
    while true;
    do xwd -id "$window_id" > /tmp/final;
    if [ -z "`diff /tmp/initial /tmp/final`" ];
    then echo "No diff";
    else echo "Windows differ";
    rm /tmp/final /tmp/initial;
    break;
    fi;
    sleep 3;
    done

    When you run this script, your mouse pointer changes to a cross-hair. Just point and click the terminal that you want to monitor and then allow it to continue with its work. As soon as there are changes, it will print “Windows differ”. Now you can as well make it send you a mail or start playing a song!

    The cool thing about this script is, it shows how flexible Unix tooling is. The way the script works is by taking a ‘screenshot’ of the terminal every 3 seconds and comparing it with the original screenshot. If there is a change, then diff outputs something and hence becomes non-zero.

The scripts have been tested in Ubuntu. If you don’t have any of the commands, Ubuntu should prompt you to apt-get install them. Although not tested, it should work in any X based system.

A review of the Nokia 5230 (Nuron) Touchscreen Smartphone

I have been waiting for a sub 200$ smart phone which is either Linux based or Android based and can work seamlessly in Ubuntu. For some weird reason I have been extemely stingy about paying for smart phones – I really don’t see a point in shelling out 30k (Rs) for a phone, when you can buy a laptop for a similar price or a netbook for half the price. Dell Netbooks are available for as low as 300$, and there are a few android tablets such as the OlivePad being introduced for even cheaper prices. Considering this I feel it is a stretch to even pay 200$ for a phone.

So when I learnt about the Nokia 5230, I was excited. It was neither Android based nor Linux based, but I could atleast keep up with the developments in the mobile space at a cheap price. It didn’t take me long to decide to buy one.

This is the first smartphone I have owned, so needless to say I am excited about the features. All I really care about in a phone (other than the regular call/messaging features) is the browsing and PDA capabilities of the phone. I had earlier decided to go for the Nokia 5233, but it does not have GPS, and the 5230 comes with AGPS for an extra Rs. 1000/- so I decided to go for it.

Continue reading A review of the Nokia 5230 (Nuron) Touchscreen Smartphone