Hosting an MVC3 (with membership) application on EC2

February 4, 2012

One of my side projects was to get an MVC3 application that uses the Razor View Engine and Membership hosted on EC2 running Linux. I found some amazingly helpful resources along the way – particularly from Nathan Bridgewater at Integrated Web Systems.

Step one of the project is to get an EC2 instance prepped and ready.  Basically I followed the cookbook instructions on Bridgewater’s site - Get Started with Amazon EC2, Run Your .Net MVC3 (RAZOR) Site in the Clould with Linux Mono.

The exact commands I used:

Create new AMI ID ami-ccf405a5 and associate elastic IP (xx.xx.xx.xx)
sudo apt-get update &;& sudo apt-get dist-upgrade –y
wget http://badgerports.org/directhex.ppa.asc
sudo apt-key add directhex.ppa.asc
sudo apt-get install python-software-properties
sudo add-apt-repository 'deb http://ppa.launchpad.net/directhex/ppa/ubuntu lucid main'
sudo apt-get update
sudo apt-get install mono-apache-server4 mono-devel libapache2-mod-mono
cd /srv
sudo mkdir www; cd www
sudo mkdir default
sudo chown www-data:www-data default
sudo chmod 755 default
cd /etc/apache2/sites-available/
sudo vi mono-default (see mono-default, change IP address)
cd /etc/apache2/sites-enabled
sudo rm 000-default
sudo ln -s /etc/apache2/sites-available/mono-default 000-mono
sudo mv /var/www/index.html /srv/www/default
sudo vi /srv/www/default/index.html
sudo apt-get install apache2
sudo service apache2 restart
Test in a browser via IP address (you should see the default apache page)

My mono default:

# xx.xx.xx.xx is my Elastic IP address
  ServerName xx.xx.xx.xx
  ServerAdmin myemail@domain.com
  DocumentRoot /srv/www/default
  MonoServerPath xx.xx.xx.xx "/usr/bin/mod-mono-server4"
  MonoDebug xx.xx.xx.xx true
  MonoSetEnv xx.xx.xx.xx MONO_IOMAP=all
  MonoApplications xx.xx.xx.xx "/:/srv/www/default"

    Allow from all
    Order allow,deny
    MonoSetServerAlias xx.xx.xx.xx
    SetHandler mono
    SetOutputFilter DEFLATE
    SetEnvIfNoCase Request_URI "\.(?:gif|jpe?g|png)$" no-gzip dont-vary

    AddOutputFilterByType DEFLATE text/html text/plain text/xml text/javascript

Step two is to test mono with a simple Asp.net page.  Put this file into /srv/www/default.  Edit with sudo and view via browser at http://xx.xx.xx.xx/test.aspx.

<%@ Page Language="C#" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>ASP.Net Test page</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<script runat="server">
private void Page_Load(Object sender, EventArgs e)
{
lblTest.Text = "This is a successful test.";
}
</script>
</head>
<body>
<h1>
This is a test page</h1>
<asp:Label runat="server" ID="lblTest"></asp:Label>
</body>
</html>

If problems are encountered check logs in /var/log/apache2/access.log or /var/log/apache2/error.log
Step three is to get MySql installed and tested with this simple application.

sudo apt-get install mysql-server
sudo apt-get install libmysql6.1-cil
CREATE DATABASE sample; USE sample;
CREATE TABLE test (id INT AUTO_INCREMENT PRIMARY KEY, name VARCHAR(25));
INSERT INTO sample.test VALUES (null, 'Lucy');
INSERT INTO sample.test VALUES (null, 'Ivan');
INSERT INTO sample.test VALUES (null, 'Nicole');
INSERT INTO sample.test VALUES (null, 'Ursula');
INSERT INTO sample.test VALUES (null, 'Xavier');
CREATE USER 'testuser'@'localhost' IDENTIFIED BY 'somepassword';
GRANT ALL PRIVILEGES ON sample.* TO 'testuser'@'localhost';
FLUSH PRIVILEGES;

Put this file into /srv/www/default. Edit with sudo and view via browser at

<%@ Page Language="C#" %>
<%@ Import Namespace="System.Data" %>
<%@ Import Namespace="MySql.Data.MySqlClient" %>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>ASP and MySQL Test Page</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<script runat="server">
private void Page_Load(Object sender, EventArgs e)
{
string connectionString = "Server=127.0.0.1;Database=sample;User ID=testuser;Password=somepassword;Pooling=false;";
MySqlConnection dbcon = new MySqlConnection(connectionString);
dbcon.Open();

MySqlDataAdapter adapter = new MySqlDataAdapter("SELECT * FROM test", dbcon);
DataSet ds = new DataSet();
adapter.Fill(ds, "result");

dbcon.Close();
dbcon = null;

SampleControl.DataSource = ds.Tables["result"];
SampleControl.DataBind();
}
</script>
</head>
<body>
<h1>Testing Sample Database</h1>
<asp:DataGrid runat="server" ID="SampleControl" />
</body>
</html>

Step four is to get the simplest possible MVC3 Razor application functioning on Ubuntu / EC2.  Again Bridgewater has a more detailed explanation of what to do at his website linked here.

  1. Go into Visual Studio 2010 and create a new project MV3 / Razor project making no changes to the default project template.
  2. Build it and locally.
  3. Ensure that these references are set to “copy local”: System.Web.Mvc, System.Web.Helpers, and System.Web.Routing
  4. Copy System.Web.Razor, System.Web.WebPages, System.Web.WebPages.Razor, System.Web.WebPages.Deployment into your application’s bin directory.  You will find these files in in C:\Program Files (x86)\Microsoft ASP.NET\ASP.NET Web Pages\v1.0\Assemblies
  5. Publish the application to a scratch directory
  6. Copy the published application to your EC2 machine.  I used git bash to tar (tarr –zcvf aws.tar.gz *) the files as Bridgewater recommends but could not get scp to work so I ftp’d the file over.
  7. On the EC2 machine cd /srv/www/default; sudo mv /home/ubuntu/aws.tar.gz; sudo tar –zxvf *.gz; sudo chown –R www-data;www-data *; sudo chmod 755 *; sudo service restart apache2 restart
  8. Confirm working from browser by checking default IP address http://xx.xx.xxx
  9. NB: I had to hit refresh several times before the application would work.

Step five is to use implement membership using MySQL.

  1. On your Windows machine.  Edit the default controller and decorate it with the [Authorize] attribute.
  2. Edit your web.config shown below.  This is where it can get hairy.  If you want to run this locally on Windows you need to install the MySQL connector for .Net and Mono http://dev.mysql.com/downloads/connector/net/.  Make sure that you reference system.web.  On Ubuntu the application uses system.data.  The trick is to add them both so you can run the same code on Ubuntu and Windows.  Also notice that I’ve made database password clear text.  As Nathan notes this is not a good practice.
  3. On the Ubuntu machine Go into MySQL and create a database called membership.
  4. Deploy the application to EC2 and test the application using step 4.

<?xml version="1.0"?>

<!--
 For more information on how to configure your ASP.NET application, please visit
 http://go.microsoft.com/fwlink/?LinkId=152368
 -->

<configuration>
 <connectionStrings>
 <add name="Default"
 connectionString="data source=127.0.0.1;user id=aspnet_user;
 password=secret_password;database=membership;"
 providerName="MySql.Data.MySqlClient" />
 </connectionStrings>

<system.web>
 <compilation debug="true" targetFramework="4.0">
 <assemblies>
 <add assembly="System.Web.Abstractions, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
 <add assembly="System.Web.Routing, Version=4.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
 <add assembly="System.Web.Mvc, Version=2.0.0.0, Culture=neutral, PublicKeyToken=31BF3856AD364E35" />
 </assemblies>
 </compilation>

<authentication mode="Forms">
 <forms loginUrl="~/Account/LogOn" path="/" timeout="2880" />
 </authentication>

<!--NOTE that "hashed" isn't supported with the public release of MySql.Web 6.3.5 under
 Mono runtime. But I can't bring myself to share sample code that doesn't hash the
 passwords by default. ;) The version included with this sample project is slightly modified to
 allow hashed passwords in Mono. I highly recommend checking out the latest version of
 MySql .NET Connector. http://dev.mysql.com

 Also, I found that you have to rebuild MySql.Data and MySql.Web
 using .NET 4.0 profile if you want it to work with Asp.Net 4.0 under Mono. This is a known bug and should
 be published in upcoming versions of the connector. -->
 <membership defaultProvider="MySqlMembershipProvider">
 <providers>
 <clear/>
 <add name="MySqlMembershipProvider"
 type="MySql.Web.Security.MySQLMembershipProvider, mysql.web"
 connectionStringName="Default"
 enablePasswordRetrieval="false"
 enablePasswordReset="true"
 requiresQuestionAndAnswer="false"
 requiresUniqueEmail="true"
 passwordFormat="hashed"
 maxInvalidPasswordAttempts="5"
 minRequiredPasswordLength="6"
 minRequiredNonalphanumericCharacters="0"
 passwordAttemptWindow="10"
 applicationName="/"
 autogenerateschema="true"/>
 </providers>
 </membership>

<roleManager enabled="true" defaultProvider="MySqlRoleProvider">
 <providers>
 <clear/>
 <add connectionStringName="Default"
 applicationName="/"
 name="MySqlRoleProvider"
 type="MySql.Web.Security.MySQLRoleProvider, mysql.web"
 autogenerateschema="true"/>
 </providers>
 </roleManager>

<profile>
 <providers>
 <clear/>
 <add type="MySql.Web.Security.MySqlProfileProvider, mysql.web"
 name="MySqlProfileProvider"
 applicationName="/"
 connectionStringName="Default"
 autogenerateschema="true"/>
 </providers>
 </profile>

<pages>
 <namespaces>
 <add namespace="System.Web.Mvc" />
 <add namespace="System.Web.Mvc.Ajax" />
 <add namespace="System.Web.Mvc.Html" />
 <add namespace="System.Web.Routing" />
 </namespaces>
 </pages>

<!--Don't forget to update this... I left it open to make it easier to debug.-->
 <customErrors mode="Off"/>
 </system.web>

<system.data>
 <DbProviderFactories>
 <clear/>
 <add name="MySQL Data Provider"
 description="ADO.Net driver for MySQL"
 invariant="MySql.Data.MySqlClient"
 type="MySql.Data.MySqlClient.MySqlClientFactory, MySql.Data"/>
 </DbProviderFactories>
 </system.data>

<system.webServer>
 <validation validateIntegratedModeConfiguration="false"/>
 <modules runAllManagedModulesForAllRequests="true"/>
 </system.webServer>

<runtime>
 <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
 <dependentAssembly>
 <assemblyIdentity name="System.Web.Mvc" publicKeyToken="31bf3856ad364e35" />
 <bindingRedirect oldVersion="1.0.0.0" newVersion="2.0.0.0" />
 </dependentAssembly>
 </assemblyBinding>
 </runtime>
</configuration>


HTML5 Validation using Yepnope

November 23, 2011

In this months’ MSDN magazine there is an interesting article on Browser and Feature Detection.  What really caught my eye was the piece on Modernizr, the JavaScript library that implements browser feature detection.

Modernizr has built-in detection for most HTML5 and CSS3 features that’s very easy to use in your code. It’s very widely adopted and constantly enhanced. Both Modernizr and jQuery are shipped with the ASP.NET MVC tools.

As HTML5 and CSS3 become more and more prevalent feature detection is increasingly relevant.

A growing number of ready-made “fallbacks” for many HTML5 features, known as shims and polyfills, can ease that burden. These come in the form of CSS and JavaScript libraries or sometimes even as Flash or Silverlight controls that you can use in your project, adding missing HTML5 features to browsers that don’t otherwise support them. The difference between shims and polyfills is that shims only mimic a feature and each has its own proprietary API, while polyfills emulate both the HTML5 feature itself and its exact API. So, generally speaking, using a polyfill saves you the hassle of having to adopt a proprietary API.  The HTML5 Cross Browser Polyfills collection on github contains a growing list of available shims and polyfills.

As .Net developers we are usually insulated from browser incompatibility issues, however, there may be situations where you are not using the .Net framework.  There is a good example of using yepnope (a conditional resource loader) for HTML5 Form Validation at CSSKarma.  I tweaked and re-implemented  the example using a .Net MVC application which you can see below.  A couple of “gotchas”:

· Modernizr is part of the .Net MVC tools that come from Microsoft, however, yepnope is not in the release of the MVC tools that at least I have on my machine.  You will need to download version 2.0 from modernizr.comto get yepnope support.

· The JQuery syntax for working with YepNope takes some getting used to.  Make sure to load JQuery first.

· Visual Studio 2010 does not seem to know about HTML5 and will warn of a validation when you use its attributes. See figure #2.

clip_image002[4]

Figure 1 – MyValidation.js – based on the example at CSSKarma

clip_image004

Figure 2 – HTML5 Source

clip_image006

Figure 3 – Output

References:

http://yepnopejs.com/

http://haz.io/

http://modernizr.com


Software Due Diligence – Part III

September 29, 2011

Software due diligence is a bit like having a home inspection done when purchasing a house.  Some problems are more serious than other.  For example, if you find that there is mold or asbestos in the basement that might be a reason to walk away.  Like a home inspection in most cases, the diligence does not reveal such serious problems with the software that you will want to back out of the deal entirely, it is typical that you may want to re-consider your valuation or take steps to manage the transition.

Red Flags

  • Architecture that will not scale (possible walk away)
  • Application cannot be re-built / run from source control checkout
  • Inability to engender a sense of confidence that the solution really works
  • Lack of forethought (this is where I’d like to go)
  • Architecture that cannot be cleanly expressed
  • Prima donna developers
  • Absence of technical leadership
  • Reliance on obsolete technology (i.e., Delphi)
  • Business logic consistently found in the presentation layer
  • Absence of any documentation whatsoever
  • Critical code that no one owns (i.e., that was developed by abc who isn’t here anymore)
  • Serious ethical breakdowns

Reasonable expectations

  • Absence of perfect documentation (even the best organizations are challenged to have up to date documentation)
  • At least one thing that impresses you as “world-class” (the more the better)
  • Good code
  • Finding that there are one or two go-to people
  • People wear many hats (product manager, QA, developer, etc.)
  • Insufficient infrastructure
  • Reliance on free services
  • Lack of published standards, metrics, or formal process
  • Informal bug tracking
  • Out of date off-the-shelf software
  • Limited requirements documents

Things to give you pause

  • Non-homogenous technology configuration
  • Bleeding edge / specialized technology (e.g., Cassandra, assembly)
  • Dependence on a service provided for free (Google Translate API)
  • Sloppy code
  • Lack of appreciation of the competition
  • Insufficient knowledge of best practices (JQuery vs. JavaScript)
  • How well will the system under examination compliment / be incompatible with existing systems?
  • Are some areas more complete than others? (some code is more battle tested)
  • Is there something that should be patented?
  • Poor user interface
  • General sense of mediocrity

Software Due Diligence – Part II

September 29, 2011

Operations

  • Review of current architecture either via documentation and whiteboard.
  • Watch the application run from an OS console. (e.g., top, perfmon)
  • Watch the application run from purpose-built administrator tools.
  • Describe the hosting architecture.
  • Where is the system hosted?
  • How redundant is the system?
  • How is the system monitored?
  • What are the biggest bottlenecks in the system?
  • Has your system ever been compromised?
  • Characterize the reliability of your system.
  • Have you done any vulnerability or penetration testing?
  • How would you handle 10X volume, 100X volume, 1000X volume? (This is a big one.)
  • Inventory of hardware and software (technology) assets.
  • Where is your source code stored?

Software

  • Review the source code (looking for good coding practices, clean architecture, exception handling, etc.).
  • Review database schema and query the live database (or copy of live).
  • Inventory custom components and software license agreements.
  • Are there any public or private APIs?
  • Review (developer) documentation associated with the code.
  • Review user-facing documentation and/or training materials.
  • Build all applications from source code and deploy to hosting environment.
  • Is there a debug / development interface?
  • Is there a database of customer feature requests or open issues?
  • Are any obsolete technologies (i.e. Delphi) in use?

People

  • Meet key employees and get to know their backgrounds.
  • Who is the go to person?
  • How do people collaborate?
  • Describe your SDLC?
  • Where do requirements come from?
  • How is the software tested?

Product

  • See a demo of all products, utilities, and supporting software.
  • Product Roadmap: Recent, Past, Present, Future.
  • Review current business model and sales process.
  • Are there any prototypes or product concepts that we should see or discuss? (These can be hidden gems)

Software Due Diligence – Part I

September 29, 2011

I was recently asked about what goes into software due diligence.  This is the first of three posts on this topic.  In this post I outline my thoughts on the process itself.  Part II is my working list of questions.  Finally, part III are some thoughts about what to expect and when to walk away.

There are a bunch of good checklists out there for buying an entire company.  See references below.  Most of these checklists talk about software diligence relatively generically. After looking at a number of different organizations for a variety of different reasons I’ve built myself a checklist that may be useful starting point for others. I think my checklist is most applicable for medium sized applications (~1MM SLOC) built by teams of 3-10 people. Larger applications probably warrant a more sophisticated approach.

This post is not about valuing the business, assessing the product, or anything to do with market position. It is all about looking at the code, how the code is hosted, and assessing the technical assets of a business. If you are doing a project for a VC they typically want to know if there are any “red flags.” There are two types of red flags – those that are correctable and those that are not. A correctable red flag is something like lack of off-site backups or a non-redundant server. An uncorrectable red flag (which typically means walk away) are prima donna developers, limited / no documentation, or an architecture that cannot scale. If you are doing a project for a business that is trying to integrate a property with their own systems they want to know about the red flags but they also want an understanding of what they are going to be inheriting and what it’s going to take to make it useful as fast as possible.

Invariably the technical examination will overlap with looking at the business itself. For example, when buying a product that claims to have a million users its prudent to query the database and see that there are at least a million email addresses in the database. Similarly, the business development folks are often after any information that they can use to value the business or close the deal.

I think that much of technology due diligence is common sense. The good news is that, if it’s done right, you quickly get a feel for the goodness of the product.  A process that has served me well is to sit in on the preliminary conversations (which are often over the phone) with the management team. I may / may not ask some general questions during that meeting. I will then follow-up with a call to the CTO (or technology lead) to get into more detail. The goal of the technology call is to confirm my understanding of the technology stack and to set expectations for an on-site visit.

I am not a big fan of questionnaires.  I believe an on-site visit is critical to getting a good understanding of the technology. More recently I’ve been challenged to find a place to visit as many of the principals work remotely from themselves. I think it is important to meet the people face to face. My primary objective is to learn as much as I can about the technology as possible. My secondary objective is to meet the development team and form an option about their respective competencies. By its nature diligence is the process of looking for problems. Rarely do you come back from looking at a business thinking that you under estimated how good it is. On the other hand I can think of many occasions where I’ve come away blown away by the people – their technical acumen, tenacity, and single minded determination to make something work.

Picking who goes on-site is a particularly important consideration. I think the minimum number is two – one subject matter expert on the business and someone who is proficient with coding in the language of the business being acquired. (You do not want a .Net person doing a Ruby on Rails evaluation). If budget permits an IT person is a very nice to have resource. Their perspective often compliments that of the business and developers.

References:

 


State of software development – 2011

July 13, 2011

The three big trends influencing computing in 2011 are mobile, social, and Software as a Service (SaaS). Start-ups and established companies alike are pushing themselves to deliver products and services that address these markets.  If you are a professional software developer you very like use:

  • · C or C++ if you work on an application where an interpreted language is not acceptable – performance intensive application (CPU or memory), security, hardware, etc.;
  • · Java if you work for IBM, Sun, Salesforce.com, or some other big organization / platform that does not embrace Microsoft’s technology stack;
  • · .Net (probably C#) if you work for a (typically mid-to-large) organization that embraces the Microsoft stack; 
  • · Java if you write Android applications; Objective C if you write iPhone/iPad applications;
  • · PHP, Perl, or Python if you write web applications on a LAMP platform that is more than 3 years old; or,
  • · Ruby on Rails if you write web applications on a LAMP platform that is less than 3 years old.

If you code in something else you are in the minority.  This can be a dual edge sword.  The good is that you are by definition a specialist and can command higher compensation.  The downside is that there may not be demand for your particular skill. 

Platform as a Service

Platform as a Service (PaaS) service, a component of SaaS – where the hosting platform is moved to the cloud, is clearly maturing.  If an application takes off organizations now have multiple viable options for quickly adding capacity.  Amazon’s EC2 and Microsoft’s Azure are two of the more well known platforms.  For example, Mashable documents how mobile phone app Instagr.am went from 0 to 3M users in less than six months with EC2.

The concept of PaaS is somewhat game-changing for start-ups.  Essentially this means that you only need to invest in the computing power that you need at the moment.  Where 10 years ago a major component of the cost of starting a web business would be the data center, today you only pay for what you use.  If I were to boot-strap a company like Instagr.am I would start with a small number of physical servers in a low-end colo facility.  (See askwebhosting.com to see how cheaply this can be accomplished.)  Only once growth justified it would I move to EC2 (LAMP) or Azure (.Net).

I do believe that Azure is potentially game changing for Microsoft.  Prior to Azure you would never (and I do mean never) hear of start-ups using Microsoft’s technology.  At worst Microsoft’s development eco-system is as good as anything available in the LAMP stack – many people find it superior.  The problem was the cost of deploying Windows/SQL server was prohibitive for all but the largest organizations.  It will be interesting to see if Azure changes the equation.

Mobile

Today mobile application development requires you to be either a Hatfield or a McCoy.  You either are in the iPhone or the Android camp. Mobile application development is dominated by applications written in Objective C for Apple devices and Java for Android devices. I am aware of one cross platform tool Titanium.  While I have not used it personally I do not hear good things about it.  My sense is that you can get away with using Titanium if your mobile app is very straight-forward and / or doesn’t’ take advantage of any device specific functionality. 

Last summer I posted a link to TIOBE Software’s index of most popular programming languages. I went back and re-visited the numbers for this year and see only incremental change from a developer’s point of view. TIBOE update their rankings each month – this is the July report. They use data from Google, Bing, Yahoo!, Wikipedia, YouTube, and Baidu to calculate its ratings. I found another site langpop.com which uses a variety of different mechanisms to calculate popularity. Broadly speaking the two charts correlate. I like the TIOBE formatting and that they update their results each month.

Capture

Source: TIOBE Programming Community Index for July 2011

In addition to these languages listed above if you work on the web a professional software developers also needs know:

  • · HTML, XML, JSON, JavaScript, and have rudimentary CSS knowledge (1)
  • · Basic SQL skills – MySQL, SQL Server, or PL/SQL
  • · JQuery JavaScript library
  • · Implement AJAX in their respective language of choice

(1) Developers rarely have advanced CSS / Photoshop knowledge. Typically commercial designers are used for this type of work.

image

Source: LangPop Language Popularity Normalized Comparison – Updated April 13, 2011


Setting Asp.Net Session Using JavaScript

July 5, 2010

I recently inherited a piece of C# code that required me to set the .Net Session from JavaScript.  (In this particular application a child window manipulates an image and the resulting state is set in a Session variable used by other modules.  This is not necessarily how I would have done it, however, this the way that the system is presently implemented.  Once Session is set the child window needs to be closed.)  There are many different examples of how to read Session variables from JavaScript but I didn’t find much on setting Session from JavaScript.  This post demonstrates how I was able to set Session by dynamically creating a 1×1 pixel image.

1.   In the calling web form create the following simple JavaScript function.  In my production application I set the Session then wait half a second then close the window.


2.   Create a new webform that sets Session in the Page_Load method and returns a 1×1 pixel.  For the purpose of this demo I dynamically created a GUID, however, in the production application I read the QueryString variable.

3.  Source code for a demo application follows.

Read the rest of this entry »


Lean and Agile

June 10, 2010

I recently stumbled across a document called the “Lean Primer” by Craig Larman and Bas Vodde.  This document is based on the “Toyota Way” which strives to build a culture of that results in sustainably great products.  Although the Toyota Way is based on techniques used in automobile manufacturing I was struck by how applicable the framework is to software development.  Indeed these parallels have not escaped the document authors who wrote a book on the subject - Scaling Lean & Agile Development: Thinking and Organizational Tools for Large-Scale Scrum – and advise companies on applying lean thinking.

The Primer quotes the book “Inside the Mind of Toyota: Management Principles for Enduring Growth”  and states that “the essence of successful lean thinking is ‘building people, then building products’ and a culture of ‘challenging the status quo.’”  The first part of that statement sounds like it could be taken right out of Tom Peters Good to Great – first the who, then the what.

The Larman and Vodde summarize lean thinking using a “house diagram” based on a 1973 diagram from Toyota’s then chairman Fujio Cho.  The majority of the 45 page Lean Primer is devoted to explaining the house diagram.

At the top of the house is the goal of lean: deliver value fast.  The shorter the cycle-time the better.  This principle is fully transferrable to agile development which emphasizes getting working software into real user’s hands as fast as possible.   I love their metaphor of the lake and rocks.  “When the water is high (large batch or inventory size, or long-cycle time) many rocks are hidden.”

The first pillar “Respect for People” struck me as very analogous to a principle found in agile development where the manager is supports the team and allows individual engineers take ownership for their work.  In an agile world, a manager’s job is to remove obstacles, coach, and offer expertise.  It is not to micro-manage tasks and assignments, bury people in a storm of email, or unnecessarily tie them up with administrivia.  Similarly the sprint planning meeting, daily scrum meetings, and the end of sprint review replace the thrashing that can occur when requirements change in a world of big-bang releases.

The rightmost pillar of the Lean house is “Continuous Improvement” and encompasses several useful ideas.

  • Go see which reminds me of Management by Walking Around which I believe originated at HP.  In the world of software management I believe it means looking at code yourself and actively being involved in the process whether by writing small sections of code, testing, or pitching in wherever necessary.  For example, when he was at Microsoft Bill Gates was famous for actively participating in code reviews.
  • The 5 whys refers to the notion of considering the same challenging problem 5 times.  For example, why does the server keep crashing?  Often times it’s useful to visually represent the problem as a fishbone diagram.
  • The Sprint Retrospective is also highly compatible with the Continuous Improvement Pillar.  The Retrospective is an opportunity for the team to discuss what went well, not so well, and what would be done differently.  In the Army we used to call this the After Action Review.

The center pillar refers to a number of familiar concepts broadly categorized in terms of Product Development and the 14 Principles.

  • Several of the product development bullets relate to the notion of leaders that build great teams that passionately care about the products that they build, the people they work with, the customers that use the products, and are grounded in a sense of purpose.  In my view this type of leadership cannot be taught – your either have it or you don’t.  Great teams, however, can be nurtured by senior management that is committed to a people-centric culture – employees and customers.
  • The technique of having a team room and visual management is one that I have found to be particularly effective.  The notion of having visual artifacts such as progress on a burn down list, the organization of database tables, or the layout of a user experience is consistently valuable in terms of communicating understanding.
  • Cadence is the heart-beat of a regular software development life-cycle.  “People appreciate or want rhythms in their lives and work.”  Timeboxing work into short cycles where a reasonable amount of work can be done well is in my mind the equivalent of an agile sprint.

Toyota’s immense historical success using “Lean” development methods validates much of what software developers embracing agile methods have believed for some time.  The Lean Primer is a very worthwhile read and is highly recommended.


What’s the most popular programming language

June 3, 2010

Today a friend sent me a link to what I think is an absolutely fascinating survey of the most popular programming languages (circa May 2010).  From where I sit this looks to be spot on.

As my friend said you could stare at this all day.  Some of the things that jump out at me:

  • C, Java, and C++ are the tier I languages.  If you can code in one of these languages you will never be out of work for long.
  • If you club together Visual Basic with C# and call them “.Net” then you can say that the tier IA languages for web development are PHP and .Net with Perl and Python falling behind as the tier II web languages.
  • As pointed out in the original post Objective-C cracked the top 10 programming languages almost certainly on the back of the popularity of the iPhone/iPad.   Clicking on the Objective-C link shows a big spike in middle of 2009.  Classic tipping point.
  • Although still the #5 most popular language the trend graph suggests that Visual Basic is dying.  It’s fallen off significantly over the past 12 months.  This is entirely consistent with my experience as a .Net developer.  Although the syntax is very similar I don’t know anyone that does VB.Net commercially.  It’s all C#.

  • In my experience the LAMP stack (with the “P” largely being PHP) is the platform of choice for start-ups for the obvious cost reasons associated with licensing Windows.  The Microsoft stack is typically found in enterprises that have already licensed windows.  The interesting middle ground is Mono – .Net running on a non-Windows platform such as Linux.
  • Given the number of high-profile sites (Twitter, Basecamp, etc.) that run on Ruby on Rails I was somewhat surprised to see Ruby’s relatively low score.  Part of the issue may be that the how different Ruby syntax is from other languages.   For example, an experienced C# programmer can relatively quickly become productive in Java.  I think over time Rails apps will become more and more prevalent likely at the expense of PHP.  I don’t know much about it but I’ve heard good thing about Groovy on Grails (Rails in a Java-like language).
  • What I don’t at all see on the list are any of the Salesforce.com specific languages like Apex Code or Visualforce.

More on Page Speed Tuning

May 14, 2010

A few weeks back I captured some notes about tuning page speed.  Since then I’ve learned a few more things.

  1. We found out the hard way that compression doesn’t seem to work right with IE6.   As unfortunate as it may be IE6 still accounts for something close to 10% of web traffic – more on my site.  Paul Turner documented how to change httpd.conf in his blog so that IE6 users don’t get compression.  I’m happy to see big companies like Salesforce.com and Google dropping ceasing to support it.
  2. One more thing to remember about working with compression is that it’s most effective on textual information like HTML, CSS, and JavaScript.  Typically web-ready images and binary downloads have already been (or should have been) compressed.  While the web server will happily try to compress these files there will be little benefit and it can slow down your webserver.
  3. Another tool that was recommended to me was the YUI Compressor which minifies JavaScript and CSS.  Minification removes any unnecessary characters from code without breaking the functionality.
  4. One note I found interesting on the Yahoo performance blog is that they estimate that between 40% and 60% of all page views are done with an empty cache meaning that the primary strategy should be to keep all pages as light as possible.  Everything else is a secondary strategy.   The other obvious thing to be cautious about when working with caching is the potential for serving stale content.  When properly implemented an HTTP 304:Not Modified status-code will be returned.  With these caveats in mind we’ve experimented with various header configurations changes to improve cachability.  Askapache.com has a number of suggestions that can be easily implemented in a httpd.conf or .htaccess file for adding cache-control headers and adding future expires headers for static files.  For dynamic pages (PHP, ASPX) headers should be set manually.
  5. Finally, all of these strategies are focused on improving web-server to browser experience.  There are a whole series of optimizations that can be done to get the content to the web-server including optimizing SQL queries using stored procedures or tools like memcache.

Follow

Get every new post delivered to your Inbox.

Join 45 other followers