How the Other Half Works: an Adventure in the Low Status of Software Engineers

When do bash init scripts run?

After years of trying to remember when all of my bash init scripts run, asking officemates what order they’re in, or looking it up in books, I finally found a chart the lays it all out for me. Bravo to Solipsys Limited for figuring it out and making the graph.

Xbox Live Indie Marketplace – The Sales Funnel from Heck!

Almost 3 years ago (to the day) some friends and I released Hoardzz to the Xbox Indie Marketplace.  We managed to botch our release by messing up our demo, which really ended up hurting our sales.  Even though Hoardzz rates in the top quartile of Xbox Indie games (it’s a pretty fun multiplayer party game if I do say so myself. It’s only a buck so you should totally buy it), our sales are abysmal.  Although Hoardzz hasn’t made a lot of money, it has been a great learning experience and a harsh introduction to the sales funnel.

Leads, prospects, and customers converts to see your game on the marketplace, download the demo, and buy the game.

In my engineering world I was trained to look at getting people that saw my work to buy my work.  I was NOT trained in getting more people to see my work.  While we were great at converting our demos to purchases (some of this was probably due to underpricing), we were not looking at the whole funnel!  This meant that as soon as we fell off the new release page, we stopped getting so many downloads, which knocked us off the best selling today page, which pretty muched killed sales.

This is the world you traverse, going through 10 levels plus 3 unlockable levels, until the final boss is vanquished.  The unlockable levels provide more challenging game play for those that taking gaming a bit more seriously.  I’ve only beat one of the lockable levels a couple of times, but it gives you sweet new stuff!

For our next release Hoardzz Heroes (now available), a prequel to Hoardzz, we’ve decided (experience now gives us that decision) to be a lot smarter.  But before I get to that, let’s look at some of the attempts we made to revitalize Hoardzz.

Hoardzz Heroes is a dungeon crawler type game where our valiant heroes get turned into bugs and sent to a strange new world. We’ve got a great story line for those that like the full experience and the option to hit start to skip it for those that just want the action.

PR from the Xbox Live Indie Game (XBLIG) community brought in some extra views for us.  The problem is most of them would only cover the new releases, ironically the ones I feel need it least.  The biggest notable exception is Writing of Mass Deduction, who is trying to find great games no one knows about (it’s a tricky task and much appreciated). However, even with the great boost that it sites do provide, a given site is only going to feature you once and so you need to figure out what more you can do on your own.

All of our heroes get turned into bugs by an evil spell. This is the unlockable hero the butterfly. You’ll have to unlock the Dinosaur Graveyard, then beat it in order to get her.

My next step was advertising, mainly as a learning step to see if it would even be worth it.  I hadn’t seen a whole lot of other people doing it and I couldn’t find anyone who had written anything about it.  As I thought about how I could cheaply experiment with ads, I realized that the Xbox Live Indie Marketplace had a sales funnel from heck!  In internet marketing, you want the smallest amount of effort possible from seeing the ad to playing the game.  This means you want to advertise on the Xbox, not on the internet.  This creates a more complicated sales funnel with many more requirements:

1. Your ad shows up on their screen (This ain’t free)
2. They pay attention to your ad (Yeah, like anyone does that anymore)
3. They have an Xbox (Just because they like it doesn’t mean the own one)
4. They can get to the Indie Marketplace (Many have never used it)
5. They can remember your game’s name (Remember, they probably saw your ad on another device quite a while ago. Generally, recall is show to do poorly.)
6. They think to search for your game (Let’s see…what indie game should I try.  Oh yeah, I saw an ad for Hoardzz maybe I’ll try that!  <- almost never happens)
7. They actually find it and download the demo (At this point we have 10% conversion)
8. They are willing/able to put money onto their Xbox Live account (minors don’t have credit cards)
9. They like your game
10. Purchase!

We’ve got 7 items and 2 additional unlockables. The best item varies with which character you are playing with. My favorite is the bomb shown exploding in this image. There’s nothing quite like hearing the echo on the BOOM! to warm your soul.

When the average Xbox Indie game sells for about a dollar, there really isn’t a lot of money that can go into ad spending.  I did a few small test runs by running targeted Facebook ads with people who were over 18 and also said they like Xbox Live Indie Marketplace games. My limited runs yielded about $1 of sales for every $1 (10,000-20,000 views) spent on ads.  Not only did this not make money, the volume was also very low so it wouldn’t even be able to create any buzz.  The combination of small target market, brutal sales funnel, and inability to cheaply reach your target market coupled with the ridiculously low prices makes the Xbox Indie Marketplace a vicious place to compete.

This is one of our bonus levels. Personally I think it is more fun with 4 players, but alas, I am a lone ranger depending on no one to save me from my follies.

With this next release we’ve decided to try something new.  We want to remove as many of the obstacles in the sales funnel as possible, and so we’ve hijacked the loading screens to occasionally show the ads of our other games.  Its free for us, it targets people who already have bought games so they usually can buy more, and we feel it is a service to our users because it saves them from wading the the sea of garbage games that is the Xbox marketplace.

Simple, and hopefully effective. We probably need to make the name “Hoardzz” a bit more prominent, but we’ll see.

It is unfortunate that Microsoft has chosen to put so little effort into the indie community. It also seems a little foolish given the success of games on the other app markets.  I just don’t think it would be hard to add the ability to pay to feature your game, it would do wonders for games that are great games that aren’t the top rated, top sold today, or top all-time seller.  While the Xbox development tools are fantastic, I have a hard time envisioning myself starting another game on the Xbox…although we do have some already in the pipeline.

---

If you are a blogger or site admin that covers Xbox Indie Games, tell me what site you run and I’ll shoot you a promo token.  If you have a game of similar quality and downloads, we’d love to cross-promote during loading screens.  If you like free Indie Xbox games, follow me on twitter.  I’ll be dropping tokens on #xblig once Hoardzz Heroes is about to fall off the the New Releases page.  To subscribe to new releases in the Hoardzz series, like us on Facebook.

 

You can contact me on twitter @ricochet2200

The Builder Entrepreneur

When I hear stories about entrepreneurs as kids, they always seems to involve kids going door-to-door selling stuff.  They have some kind of arbitrage play where they buy low, sell high and make a little bit of money (or sometimes quite a bit) doing it.  That was never me.  I hate door-to-door sales, I hated the fundraisers that made me go door-to-door.  I would always sell just enough to get the lowest tier, 5 cent prize so that I could at least get something.  I have no compulsive need to sell stuff and I never will.

What I did have was a compulsive need to build stuff.  Cool stuff.  Complicated stuff that required me to learn theory and apply it to practice.  My first love was music and so my parents put me in guitar lessons.  I was living in Hawai’i at the time so I ended up learning Hawaiian Slack Key Guitar.  I got good enough to play most of the songs on Revere (see CD cover below), and also played the baritone in the school band and picked up piano on the side.  Music is awesome.

Hal Kinnaman was my teacher. Hawaiian Slack Key Guitar is chill, laid back guitar music…the kind you actually hear people play if you go out in the old sugar cane plantation communities.

When I moved away, I dropped guitar but managed to learn enough from a friend to do improv piano.   It started with some theory and some drills, and through hours and hours of practice I began to get good enough to make up melodies on the fly…you could call it JIT music composition.  I loved it, the technical creativity was invigorating and relaxing all at once.  I soon developed a compulsive need to play the piano, and do piano drills, and get better.  It satisfied my need to build complicated, cool stuff.

College was fantastic, but there was a notable shortage of pianos.  My creative itches were going crazy without an outlet and fortunately my computer science major scratched it unexpectedly well.  I didn’t realize it at the time, but for me writing code is more about creativity than playing with tech.  Fred Brooks explains it surprisingly well:

The programmer, like the poet, works only slightly removed from pure thought-stuff. He builds his castles in the air, from air, creating by exertion of the imagination. Few media of creation are so flexible, so easy to polish and rework, so readily capable of realizing grand conceptual structures….

Yet the program construct, unlike the poet’s words, is real in the sense that it moves and works, producing visible outputs separate from the construct itself. […] The magic of myth and legend has come true in our time. One types the correct incantation on a keyboard, and a display screen comes to life, showing things that never were nor could be.

—The Mythical Man-Month

Over time I have learned that I have a compulsive need to build things  (video games, robots, electronics kits, and companies).  However unlike the salesperson who is expected to go start a company or “do business”, my compulsion leads people to expect me to shutup, work for the man, and sit in a cubicle (or worse an open floor plan) with no natural light.  That’s just not me.

This the least creepy looking Larry Ellison picture I could find.

I’ve been told that Larry Ellison likes to say, “If you don’t build it and you don’t sell it, I don’t know what you do for me.”  If the two  parts of a software company are build and sell, why is the compulsive seller viewed as “the entrepreneur” and the compulsive builder viewed as “the engineer” destined to work for the man (Silicon Valley has been doing a good job at changing this perception)?  I believe the answer may be risk tolerance.  Sales commissions immediately weed out anyone without risk tolerance the same way college weeds out those that aren’t technically minded enough to code.  The result is that while there are still engineers with high risk tolerance, they are diluted by those who have never been culled by that requirement (this also applies to social skills, business sense, style, athleticism, and a ton of other things).

If you are an engineer that has tolerance to risk, maybe you are a Builder Entrepreneur.  Go find a salesperson with some business sense and become the next Bill Gates and Steve Ballmer, or the next Steve Jobs and Woz.  Become the next Eric, Sergey, and Larry.  Innovation requires risk, and our economy needs more jobs.  Best of all, there are already plenty of aspiring sales entrepreneurs that need you.

I Hate Open Floorplans, It Makes Roger Come Out…

Last year I visited Adobe’s office in the Bay Area.  It was a wicked awesome office with a ton of really cool things, basketball court on the roof, wind turbines, and designer space. However, when they went on to tout their collaborative open workspaces I realized I couldn’t work there.

On the first day on one of my jobs, I came into work and was given a paper with information needed to do my job.  It was highly organized and efficient but I noticed the name on the paper was for Roger Smith.  “Is there anything else you need?” asked the office administrator.

“Um…my name is not Roger,” I replied.

“Oh…” she said not sure how to respond.  “….um…What is it?”

I kept my Roger ID card and soon had the nickname of Roger.  It was kind of a fun inside joke that the office enjoyed, and I was instantly known by everyone.  I’d occasionally put my name down as Roger Smith and the joke kept going.

A couple years later I got married to my great wife (now mother of my two boys).  She would sometimes call me at work and would get confused because I was so “angry” with her.  I was disengaged and gave short answers.  I was not the guy I was at home, and so she started referring me at work as Roger.  She couldn’t figure out why I was sweet at home but distant when she called me at work.  She was not a fan of Roger.  Over the years I have talked to many other programmers and many of us seem to have this problem.  We are not actually angry, we are just lost in our own worlds and are still waking up from it.

Peopleware spends quite a bit of time explaining what it calls “The Flow”.  It’s that hyperproductive state you get in where you all of the sudden get a ton of stuff done.  You look at the clock and hours have passed by and you were completely unaware.  It is awesome.  It needs to be guarded in order to maintain productivity.  This isn’t a concept unique to just Peopleware, even The Social Network regularly had scenes where people were “Wired In” and were supposed to be left alone.

Getting into the flow usually takes at least 15 minutes.  If you are programming, this is the time it takes for you to remember what you are doing and then re-create the data structures and algorithms in your head.  You are simultaneously seeing what is and what you are trying to make.  You are not aware of the world around you…until the phone rings.

 “Hello? (I think this algorithm has a memory leak). Yes honey I’ll grab that on the way home. (a pointer to a pointer to a pointer, who wrote this crap). …My…day…is…going well… (Oh man, I think I wrote that code…must have been doing it while talking on the phone)”

It is no wonder programmers seem a little out of it when you call them on the phone or come in unannounced, they are still a little bit hungover from their programming drunkenness.  On the flip side, if they get calls every 15 minutes they never really get in the flow and therefore never really get productive.

What does this have to do with open floor plan offices? When I was working my desk faced the door of a busy hallway.  I dislike working with people walking around behind me (I offer no explanation, it just gives me the creeps) but if I faced the door I would look up every time someone walked by.  It was just enough of a distraction to knock me out of my flow.  I ended up putting a bookcase in front of my desk to block my view from the door, it made me appear anti-social, but I got things done!

My view from my desk.  Awww, the peace provided by wood.

Open floor plans cause employees to lose control of their workspace.  They can no longer filter distractions to protect their flow.  They are bombarded by people walking around, conversations being had, and whatever else.  Collaboration is great, but I never had any problems with it when I had my own office, people just come in and talk; no one else needs to be disturbed.  “Collaborative spaces” have only made me less productive.  Open office plans try to solve a problem that does not exist and make things worse in the process.

A Journeyman or a Master?

I had the opportunity to do my internship at LiquaDry, a fast growing food ingredient company that makes barley grass powder that go into products like Green Magma.  It was a pretty random place to work, but gave me great insights on what it is like to work at a fast growing (non-tech) company.

I finished my assignment faster than expected and the next assignment coming down the pipe ended up being much smaller than anyone anticipated.  With several weeks of the summer still available, I ended up creating a dashboard to track some of their production metrics.  As I worked through it, I tried to recruit anyone I could (to no avail) to hope on and take it over.  “90% of what I am doing, anyone can do.  They just need to have  some interest,” I would say.  “The other 10% I have already finished…and if anything else comes up you can contract me later on.”  They weren’t opposed to having me do future work for them, but I never really did find someone to continue development on what I started.  Even though I am confident that someone else could have done a lot of my work, I never really got to test that belief out.

I had to opportunity to talk with Ralph Yarro at ThinkAtomic this past week as well as Steve (can’t remember his last name) about a lot of the ventures they are working on.  Steve has a software background and we hit it off right away, probably because our different experiences have led use to so many of the same conclusions.  One of the things that ThinkAtomic does is hire kids right out of high school and teaches them to code on the job.  Steve described it as similar to electricians going from journeyman to apprentice to master.  Steve said he believes he could teach up to 10 people and is working to hire more people.  “I just need them to want to program,” he told me.

One of my concerns with this approach was that most people I know who have tried to skip college never really learn the advanced topics and therefore never become masters.  Steve agreed with me, but was able to identify some people that had made the full advancement.  I didn’t realize this at the time, but my old officemate, now Founder of Saltstack, Tom Hatch is probably one of the best examples of people successfully doing this.

While not always the solution for every job, a single master programmer with a teacher’s heart can create a solid development team from people who only have interest.  In the current market of scarce developer talent, maybe it’s time to make your own talent.

How long would this project take?

This is one of the questions I get most often and one of the hardest to answer.  The best way to answer this question is to answer in FTEs (Full-time Employees).  It is fairly easy to do because you just find some features that don’t have a lot of overlap and you assign them to different people.  Once the features have been divided to the smallest groups possible, you could the number of people and that is your answer.  You cannot develop this project any faster than this.

“It takes 9 months to make a baby no matter how many women you assign to the task”

“Men and months are only inter -changeable when a task can be partitioned among workers with no communication between them”

“Adding more engineers to a project makes a late project later” (because people who should be working are now training new people on the project)

– The Mythical Man Month

Unfortunately for me, most people also want to know how many calendar months a project will take as well.  This is tricky because up to 90% of software costs happen after release.  This comes from bug fixes, absent features, performance improvements, supporting new platforms (android can be a big culprit here), or just re-writing some poorly written code to reduce technical debt.  Even before the project is released, defining how to handle every error case gracefully can’t be known upfront.  To a certain extent you will always be asking, “I have a problem I want you to solve but I can’t tell you what the problem is.  How long will it take to solve it?”

Even though providing estimates is hard, developers ultimately need to figure out a way to come up with one.  While there is no exact answer, I have found 3 ways of approximating how long projects will take:

1. The best way to figure out how long something would take is by using a comparable project that the coder has already done.  This is pretty possible for simple web pages or apps.  Or anything that requires standard CRUD (future post to explain that).  Note that this is much more possible when the project is small and simple.  It is also common to be able to use this method on version 1.0 but not on future versions because you are distinguishing yourselves from competitors and therefore writing code that has never before been written.
2. My good friend and past co-worker John Walker (not this John Walker) likes to use this method.  Divide the project up into the smallest tasks possible.  Write down how many hours, days, weeks, or months you think each task will take.  Apply the following conversion: If it takes hours, say it takes a day. If it takes days -> a week.  Weeks -> month.  If it takes a month or more, you have no idea how long it will take or what you need to do.
3. I came up with my own way of doing estimates which ends up being pretty similar to John’s.  Divide each task into the smallest sub-task possible.  Estimate the worst case for how long it will take  each piece to complete.  Add up all of the estimates and multiply it by 4.  Round up to the nearest prime as a sarcastic way of protesting the futility of this exercise.

For large unique projects developers can rarely know how long it will take.  It’s like asking “how long will it take you to find a cure for cancer?”  However, most management teams refuse to accept that answer and so developers do a little dance where they figure out what you want them to say and then try to add a little padding.  What else can they do?   It is common for corners to get cut because people are trying to meet a deadline that should have never been created.  You need to understand that sometimes estimating is hard and allow for changes to the schedule to happen.  Until your coders can break their tasks into pieces smaller than 1 month, do not plan on any marketing events surrounding your release.

The last thing to consider is that when you are creating new features on top of an existing project (ie. version 2.0, 3.0, 4.0…) you should add an additional 20% for rework (coders call it refactoring) of the existing codebase.  This is to pay off technical debt (read the article in the link) or invest in code that will pay off in spades down the road.  People view Google’s 20% time as a way to create new innovation, but I’ll bet most of it is really just paying down technical debt.

Estimating how long something will take is hard and frequently impossible.  Even if you have been able to do it successfully in the past with small projects, it will get more difficult over time.  It is a good idea to make sure your developer is applying some kind of padding to his numbers.  Many younger coders haven’t always figured this out so you may have to just multiply their estimates by 4.

The World is Flat^H^H^H^H Outsourced?

On old Unix systems, backspace would show up as ^H.  I couldn’t immediately find a way to strikethrough my title so I used ^H.

I find a lot of people who want to outsource their development oversees or to another company.  There are clearly a lot of benefits to not having to worry about doing it or getting a better price.  The benefits of outsourcing are readily apparent, so I’m going to focus on some of the times where outsourcing is not a good idea.

Programming is a knowledge acquisition activity, rather than a construction activity.  People normally say “build” or “develop” software which incorrectly associates it with building a house.  Constructive activities are easier to determine what needs to be done, how to fix what is broken, and how much progress has been made.  Writing software is very frequently not this way.

Programming is more closely related to finding a cure for cancer. The primary outcome of research is to learn something, the research paper written at the end does not instantly put that knowledge into anyones head.  No matter how well the research is done, or how well the paper is written, future researchers will not be able to continue where the original researcher has left off.  This is because the next researcher has to spend a lot of time understanding what the paper means.

In programming knowledge is retained in the head of the programmer, the code is an artifact (or by-product) of that learning.  While the artifact can be readily be transferred to another person, the knowledge cannot.  Swapping programmers on a project is like swapping cancer researchers, it can be done, but most of the work is lost.

To put this in perspective for a business person.  Suppose your company had outsourced a very complicated spreadsheet to do some forecasting.  It has 200 tabs full of formulas and cross referencing, but the first tab contains all the results so no one ever really looks at anything else.  Your company saved a bundle and produced something arguably better because it didn’t have the right talent at the time.

Your boss comes to you one day and says, here is this spreadsheet, about 10% of the time cell D46 shows and error instead of the correct answer.  You know how to do spreadsheets, I need you to fix it before our presentation in 2 days.  You should be able to do this easily because all of the work is already done, you just need to fix the bug.

You have created several spreadsheets this complicated before and so you aren’t immediately worried about fixing this one.  Had you been working at the company earlier, you could have written this one. You spend a few hours backtracing the cells to find out how the values are being generated and find that it is really complicated.  You aren’t really sure why a lot of those tabs exist and some of the formulas seem wrong.  You wonder how this spreadsheet is producing any correct answers because it all seems wrong.  However, you do know that the final answers seem to reflect the actual business.  You decide you have three options:

1. Spend a lot of time learning everything about this spreadsheet to fully understand it.
2. Make some guesses about how things work and find a quick fix.
3. Re-write the whole spreadsheet from scratch.

Time is running out and so you choose option 2, an attempt at a quick fix.  After all, this is just one tiny bug and you have a time constraint.  You spend a few more hours on it and change some things so that the error never shows up.

A few days after the presentation your boss comes in with another bug in the spreadsheet.  You go with option 2 again.  You silently wonder to yourself whether your quick fix created the bug.  This process repeats a few more times and your boss starts to express concerns that you are not fixing bugs, you are just replacing one bug for another.  You start to understand why some of those “extra” tabs are there and you realize that you broke the spreadsheet and that those tabs aren’t being used anymore because of your changes.  Uh oh.

You realize option 1 (learn everything about the spreadsheet) is no longer viable because you broke things and don’t know how to put it back.  Before you have a chance to think about re-writing it, you get moved to another project and someone else is now in charge of that mutilated spreadsheet.  Which option do you think he will pick?  I’ve seen people pick all 3 options, so whatever your pick is you have a 67% chance of being wrong 😉

This is a pretty good metaphor for what happens to code that is brought in from the outside.  It’s also a pretty good metaphor for what happens when you move coders to different projects.  After the code has had about 3 authors, the project usually needs to be scrapped and re-written.  When I worked at Novell, my department boss decided it would be a good job to do a reorganization.  Basically all of the coders and all of the projects were shuffled.  The results were catastrophic, but I don’t think my department boss ever realized what he had done or why productivity plummeted.

Outsourcing definitely has its place, I’ve tried to provide some general rules to help you decide if you should outsource your next project.

Do Not Outsource When:

• You are a software company.  Never outsource your core competency.
• You are writing complicated code.  Even changing authors once can be expensive.
• You know you will need a lot of future changes.
• You don’t think you will be able keep outsourcing to the same person.

It May Be Okay To Outsource When:

• You know you will rewrite this code.  For example a rapid prototype.
• The project is small.
• You are able to ensure that no future changes will need to be made.  This means you can validate there are no bugs and you know you will require no new features.
• The project is not central to your core competency.
• Project is mainly displaying information to the user interface, very few algorithms are needed.
• The project is simple.  For example, CRUD applications only Create, Read, Update, and Delete data from a database.  This is built-in functionality for many programming frameworks.  Be careful though, I don’t know how many projects start out as this and then eventually become monster-sized projects.

There are exceptions to all of these rules, but these are definitely good things to consider before picking an option.

Should I Start a Tech Company?

I get asked this from time to time and most of the time I try to discourage non-coders from starting a tech company.  I find it totally okay if at least one of the founders have some coding background, but otherwise it will be like sprinting in the dark–run as fast as you can and just pray there is not a wall in front of you.  Good luck 😉

I have found some articles over the years that articulate some of the reasons why this is tricky, but this one is my favorite.  Here are some of its snippets:

“You need to learn to sing. Because if you don’t, you’re always going to be at the mercy of some a–hole singer.”

“I have this idea for a song. But I’m not musical, so I need to find someone who will write, perform, and record it for me.”

“I have this idea for an app or site. But I’m not technical, so I need to find someone who can make it for me.”

In case you are confused, he is substituting making music for making software.  Why do we think we can outsource making software, would you try to do that with music?  You need to be in control of you company and you can’t do that if you can’t go and fix it yourself when there is no one around.  Read the whole article, it’s short.

David Bateman, founder of Property Solutions came to BYU last year and spoke at an entrepreneurial event.  He told us a story about how his developer became unavailable and he had to go in and figure things out until he could get someone else to do it.  What would he have done if he didn’t have that option?  He went on to advocate founders learning to code, at least enough to be able to keep things going when in a pinch.

Can you start a tech company if you can’t code?  Sure, you can even succeed.  But take a quick look at the biggest software companies, how many of them were founded by someone who couldn’t code at all?

Hello World!

I am currently a BYU MBA student studying entrepreneurship.  In my prior life I wrote software and so the combination of the two sets me up to get asked a lot of questions about tech, engineers, and management.  I figured I might as well write some of these questions down with what I think the right answer is.  More posts to come, and feel free to ask me some questions.

Geek Reference of the Day: The first program you write in most languages prints “Hello World!” to the screen.