Category Archives: Software Development

Agile Power Dynamics

Agile, Analysis, Software Development

In a previous post*: http://nayrb.org/~blog/2015/12/31/agile-basics/

I talked a little bit about Agile and what I thought it meant.

Today, I want to talk a little about how decision-making power is different in an Agile world.

Fundamentally, for me**, ‘Agile’ is all about codifying some of the hard-won lessons from the first 55*** years of computer programming.

There are a number of interlocking ideas in the manifesto, and I keep finding new wrinkles to them when new situations arise, so I’m going to focus on power dynamics.

For this, I will need the strawman of ‘Waterfall’, the ‘development method’ most often contrasted with Agile. Most people represent ‘Waterfall’ as a number of time-sequential silos, with varying amounts of feedback between the silos:

waterfall_1_e96847237d43

waterfall-diagram-in-powerpoint

What is not generally mentioned about ‘Waterfall’ is that estimation is generally performed with minimal input from those implementing whatever solution is called for. This**** generally leads to time and feature estimates being imposed on those implementing the solution.

“You will be solving this problem, and it will take you 3 days to do it.”

It is probably not surprising then that large software projects are prone to cost overruns, as those doing the work are not generally involved in the planning process.

What Agile tries to do is to add and tighten feedback loops. Whereas in a ‘Waterfall’ process, you might release code every quarter, in an agile process, you might do so every two weeks, or multiple times a day.

Key concepts here are subdividing features into smaller and smaller bits, to make the smallest useful change each time, but do it very often.

Getting back to decision-making power, what Agile does is to separate Prioritization power from Estimation power.

Agile gives the Estimation power to the team that will be doing to work*****, while retaining prioritization power in Management. You say that it separates the ‘How’ from the ‘What’******.

This separation encourages more buy-in from staff, and can stop bad ideas earlier. Most importantly, it focuses each of staff and management on what they (theoretically) do best. Management is hired to make decisions about what should be done next, developers are hired because they know how to do things well[7*].

Next time, we’ll tease apart more of the manifesto.

*Same disclaimer applies: I currently work for a company. That company does Agile. From my limited experience, I think it does it well. I am not talking about that company in any way, shape, or form in this post.

**And many others

***I’m going from ENIAC, the first actual construction of a general purpose computer. Lest you think I’m forgetting about Ada Lovelace, you may want to read about the first six (all female) programmers of ENIAC: http://eniacprogrammers.org/

****There are also often issues of under-estimating the cost of things in order to obtain a contract/approvals, but this is outside the scope of this post. Suffice it to say that one who was singularly focused on landing a contract would have great incentives to ignore feasibility and estimation objections from those who might be implementing it.

*****Agile also tries to make sure that the team has someone skilled in every relevant part of the company, but that is outside the scope.

******The ‘Why’ is outside scope of this article. 😀

*******You are confident that your hiring process is bringing in good developers, aren’t you?

Agile Basics

Agile, Idle Speculation, Software Development

Disclaimer: I currently work for a company. That company does Agile. From my limited experience, I think it does it well. I am not talking about that company in any way, shape, or form in this post.

I don’t recall when I first heard about Agile software development. I probably heard about it from Slashdot, when I was still reading it during grad. school.

First up, the Manifesto itself:

From: http://agilemanifesto.org/

”
We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more.
”

Overall, it feels like a very human approach to software development. I’ve always enjoyed speculating about the situations which may have led to rules being formed*.

It also feels like it was written by a group of people who were actually interested in solving problems, perhaps by cutting through Gordian Knots of rigid process and planning.

Their conclusion was that problems will occur, things will change, and you want your process to accommodate that as much as possible, to enable and encourage people to talk about these earlier and more effectively. More of a ‘getting to yes’, rather than rear-covering or posturing.

Now, they further broke down the above four statements into 12 principles:

http://agilemanifesto.org/principles.html

1) “Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.”

This one seems pretty self-evident, but there are a surprising number of people who have job descriptions at odds or orthogonal to this, especially as organizations get larger.

2) “Welcome changing requirements, even late in
development. Agile processes harness change for
the customer’s competitive advantage.”

This might be the toughest (it was for me, and I consider myself good at this), as humans are naturally lazy and resistant to change.

Evolutionary Psychology : Laziness

I wonder if this can only be overcome through experience, by knowing how much more pain will happen if a particular shortcut is taken or some important stakeholder is ignored. (Even knowing that someone should be listened to instead of letting your lazy brain edit them out can be difficult.) Either way, a good argument for having at least one person (that you listen to!) with experience on the team.

But if you’re having requirements changing too often, your stories are likely not ready before you start, or they are too large, leading you to:

3) “Deliver working software frequently, from a
couple of weeks to a couple of months, with a
preference to the shorter timescale.”

‘Fail early, fail often’ goes the quote. Coupled with 2), it becomes much more difficult to be working on the wrong thing when you’re allowing for updated requirements every time you start a new short story**.

4) “Business people and developers must work
together daily throughout the project.”

I feel like the people who put this list together had experienced a lot of communication problems where they worked. This one is really helpful. There are few things more annoying than being ready to work on something but not being able to reach the person who needs to make the decision.

5) “Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.”

From the people I’ve talked to, this is every programmer’s dream. All the best working environments I’ve been in have been like this.

6) “The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.”

Yes. And it drops off considerably, even to ‘face-to-face’ Skype/

7) “Working software is the primary measure of progress.”

Many people have different measures of progress, leading to organizational misalignment.

8) “Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.”

As much fun as ‘feast and famine’ is, you’re probably not doing your best work souped up on adrenaline and coffee. (Or maybe you are. If so, you should take some time off between binges.)

9) “Continuous attention to technical excellence
and good design enhances agility.”

‘We put brakes on the car so that it can go faster.’ ‘Legacy code is defined as any code with inadequate test coverage.’ All the time you spend hesitating*** because you’re worried about breaking old crappy code is time you’re not building features or refactoring old crappy code.

10) “Simplicity–the art of maximizing the amount
of work not done–is essential.”

Think Apple. Think Oblivion when they decided to voice all of the dialog, and all the streamlining that inspired/required. Think about that super-expert programmer you know who can tell you why every single line of code they wrote is there, and also why everything not there is not there.

11) “The best architectures, requirements, and designs
emerge from self-organizing teams.”

Ask the people who know the most about something to make the decisions****.

12) “At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts
its behavior accordingly. ”

Continuous improvement. Read ‘The Goal*****’. Really. It will improve your life. Also retrospectives.

*Similar to reading Wikipedia and trying to figure out the actual events behind the dry descriptions…

**I still remember the day when I felt I had ‘graduated to short stories’, as they tend to pack more large ideas per page. Longer novels tend to be more meditative/escapatory?

***And working around crappy code…

****I’m currently most comfortable with some mix of Scrum and Kanban. They have a specific separation of powers between the team (handles estimation) and the product owner (handles prioritization). To me, this seems totally reasonable, but re-reading the manifesto above, there are many levels of Agile actualization above that. (Think Valve.) (Also, the product owner is part of the team, just with a specific role to play in addition to the general team tasks.)

*****https://en.wikipedia.org/wiki/The_Goal_%28novel%29 My favourite business book. Told in a novel format. Some of the story has dated references (before most of feminism), but is still very useful.

Automation and Machine Learning

Idle Speculation, Software Development, Thoughts on Thoughts

When we ask a computer for help with a task, what are we asking for?

1) Help with automating a repetitive task
2) Help with a decision

1) Help with automating a repetitive task
There are various ways you can automate a repetitive task. You can:
a) Ask your computer to do the same thing again and again, regardless of input (display the home page)
b) Give it some simple rules to follow (if they try to navigate to a non-existent page, show them a 404)
c) Give it some complex or not fully understood rules to follow (based on our tests, these are the solutions you should attempt, in this order)
d) Give it inputs, and have it adapt (‘Watch me perform this industrial assembly task, now you do it’)

2) Help with a decision
There are various different ways you can use a computer to help with a decision. You can:
a) Display data in various interesting ways (Data Visualization)
b) Give it the data and some rules to follow (Standard decision automation)
c) Give it the data and a desired output/scoring function (Supervised/Reinforcement Learning)
d) Give it the data and nothing else* (Unsupervised Learning)

This is somewhat of a false dichotomy, as adding new types of decisions allows more and more automation.

– Search (inputting words, pictures, video into a search engine and asking for a result) generally started with 2.a) (Data Display), and seems to be trying to move up the decision hierarchy, anticipating questions and the rules the user would want it to follow. This seems to be generally done with statistics, but I expect this would be switching over to pattern-finding neural nets
– Clustering (throwing a bunch of data into the hopper and getting groupings back) is also mostly in the Data Visualization bucket. It could also be an input into a machine learning algorithm, which would then be trained to make decisions based on these clusters
– Machine Learning (giving a bunch of data and getting a decision or pattern out) can be used for most or all of the options above, and similar to how computers have gotten ‘fast enough’, Machine Learning is becoming ‘good enough’ or ‘easy enough’ to replace many of the above.

So, as a human, when do you choose each of these? Assuming the options get more difficult going down the list, you would:

1) Start by googling various things (mostly to see what has been done before***).
2) You could then look at the data, clean it, and try clustering it into groups, to see if any of them made sense for the decision you wanted to make.
3) If neither of these worked, or if you wanted more, you could derive a scoring function for the output you wanted, then supply a Machine Learning algorithm with a substantial amount of data, and see how optimal it could make the decision.
4) If you don’t even know what decision you want, or are having difficulty making a scoring function, you could throw the data into an unsupervised learning hopper and see what comes out.

At each of the steps above, you can hive off parts and automate them, either using rules derived from the patterns you’ve found, or using flexible rules from the Machine Learning algorithm. You may find you can accomplish most of your task without having to resort to complex or incompletely understood algorithms.

More examples in subsequent posts. Stay tuned. As you can tell, the categories above have not fully crystallized.

*Unsupervised Learning has a number of levels** in it, such as ‘Find Features’, ‘What is the Question?’, ‘Why?’, etc…

**Not that everything is hierarchical, but this is convenient for discussion

***This is the ‘literature review’ portion of anything we do now

Regulatory Capture and NP-Completeness

Economics, Politics, Software Development

A common conversation in our household:

S: “So, why do you think happens?”
Me: “Regulatory Capture.”
S: “Oh. Right. :(”

I like to sort this with my general wont to cut Gordian knots*, but perhaps it is also useful because it allows you to reduce the problem under discussion to a well-known problem, which is known to be insoluble in very specific ways.

Perhaps we will find that the solution to the P=?NP problem is the same as the problem of regulatory capture**. I think it’s more likely that P=?NP will be solved first.

*Or to be a troll.

**Regulatory capture I think was first described for me in ‘Yes, Prime Minister’, as the inevitable co-option of the body which regulates an industry by the industry which it regulates. This is generally because they talk to each other the most, combined with the huge financial incentives.

“So, what was the issue?”

Engineering, Software Development, Thoughts on Thoughts

So, we were debugging a common ground issue today at a in the amazing Helios Makerspace in Montreal.

We got as far as we could ourselves (debugging one common ground problem, and tracking the data signal through one board and on to the second), then we got stuck. We tried a bunch of things, but it was only when the came in, and determine it was two problems: A second common ground problem, combined with the first (Arduino) board having too many outputs and so not outputting a high enough voltage.

The really interesting part (aside from learning again how important common grounds are) was watching the engineers in the room (the people who were building things, perhaps or perhaps not engineers) all run over as soon as the problem was solved and someone asked the question “So, what was the issue?”

Anytime someone is agonizing over a problem for hours, there is bound to be some learning for those around…Thinking about this from a min/maxing perspective, someone spends hours solving the problem, then you spend 2mins learning about the solution, and then you add it to your list of things to try/check when debugging, taking maybe 30s to possibly reduce your own debugging time by hours.