Code toxicity

So finally I found a piece work that is in line with what I was thinking in terms of code entropy. What I like about it is that it's actually quite comprehensive, as it takes into account several factors, such as function length, cyclomatic complexity, nested-if depth, etc.

Well without further ado, here's the link:

Doesn't look like I'm gonna be taking the world by surprise by making a brilliant academic thesis on the matter.

Stages of development for code written from scratch

I'm going to try to reverse engineer the process I go through during software development. At the end of the day, this is some sort of optmization with respect to "maintainability" and "clarity". I'm haven't quite nailed down the formal definitions of these, but I'm going to try at a later stage.


So here's the process I go through if I start working on code from scratch:

1. Rapid prototyping. Use structs or classes with all public data members and functions. At this point we don't know what the structure of the problem is, so there's no point in "locking down" classes or hiding members. When the structure of the problem is not clear, "boundaries" between classes are arbitrary. Which data and what functions belong together will become clear later.
2. Exploration. Getting some results the quickest way, so as to get some iterations, user feedback, etc. 
3. Paradigm adjustment - The problem space is clearer. Classes actually correspond to concepts in the real world altho the data they operate on might be elsewhere. Even though the problem space is clear, the vocabulary used in the prototype no longer fits. 
4. Consolidation. It is now clear what problem we're trying to solve and how we're trying to solve it. As concepts are clearer so are the class boundaries. Most operations should now be on members of the same class. Most data members should be private, as they're no longer needed from the outside. Implementation details are now hidden and client code only needs to know a small fraction of it, the "interface", the API , via which the user can interact.

I believe in the end this process is about reducing code entropy, which I'm trying to define. I will ultimately need to:

• Define "code entropy" more formally
• Refine the process described in this post
• Show that the process described in this post leads to code with lower entropy

Revant previous posts:

Code entropy

Why manual linked lists are evil

Recently I've been reminded of a pattern in code I particularly dislike. Manual traversal of linked lists. I couldn't quite put my finger on why I have an aversion to this practice, but I'm gonna try to pinpoint right here in this post, and solve this moral dilemma once and for all.

The pattern looks something like this (here i'm not talking about design patterns, just a pattern of usage).

//declaration code
struct SPointsList
{
    SPoint m_CurPoint;
    SPointsList* m_Next;
}


void foo(SPoint& pt);


//... 
//client code
SPointsList points; 


//.. some annoying initialization code goes here, creating a non-empty valid list
SPointsList* pt = points.m_CurPoint;
while (pt!=NULL)
{
    foo(pt->m_CurPoint);
    pt = pt->m_Next;
}

So what don't you like about this code, Mr. Refactoraholic? It's efficient, it's compact, it's not using templates, it's doing just what it's supposed to do and no more. Doesn't that fit into your favorite K.I.S.S. principle? Aren't you just being a tight-ass abstractionist who is just waiting for an excuse to use some templates and iterators that you read about in your favorite design pattern book?


Well, I'm glad you asked. Here's some of the things I don't like about this code:

• It reinvents the wheel. Implementing a linked list from scratch is an exercise that I've learned how to do in high school, and yet in practice try to avoid as much as possible for reasons below
• It does pointer arithmetics, which is an overused feature of C / C++, most other languages have wisely chosen to get away from. 
• It is prone to off-by-one errors, since you have to be careful not to hit uninitialized memory or to get yourself into an infinite loop. 
• Client code also has to be careful to construct a valid data structure (the last element needs to point to a NULL, so that elsewhere in client code we could check for it). I didn't include an example of this, simply to keep my blood pressure low and my stomach from getting too upset. 
• All in all, the main negative thing about using manual linked lists, is that this approach exposes implementation details to the client, and client code everywhere has to adjust to that. If the representation changes, client code has to be re-written, and that's a real waste, as this code could have been written in a more flexible way from the start. 

One way of writing this code would be:

typedef std::list < SPoint > SPointsList

//client code:

SPointsList points; 

//some initialization code goes here, using list::insert, or list::push_back 

SPointsList::iterator itr = points.begin();

for (;itr!=points.end(); ++itr) {

    foo(*itr);

}

We can argue about how pretty this code is or that it uses more characters than the original pointer version. But in the end, this approach hides all the nasty details and pointer arithmetics, and changing the representation does not affect the client code at all.

There's also a more neutral approach that avoids using STLs, where we basically only implement the functions that are needed, but still use the iterator-based approach to hide the details of the pointer math. Here's a very sloppy version of that:

struct SPointsList

{

    SPoint m_CurPoint;

    SPointsList* m_Next;

    void AddNewPoint(const SPoint&, SPointsList* pos);

    void RemovePoint(SPointsList* pos);

    SPointsList* GetNext() {return m_Next;} 

    bool IsLast(SPointsList* pt) {return pt==NULL;}

}

Perhaps only implementing the functions you need can reduce code bloat and allow you to special-case optimize access to your data structure, but you have to do more work to reinvent the wheel, and error-proof your internal pointer math for adding / removing elements. 


[It seems like blogger is not the most code-snippet friendly of environments, so I'm gonna look into switching to wordpress or something else]

K.I.S.S. part II

This post is about keeping it simple on the solution side of things:

Now it's easy for me to say "don't make shit complicated". I mean some shit is complicated by nature, and the only thing that's left to do is get up to speed on the details until it becomes clear. In addition, due to the exploratory nature of software development, often shit only becomes simple after you've done it the complicated way. Our understanding of the problem may grow as time goes on, and based on the improved understanding, the code may start looking clearer, and typically more compact.

I think it was Mel Gibson who once said about acting "it takes a lot of work to make it look effortless". As a refactoraholic, I can testify to the fact that "it takes a lot of work to make code look easy". A person looking at simply written code might be tempted to say "oh yeah, I could have thought of it myself". Just like a good textbook that does a good job of explaining something, might get you quicker to the eureka moment, so will compact, loosely coupled code, with good comments and variable names that make sense get you to find logical errors and bugs, as well as make adding new features faster.

After refactoring I often stumble on some "obvious" logical errors in the code. It can look like a dumb mistake in the logic, but it only became obvious after 2-3 rounds of refactoring. Previously this mistake may have been burried so deep, it would have been much more difficult to "see".

K.I.S.S.

As I look at the experiences I've had handling difficult code written by other people, some patterns seem to emerge. One of the most persistent patterns that seem to be causing most of the software maintenance issues I've encountered is people making shit too fucking complicated! And by that I mean people not following the principle "Keep it simple, stupid" (aka, K.I.S.S.).

There is one way of making shit complicated that I am attacking and that is taking care of unnecessary use cases. In my opinion, the code that is most difficult to maintain is code, which has a lot of what-ifs in it, a lot of just in case handling. This coding strategy is the root of all evil, and together with premature optimization forms the two black pillars of death. So here I'm going to try to make a distinction between:
 (1) unnecessary code to solve a necessary problem, and
 (2) code to solve an unnecessary problem (the code in this case is also, by definition unnecessary).

The first case is about proposing a solution that's unnecessarily difficult, the second case is about proposing a solution that's not necessary at all. The first case is about overcomplicating the solution space, while the second is about overcomplicating the problem space.

What if the hero in our game is hanging down from a helicopter, and a bullet enters his left eye? Should we simulate the eye movement on the right eye as the bullet gets closer? What if one of his eyes gets streamed out in the process? What if there's an eye-patch on the left eye? Should we simulate it with cloth physics?

OK OK, stop right there. How about let's consider if this combination of circumstances needs to be handled at all?!

Life of a coder is difficult enough as it is, we shouldn't have to solve imaginary, or potential problems before solving real ones. One of the most obvious signs that code is solving an imaginary problem, is large amounts of boiler-plate code, a lot of methods and files that seem to be doing very little, as if the original author of the code had something grander in mind, and put in some code just in case. What's makes matters worse is that the code for solving a non-existing problem is living right alongside the code that solves an actual problem. Then, the reader of the code in question has to not only understand how this code solves an actual problem, but also what other than the actual problem is this code trying to solve?

Continued in this post: http://refactoraholic.blogspot.com/2011/04/kiss-part-ii.html

How to refactor huge chunks of unfamiliar code

I've had the pleasure (or the misfortune) of refactoring some large chunks of code, with the following characteristics:

• It has at least one very large function (300-900 lines)
• It has no clear owner, as at least 5 people have been authors on different parts of the code, and 2-3 of them already left the company
• This code has gained the reputation, that nobody knows what it's really doing
• There are plans to do some cleanup, but nobody knows when or who is gonna do it, and nobody dares to take the responsibility. This project is always on people's radar, but never quite higher on the priority list than everything else they gotta do
• Besides, respectable programmers got better things to do than to refactor messy obscure code, right? Time to call in the Refactorinator. MWAHAHAHAHA

Here's the process backwards engineered from how I typically do it

• Identify the huge functions
• Identify major logic blocks within those functions
• Typically these will be outer while or for loops (inner loops might just be too hard to tackle right away, if they have dependencies on variables computed in the outer loops)
• Factor the identified chunks of code out into their own function

• Visual Studio has "Extract Method", which does an OK job of automating this process
• Typically there will be too many parameters in such a function, consider if you can 
• Make some of the parameters members of the class
• Recompute the same values locally inside the new function (some of these computations will be cheap, like an indexed array lookup, and it pays off to reduce the size of signature for the new function).
• Make it clear what the input and the output of each function is. 
• What values are being consumed, what values are being modified?
• const and static functions are preferrable, as they don't have side effects
• If it's not possible to make the function const or static, try to see if you can make some of the parameters const. That is usually possible, especially as the newly created function is smaller in scope and will modify less of the data
• Factor out a few functions this way if possible
• Are there any patterns emerging?
• Are there certain groups of parameters that are getting passed around a lot?
• If so, form a struct of these common parameters, which should further simplify the function signatures for the new functions
• Are there common operations on these groups of parameters?
• In this case, we might start adding functions to the struct
• Is this group of paramters sufficiently isolated that they are no longer needed for individual access from other datastructures?
• Then we have the birth of a new class.
• Resolve overlapping or duplicate data
• This is the most annoying part of the process. It very often happens that the datastructures floating around in the super-large function / class are similar (but not quite the same) or contain duplicate elements. 
• For example there can be two structures passed around, which are partially different, and partially they represent the same data, and then that data is copied back and forth to keep in sync.
• Try and see if you can disentangle this knot by using the member of only one of these overlapping structs / classes. That way you can avoid 3 bad things:
• needing to keep them in sync (simplifying logic), 
• copying them back and forth (saving CPU cycles and reducing code size)
• stroring of duplicate data (saving memory)
• This is often far from trivial, and might require some domain knowledge, or actually sitting down and understanding the code
• Hopefully after the massive code-shoveling, your understanding is much improved, or it could be time to check with one of the owners
• Be prepared however, that the (partial/ former) owner is also confused, doesn't fully understand the system, or doesn't recognize the code after you've messed with it
• Regardless, there could be some useful information that pops up as a result of this communication
• I will probably elaborate on this last step in another post (this one was probably a handful already). I don't think I have the process of resolving overlapping data nailed just yet, but I believe with more experience, I will arrive at a more solid solution.

F# for beginners?

I think not:

"The mapfirst function takes a function as the first argument and applies it to the first element of a tuple that's passed as the second argument"

Try explaining that to someone new to programming...