Michiel . van Vlaardingen

There is excellent information available on testing Flask e.g. http://flask.pocoo.org/docs/0.12/testing/

One thing I ran into was the need to set up a user session for most of my test cases.
The documentation suggests something like:

with app.test_client() as c:
    with c.session_transaction() as sess:
        sess['a_key'] = 'a value'

but repeating that in every test case (especially with a bit more code to set up the session) is not really DRY.

Now the same documentation suggests a very elegant trick using context managers for setting up users on the global object:

from contextlib import contextmanager
from flask import appcontext_pushed, g

@contextmanager
def user_set(app, user):
    def handler(sender, **kwargs):
        g.user = user
    with appcontext_pushed.connected_to(handler, app):
        yield

Inspired by this, I figured that they can be combined into:

@contextmanager
def user_set(app, user):
   with app.test_client() as c:
       with c.session_transaction() as sess:
           start_user_session(sess, user)
       yield c

and then use it in a test like:

user = ...
with user_set(app, user) as client:
    client.get('/logged_in_resource')

Everyone who has ever coded anything significant  (and probably everyone who has ever done anything that requires creativity) will recognise the problem of getting stuck. Most of the time it is caused by lack of information or confidence to move forward. Over time I’ve identified a number of reasons I get stuck and also strategies to deal with this.

Perceived Dependencies

In larger systems, components often have to interact. If none of these components are properly defined or all need changes, it can be hard to get started.
Strategy: Fake a little, build a little.
It’s tempting to start things from the beginning and just write the code sequentially, however if things are not entirely clear it can work very well to start at a very very high level.
I usually just outline everything in 3 to four steps, each being a commented line. For example if I need to classify emails into categories, I might write something like:

// get e-mails
// get defined categories
// classify e-mails

Then write each step as code. This is the time you will need to start thinking about how data is passed around. Whether you will need objects or just lists of strings, etc.

// get e-mails
emails = get_emails()
// get defined categories
categories = get_categories()
// classify e-mails
classifier = new Classifier(categories)
for email in emails:
    scores = classifier.get_score(email)
    category = get_top_category(scores)
    print email.title, category

To test these assumptions you will need to actually implement the functions/components you’ve used. However, key to this strategy is to now fake these functions/components as much as possible.
e.g. I may implement get_top_category as:

def get_top_category(scores):
    return scores.keys()[0]

The reason to do this is not because I don’t know yet how to implement this, but because I don’t want to get stuck in details. (e.g. what happens if there is a tie, what if all are zero, etc etc) Things become so much easier when you have an end-to-end solution that just compiles.

Can’t wrap your head around it

You all know this feeling, but the cause is less clear. Ill defined tasks,  generic approaches,  feeling tired,  they all can lead to this.
Strategy: Second Pair of Eyes
Not everything needs to be solved on your own. If you are not clear on the ‘what’, just talk to a coworker, manager or customer. If you are not clear on the ‘how’ just start explaining to a coworker why you can’t move forward on this.
A very important part of this strategy is the need to explain and define your problem. Even the act of just explaining it may resolve the issue. In that case it’s called Rubber duck debugging: the act of explaining your problem to a rubber duck.

Choice

This is probably the biggest show stopper:  as soon as you are aware that there are multiple options with various tradeoffs, it’s hard to proceed.   I’ve seen bad programmers finish things quicker, just because they were happy with any solution that worked, in their universe there were no trade-offs to be weighed.
Strategy: Writing
If choice and tradeoffs are involved, you need to start writing. Just create a document with bullet lists of:

• Everything that is already given (e.g. API urls, relevant documentation)
• Requirements
• Important conditions (e.g. needs to work when device is offline)
• Potential issues (e.g. there may be too many items to fetch in a single API request)

Then for each of them come up with at least two approaches. Then for each of them also add potential advantages and downsides. And then again for each downside, figure out how to solve it.
For me this usually becomes a nested list of four or five levels.

One of the big advantages is that you can leave this for the next day to refine, without having to start all over again in your mind. Furthermore, you can easily refine this into documentation after the decisions are made.

Boring…

The idea of heaving to spend days to do some boring, semi-repeatable task that could have been avoided when better choices have been made in the past  can certainly demotivate you.
Strategy 1: Embrace it
Sometimes you’ve got plenty of energy, but can’t really move forward on an issue (e.g. due to one of the reasons mentioned above) this is the perfect time to do this boring task you’ve been putting off. It is 100% clear and every minute you spend on it will actually move you forward.
Strategy 2: Make it interesting
Boring things are also often the things that are easy to automate. This might be the right time to learn more about:

• advanced find/replace features in your IDE
• find/grep/sed on the commandline
• UI automation

Strategy 3: Delegate
What is boring to you, may be a nice task for someone else to learn a new skill. e.g. writing UI tests might be boring, but it can be a valuable and marketable skill for your intern.

Bugs

Lots of people get stuck on finding and fixing bugs.  But remember, in debugging there is always a next step to take.
Strategy: Identify and check your assumptions
At the root of every bug is a bad assumption. This can be an assumption on what your code will do, the contents of a variable or the correctness of an external dependency. Debugging is just identifying your assumptions,
I’ll write a follow up post on debugging strategies soon.

As a developer, I don’t particularly enjoy customer support. Questions are either about things that are obvious (or should have been) or about things that I don’t have a clue about either and need a lot of investigation. Some customers are just plain rude with “it’s not working” in the subject and nothing else.

At first sight it seems to be a huge waste of your expensive developer time to spend time on this. However, I strongly believe you should. Not only do developers bring value to the customer by providing more knowledge and understanding about the product. Providing accurate support is a guaranteed way to turn dissatisfied users into your most loyal customers. It also helps improving the product thus reducing support requests and increasing customer satisfaction.

Read all 5 reasons why in the full article

For Observu implementation we are using Pivotal Tracker to keep track of features and progress. I’ve posted a full review on our Observu Blog. To summarize:  the idea of having a single ordered list of features instead of just a bunch with priorities is amazing. It gives you so much more information and forces you to make decisions on what really needs to happen and what needs to happen first.

You’ve got so many ideas in your head that you just don’t know where to get started. As a result, you start procrastinating on stuff you do feel passionate about. At the same time, you’ve got enough energy and are motivated to get stuff done.  It’s a bit contradictory state of mind, that happens to me once in a while.

The problem is anxiety to commit to one project and  even more so: leave all those other things alone.  I believe the key to overcome this problem is to start on anything.  In the end it does matter much more that you’ve made progress and added value than it matters which choice you make.

Of course, that’s easier said than done. ‘Pick anything’  is still a choice and choices are hard. Therefore I use a very simple solution:  pick the easiest task that still adds value.   In the end you probably will have to do all those tasks anyway, so why not start with the easiest one?    Especially  in programming,  getting anything done, will give you more context to solve the next task.  Tasks that may have seemed daunting at first, become easier and better defined when the system around it takes shape.

In programming you often have the choice between  quick and dirty (often copy/paste)  and  smarter and DRY (don’t repeat yourself).  The assumption is that the latter will save you time in maintainance at the cost of some more initial effort.   However, I am convinced that independent of what you choose it will haunt you at some point in time.

It’s not difficult to imagine how copy and paste will haunt you:   things will change… and if it’s one of those things you’ve copy and pasted a million times over the place,  you’re not going to be happy.  Especially if the initial code is not yours, you will have no idea where it has spread and if changing it in one place will fix it everywhere.  (Not to mention the slightly altered cases in some place, which will render a simple search useless)

However, the smarter way can also haunt you. First of all,  ‘smarter’  is also  often harder to understand at first glance.  You need to find out where stuff is defined, as code is less straight forward.  Also, DRY often means generating some output based on a more abstract specification. (either a configuration structure or maybe even a DSL (Domain Specific Language)   That’s all great, until there is a feature that doesn’t fit your framework. (and you WILL encounter this at some point)  You either have to spend a  lot of time to incorporate it cleanly or fix it in an even less elegant way than the one you were trying to avoid.

I’m not going to say you need to stay away from smarter programming, however you must realize that it will not protect you from having to do more work than anticipated for what are essentially minor changes.   Every line of code that you put in, will make it harder for some unknown features to be added later on as you can’t foresee everything.

A few days ago I wrote about procrastination and by coincidence I came across an article about changing your working method to embrace the interruptions, instead of trying to find long stretches of concentration (‘the zone’). It is a nice read, however, more importantly in the comment section I came across something that really appealed to me: The Pomodoro Technique.   I had seen it before, but it didn’t jump out to me at that time.

The basic idea is:  work for 25 minutes on a single task, with a timer ticking back the time left  and then take a break. This is augmented by a few other important steps, such as: noting down your distractions, evaluate them only after the 25 minutes are up and crossing of the completed intervals per task.

As I don’t have a proper timer yet, I looked for a software one, that works under Ubuntu as well  and came across:  Focus Booster.   It makes the ticking sound and also very nicely automatically starts your 5 minute break timer after time is up.

I just started using it, so I’ve only done a few of these ‘Pomodoro’ intervals, but I feel very confident that this will actually work as it makes you very aware of the distractions you need to avoid/delay  and the time you have available.  Starting the timer gives a feel of commitment about doing something NOW instead of  in a few minutes. Furthermore, having a timer tick back, give you back a bit of that deadline stress that makes you feel productive, without the disadvantages of real deadlines. And, not unimportant: it feels good to cross of real productive time.

My whole life I’ve been trained in sticking to deadlines.  School and University is nothing but doing a reasonable amount of work  for an event fixed in time (tests,  reports to hand in, etc)  Which, if you are anything like me,  means  procrastinating until the exact moment it can’t wait any longer an then work very hard.    Procrastination is rewarded:   it gives you more free time  and good results, so by the time  you get to your graduation, it has been perfected to find exactly the minimum amount of time needed to still get good results.

However, this skill gets you nowhere on real projects.  Projects like  your graduation or  developing a new product, where there is no fixed ending and you really want to get done as much as possible, as soon as possible.  ( I can probably make a todo list that fills the rest of my lifetime )  Of course there is the fun/interesting stuff that will never get you stuck, however every project (even the most fun ones)  have those tasks that you just need to get done. Not being able to do so in a timely manner is very frustrating.

The first obvious way to try and fix  this, is to create artificial deadlines, to get back that feel of urgency. However, deadlines that you set yourself don’t work, I’m just way to much aware how arbitrary they are.

Furthermore, on real projects, the amount of work is not known in advance. So there is no way to determine when to start to finish ‘just in time’. On the other hand lots of stuff you delay, might take far less time than you expected.  (You actually spend more time thinking about how much time it is going to take)

Procrastination is also closely related to getting ‘into the zone’ (Read: Joel Spolsky on that topic), but to be fair,  there are also lots and lots of activity that do not require any ‘zone’ at all. So, I feel a strong urge to  ‘solve’ this productivity mystery.   One of the obvious choices is reading about the ‘getting things done’  method. However I’ve never succeeded in making it work for me:  Writing everything down in a system, makes it even more overwhelming then just managing it in my head, as my brains are a lot better capable in hiding the ‘someday/unimportant’ stuff that I come up with.

I’m still in doubt though whether this is not just your brain telling you that there is only a fixed amount of productive time and the idle time in between is just needed to solve complex problems. However, there are a few things I did find, that do help:

• Committing to a task, by telling someone that you will do it: ‘right now’.  (Not some time in the near feature, really: NOW)
• Do another task instead, that you feel really passionate and confident about, to get going
• Split up your tasks into such small subtasks that it’s impossible to not finish it (So every time you do any work, you actually finish something, instead of going from   ‘busy’ to ‘still busy’)
• Work together on a project,  you can keep each other going  ( I believe that is one of the biggest arguments in favor of pair programming)

Are you a procrastinator as well?  And what do you do about it?

My previous post was a bit generic, but now I’ve encountered the perfect example of how building a Framework slows you down:

Lets say we are building basic user account confirmation. In a plain-php way I would write:
mail($email, “Your account confirmation”, “….”);

Now in our framework it has evolved to something like:
mail($email,  $settings->getSetting(CONFIRMATION_MAIL_TITLE), “…”);
and the actual text is loaded from the settings table in the database (so our non-tech partners can edit as well).  So all is now configurable and nice and quick and smart right?

NO, because in a new project, the database is clean, so I get a nice “confirmation.mail.title setting is not defined”.    And now this is where I slow myself down:  Instead of just adding this entry to the database I think:  hey, I shouldn’t get this error, this should have a default setting.   So now instead of having to write this one simple line of code I have to:
– find a way to store all default settings in some file in the framework distribution
– adjust the settings mechanism to use those defaults if there is nothing in the DB
– retrieve all data in the settings table from a previous project and store it in the file

And as soon as I have this thought cross my mind, the focus on the original project is gone and we are back at the infinite task of the perfect framework.

If you develop a lot of projects and are a bit like us, you will recognize this: the reusability dream.  It starts of with a little itch that you are not taking advantage of the knowledge and work in the previous project.  You start with some simple database handling and utility functions and all is fine. For a while…

The next stage is actually creating reusable functionality….   The reason is that you know it will haunt you do the same thing in multiple projects  using copy/paste  as it will need to be fixed some time.

This is also the point where your library turns into a framework: Everything needs to be done in a particular way to fit in your nice reusable scheme of things.  Your project no longer just uses a few nice functions,  it completely depends on it.

As I’ve written before the reason I like plain PHP is what I call the notion of proportional time.  The time you spend is proportional to the complexity of the feature you are building.

The problem with frameworks is that they have smart things built in. The framework makes it easy to do stuff that usually takes much more time. However, it’s almost never smart enough. At this time your framework has evolved so much that every time you will have to pick your poison: either go through a painful adaptation to your framework OR recreate it without your framework and go through the painful tinkering an bugfixing that’s all built into the framework… how frustrating.

In our case this stage expresses itself in SLOW development.  Every small step forward becomes way more complex than the actual feature that you need to built.  It’s completely against the principle of proportional time.  You are locked in your own Framework.

Now is the time that you need to consider whether or not: worse before better applies or that it’s just going downhill.    Two years ago we had a framework where the latter was clearly the case and we just had to drop it.   The problem with it was that it was really an all-or-nothing approach.  In our new framework we can easily take just a step back and use only part of the framework until we have clear ideas about the next level.   We still experience the  locked-in situation from time to time, but it’s much easier to get past it.

Our path to Framework enlightenment has the following steps:

• No collisions: make sure that the library/framework does not collide with other code, this will mean that you can drop it into an existing project without the need to rewrite it first
• Layers: create layers of tools, functions and abstractions. If one layer is to restrictive, you can at least use all tools from the other layers below it
• the framework needs to be out of the way:  You must always be able to do things outside the framework and still access it’s data.
• Everything needs configuration and even more important: defaults
• Override everything: be able to replace small parts of functionality without the need to recreate it completely.
• Constants do not exist:   at some point you will want to be able to manipulate it with code
• Move everything to the framework: if you want to reuse it, don’t create it inside your project directory
• Have a (documented) process (including best practices) to create new things for and using the framework.

We are not there yet, but are confident that this time that the benefits outweigh the extra effort.