I work what I call "afternoon shifts." I come past noon and stay very late into the night. I get tons of work done and I come up with many ideas that are outside of the box of the day-to-day business. My productivity and creativity is not distributed evenly though. Productivity increases continuously during my afternoon shift and peaks before I leave. Creativity has high peak around 8:00 pm and it's very low other times.
The productivity curve is pretty easy to explain. As the day gets later, there are fewer and fewer people every hour. Pressure and distractions disappear. Concentration improves. The dim light helps the concentration too. The creativity peak is caused by dinner break that causes me to stop coding and start thinking about more abstract ideas. Conclusions I reach during this time then feed back into increased productivity past 8:00 pm.
Why don't I get similar creativity peak during lunch break? Firstly, I eat my dinner over the keyboard and during paid working hours, which motivates me to do work-related research during this time. But more importantly, I get plenty of privacy around 8:00 pm. I don't have any privacy around 1:00 pm in my employer's open plan office. Even though it seems surprising at first, privacy appears to be essential for creative work.
The problem with creative work is that it involves tons of stupid ideas. I eventually figure out that the ideas are stupid and I weed them out, but it often turns out that some of the ideas are not that stupid after all. I develop these not-so-stupid ideas until they are strong enough to survive critical evaluation. I can then present them to others. Privacy essentially protects my incubating ideas while they are too vulnerable to critical comments.
Should I have no privacy, these ideas would be shunned and killed shortly after their inception. Open plan office just encourages everyone to look over my shoulder into my notes to get a good laugh. Even if my coworkers are diplomatic enough to avoid such brutal peer pressure, they certainly think "this guy is nuts." It's then hard to sell them the good ideas once they learn to ignore me because of the bad ones.
I might be exaggerating a little, but seriously, try to sit down in an open plan office and do something other than what the boss asked you to do. Are you trying to hide it or not? Feeling a little peer pressure?
Social norms are indeed everywhere in the office. Imagine someone in hurry. Looks funny, doesn't it? But you don't want to look funny, do you? People will subconsciously slow down in order to maintain their dignity. You cannot walk while thinking, because it annoys everyone else. You cannot have a snack to improve concentration. You cannot think with your legs on the table. You cannot make face expressions while thinking.
Generally you cannot think at all in open plan office. You are in some weird conversation mode while others are present. Thinking is not compatible with conversation. You can either focus on others or your thoughts, not both.
Social norms might have greater impact on productivity than the noise and distractions. That's why just putting on the headphones doesn't help. The other problem with headphones is that they are themselves a source of noise.
I see that open plan office solves a problem - the problem of walking through a company full of strangers and wondering what all the others are doing and what are their names anyway. There must be a less destructive solution though. In my case, I spend the first two hours of my day talking to people and working on issues needing immediate attention. I can then spend the rest of the day coding alone.