An article on writing templated WPF custom controls

This is something that we had to discover with reflector when I wrote the Excel-like control we used for the National Express WPF application, so it's good to see people writing about it, especially when it's Charles Petzold in msdn magazine.

An additional thing you might want to do is have properties defining the templates for different sub-elements used by defining a dependency property of type DataTemplate. You may also want to create new sub-controls, e.g. a CellPresenter, that you can add to the template to define where in the template other components of your control will get injected.

That should probably be an article in its own right. Maybe this weekend if I recover from my sore throat.

Merry Christmas

While some would consider Christmas as a catholic feast, I'll consider myself the Santa Claus, Christmas tree, tinsels, balls, presents and big dinners as having no fundamental Christian ties. It's a pagan feast and as such, merry Christmas to all men on earth, whatever their religion.

XamlPadX V3.0

Lester just announced (well, on Wednesday but I have a big backlog because I spend my time reading the alt.net chat) XamlPadX 3.0. I know we didn't use the tool on my previous WPF projects, but I think it's the only XAML quick and easy tool that has seen an update recently. Will need to have a play during the festive season!

As a follow-up on the mocking issue

Over there on CodeBetter.com, David Hayden has posted something about Model-View-Presenter that is a typical example of the confusion I was talking about in my post on why mock frameworks suck.

The interface wasn't mocked, what David did was to create a test double (a fake or a stub depending on how you look at it, but not a mock!) using a mock framework. Again, I don't object on the use of the mock framework (see my previous entry as to where my objection is), but I think we're muddying the water for everybody if we don't get the semantics correct.

Or am I really the only one thinking you should use the right tool for the right job?

Technorati Tags: UnitTesting,Mocks,RhinoMock,Model-View-Presenter

Why Mock frameworks rock

Ayende just commented on my post about writing delegate-based test doubles. He  rightfully highlights that through the method I've described, it becomes clunky to write interaction testing. I'll quote my previous post.

To sum it up, mocks are used when you want to test how your object interacts with another object, that's interaction testing, whereas all the others are used when you want to actually test your object's functionality

Let's forget the discussion about the value of interaction testing and mocks in general for a second. If you do decide to test how your object interacts with one of its dependencies, you're writing a mock and then using a mock framework makes sense. The API used will let you test the interaction conditions much faster than writing code. For example, making sure Initialize() is called only one, you'd have to write a fake using the following.

[Test]

public void InitializeIsCalledOnce()

{

    FakeConfigurationProvider provider = new FakeConfigurationProvider();

    int callCount = 0;

    provider.Initialize = () => { callCount++; };

 

    ClassThatDoesSomething obj = new ClassThatDoesSomething(provider);

    Assert.AreEqual(1, callCount);

}

As you can see, it's entirely doable, but it's starting to get a bit clunky.

The only reason why you wouldn't want to use a mock framework to do such interaction testing is when your dependency is provided by someone that provided a test harness.

Maybe that's where one day we'll be, with developers of a component providing you with a pre-written fake that, based on configuration, will throw at you any possible error combination, and check in which order and how often you called methods for each use-case you have. We're not there yet (or at all) but providing a component and a test harness for the users of that component seems to me a more long-term solution.

There is the question of the intrinsic value of interaction-based testing. In some cases, it is a requirement, in others it's not. Let's say my component has an Initialize() method. What is its behavior when you call it twice? It could be that the object is reinitialized transparently (if it has no side efects), or it could be that the object throws when initialization has already been done. In the second case you don't need a mock. In the first one your tests will catch any introduced error for any non-trivial task (lost a transaction, file handle gone). You could check that an Insert method only calls the repository class twice. But if you test a select after an insert and you get back two objects, you have a failing test.

The interesting question is, what if you don't detect an error, and the Insert method is idempotent. I'd have a tendency to think that those conditions, where you use an object in the wrong way and obtain the right results are probably of lower priorities than other tests. I'd never suggest you shouldn't test that, I'd suggest that writing those tests have probably lower priority than tests in the rest of the system.

Finally, in my last post I highlight the fact that writing your double using delegates, you can provide a default implementation that reacts the way you want, and you can replace the call with specific code on a per-test basis. I'd argue that in those conditions Mock frameworks will have a harder time providing the same level of features. Do you really want to redefine your mock every time? Do you really just want your mock to throw or return a simple value?

The point of my previous post was to suggest that Mock frameworks are over-used and not leveraged for what they are really good at. And they really shine when the objects you depend on don't have a behaviour that lets you receive errors when the caller is not respecting the documentation: done in the right order / too often, etc.

Mocks are a great tool, but shouldn't be used all the time as a general solution to writing tests. You have to pick the right tool for the right job.

Why Mock frameworks suck, and how to write delegate-based test doubles

There is a big rift in test-oriented developers about when or how to use mocks, what the difference between a mock, a stub, a fake and a dummy is, and what mock frameworks bring to the game. I won't bore everyone with my explanation of the differences, Martin Fowler is well known for formulating patterns better than anyone else, so I'll point you to his Mocks aren't stubs article. I'll also point to Roy Osherove that has a much more simplified entry about the topic titled Mocks and stubs - The difference is in the flow of information.

To sum it up, mocks are used when you want to test how your object interacts with another object, that's interaction testing, whereas all the others are used when you want to actually test your object's functionality, which I'll consider to be unit testing. Interaction testing depends on more than one area of expertise as it involves your mock being called, and because I don't consider the mock part of the unit, I'd argue that you're not testing a unit.

In the case where you write the code from the ground up, I strongly believe there is no real reason to use mocks, as you can bake dependency injection, interfaces and fakes in your API from the ground-up. When trying to put old code under test however, mocks are a valid approach.

The gray area is when you want to replace one of your dependencies and use a mock framework to implement a fake. Let's take an example that's quite common, abstracting configuration information. Your first step would be to implement an interface that wraps the initialisation and the property you want to retrieve, which we'll call IConfigurationProvider.

One way I often see being employed is using a Mock framework to do the heavy lifting of implementing the code. As such you'll often see code that looks like the following.

IConfigurationProvider provider = Mocks.DynamicMock<IConfigurationProvider>();

Expect.Call(provider.ContentLocation).PropertyBehavior();

Expect.Call(provider.Initialize()).IgnoreArguments().Throw(new ConfigurationException());

My main issue with code like this is the use of a domain language to write code. You see, what you're doing is providing an implementation of an interface, but rather than write the code, we use a mock to write... Well, code. Let's see what it takes to write a fake implementation.

public class FakeConfigurationProvider : IConfigurationProvider

{

    public string  ContentLocation

    {

        get { return null; }

    }

    public bool  Initialize()

    {

        throw new ConfigurationException();

    }

}

Obviously that's much more code. Or is it? Let's see a character count:

• Mock code
  • Characters (no space): 220
  • Characters (with spaces): 260
• Fake code
  • Characters (no spaces): 163
  • Characters (with spaces): 220

As you can see, what seems at first to be a more compact syntax is actually more verbose. Furthermore, you use an API to write code. I'd say that writing a fake using a mock framework is as useful as writing your unit tests by using reflection.

PS: I'm no Rhino expert and I expect someone to give me a more compact syntax if there's one so I can update this code.

Some people will argue that you can use the Stub<T> method with RhinoMocks to achieve the same with less code. I'd reply that creating my FakeConfigurationProvider once will ensure the same logic implementation is used in every single test that leverage the fake. To which mock fanatics will reply that you will want to change the implementation of one method for a specific test, and my way of writing fakes will create a new class for each test.

So here's my response, and the trick I use very often when writing my tests. We'll use a .net feature that doesn't get used very often, explicit interface implementation, coupled with delegates. For each of the methods in my class, I'll declare a delegate, a public property of the delegate type, together with a value (the default implementation), and finally I'll explicitly implement the interface and call the delegate property. Code speaks better than prose, so here it is.

public class FakeConfigurationProvider : IConfigurationProvider

{

    // Delegate types

    public delegate void InitializeDelegate();

    public delegate string get_ContentLocationDelegate();

    // Defining fields for our delegates

    public InitializeDelegate Initialize = delegate { return; };

    public get_ContentLocationDelegate get_ContentLocation = delegate { return null; };

    // Implementing our interface

    bool IConfigurationProvider.Initialize()

    {

        Initialize(); // calls the delegate!

    }

    string IConfigurationProvider.ContentLocation

    {

        get { return get_ContentLocation(); }

    }

}

Note that by default I don't do anything on Initialize. We'll inject the exception throwing later.

This fake is made up of more code, but the flexibility and clarity gains in your tests is enormous. To proove it, let's create a quick object that uses our configuration provider.

public class ClassThatDoesSomething

{

    public string ContentLocation { get; private set; }

 

    public ClassThatDoesSomething(IConfigurationProvider provider)

    {

        try

        {

            provider.Initialize();

            ContentLocation = provider.ContentLocation;

        }

        catch (ConfigurationException) { }

    }

}

And now let's write a test for the default case where everything goes according to plans in our Initialize method and nothing throws.

[Test]

public void CreatingTheClassSucceeds()

{

    FakeConfigurationProvider provider = new FakeConfigurationProvider();

    ClassThatDoesSomething obj = new ClassThatDoesSomething(provider);

    Assert.IsNotNull(obj); // will never be null anyway, keeping the Assert for more obvious expressiveness

}

As you can see if you run this test, the default implementation of my fake for that method doesn't throw. Now let's see how we can change this to throw exceptions. I've used C# 2.0 delegate notation, with the C# 3.0 notation commented out.

[Test, ExpectedException(typeof(ArgumentException))]

public void CreatingTheClassFailsWhenExceptionIsThrown()

{

    FakeConfigurationProvider provider = new FakeConfigurationProvider();

    provider.Initialize = delegate { throw new ArgumentException(); };

    // C# 3.0 Syntax is shorter:

    // provider.Initialize = () => { throw new ArgumentException(); };

 

    ClassThatDoesSomething obj = new ClassThatDoesSomething(provider);

    Assert.IsNotNull(obj); // will never be null as if something throws we won't reach the assert

}

It only took me one line of code to redefine the implementation of my method. Best of all, I use C# to implement my code rather than use the RhinoMock API. Better yet, I can reuse my default fake implementation to be a fully functional in-memory object that reacts as expected, and inject exceptions in any of the methods of my fake object, which is much more reusable. Even better yet, I'm still enjoying compile-time checks that are not applied with any mock framework.

As someone once said, when I see Mocks being used, I reach for my revolver. You've learned already that the test is written in C#, you know your documentation is (primarily) your tests, now learn that code that replaces code is better done in code!

IE8 is coming?

Off to bed. But before going, a question... Are we going to see IE8 being announced soon?

While on the subject of MacBook Pro

If your DVI output resolution is limited, Apple has finally released a new nvidia driver. You can find it here:

Boot Camp 2.0: Video issues with Windows Vista or XP

How to wipe Mac OS out of your MacBook

What an eventful week. As I sit here waiting for countless installs on my VM to get MCMS up and running, I thought I'd take the time to blog about my beloved MacBook Pro.

As some of you already know, I tried really hard when I got my second MBP to like MacOS. I ran exclusively on it for a while, and was running everything under Parallel, and then on Fusion.

The combination of lack of Aero, my mostly windows exclusive world, and with all its bad things the comparatively fantastic outlook (albeit I didn't try entourage), I switched back to full time bootcamp on vista.

After a while though, I realized that no amount of uninstall would reduce my Mac OS partition size, and keeping gigs and gigs of used space around just for the occasional firmware update got me thinking, why am I keeping Mac OS around at all? And as I installed on my main OS way too many beta and alpha of different products, it was time for a big cleanup.

So here's the steps I followed to be running a Vista only MacBook Pro.

1. Install Bootcamp 2.0 from the Leopard DVD. Follow the steps to burn the drivers CD and let it repartition your drive.
2. Reboot with the Vista install CD.
3. Don't install anything, go to the recovery console (on the first screen, instead of Install now use the repair bit.
4. When you're in the console type diskpart to start the disk management utility.
5. Wipe the whole drive completely, removing every volume and partition (yes that includes the EFI partition, but so far I've not seen any benefits in keeping it around, considering the very poor EFI use being done on the Mac).
   You do this by either select drive 0 and clear, or select partition n and delete override until none is left.
6. Once all the partitions are removed, convert your drive to MBR by doing a select drive 0 and convert mbr
7. Restart the machine and go through the installation screen. When asked to select your drive, do not create anything on it, just select the unreserved space and let Vista do the rest.

Bootcamp is actually several components:

• An update to provide a legacy BIOS for non EFI-aware OSs.
• A partition manager with dynamic resizing of HFS+ partitions and NTFS / FAT partition creation capability, with a dual GPT / MBR partition synchronization.
• An install applet that runs the previous two
• A set of drivers packaged by Apple, and containing Apple and other vendros drivers in one big install.

Now I think there's one point that's not been explained much. The reason why you convert your disk to MBR in my earlier point is because GPT is not supported as the boot partition for windows. It's a bit of a pain point to figure it out, and when you switch to MBR you won't be able to install Mac OS on that drive again. The way to make both coexist is to provide both a GPT and an MBR on the drive. The mac partitioning utility supports that, and if you run another tool like Partition Magic you may be able to make everything coexist.

As for me, I don't need or use Mac OS, and finally got all my software installed.

I wonder if using hacks I've seen published here and there, I could run Mac OS in MS Virtual PC? Or maybe install it on an external USB Hard drive?

Moving on to greener pastures

After a very successful run of technological achievements on my current project, it seems there's human incompatibilities that can't be patched and I am to suffer a cold reboot. As one of my previous clients wants me to work on some of their projects, I'll suffer only minor downtime. I can't say the same for this project I've not even been asked to do a handover for, but I wish all the best success to the team I've worked with, as I know that, with the IT people, everything ran smoothly and we did well. They will do well. Farewell.

In the meantime I'm taking a week off, so I'll be able to post the content I've had pending for a while, including MVP, SQL user instances and more WPF stuff.

Managing managers follow-up

I just wanted to point people to Joel's Evidence Based Scheduling. A lot of people often misinterpret my rant about estimates being an impossible dark art as going against tracking like the one developed by FogCreek (damn I wish I was attending their talk in London). The reality is that what I'm saying is reflected in the model:

The mythical perfect estimator, who exists only in your imagination, always gets every estimate exactly right.

Most estimators get the scale wrong but the relative estimates right.

And the one I'm the most in tune with.

When the project begins, the technical managers go off, meet with the business people, and come up with a list of features they think would take about three months, but which would really take twelve. When you think of writing code without thinking about all the steps you have to take, it always seems like it will take n time, when in reality it will probably take more like 4n time. When you do a real schedule, you add up all the tasks and realize that the project is going to take much longer than originally thought. The business people are unhappy.

Inept managers try to address this by figuring out how to get people to work faster. This is not very realistic. You might be able to hire more people, but they need to get up to speed and will probably be working at 50% efficiency for several months (and dragging down the efficiency of the people who have to mentor them).

You might be able to get 10% more raw code out of people temporarily at the cost of having them burn out 100% in a year. Not a big gain, and it’s a bit like eating your seed corn. Of course, when you overwork people, debugging time doubles and a late project becomes later. Splendid karma.

This all comes down to estimating a release date, certainly not thinking that your original estimate will be any accurate, because it won't. What do you end up doing?

If you do make a schedule, even before you start working, you’ll realize that you have to cut something, so you’ll cut the easy/fun feature and just do the useful/important feature. By forcing yourself to chose some features to cut, you wind up making a more powerful, better product with a better mix of good features that ships sooner.

I'm feeling happier today. Now go and read Joel, I don't agree with his prose all the time but his articles are deep and show a level of understanding of the writing software process that never ceases to amaze me.

WPF ClearType and the case of the blurry font

While at TechEd I had lunch with IanG and the discussion around the blurry text in WPF came around. I mentionned that this was due to sub-pixel positioning: fonts are now positioned based on where the font designer intended to put it rather than at the pixel boundary. You gain in accuracy but you loose in sharpness. I also mentioned that fonts were rendered fatter on mac to account for this blurriness, and spent a couple of days wondering if I said somehting very stupid.

So I googled a bit and found where I got some of these ideas. Let me direct you to Joel on Software for the beginning of your cyber journey towards font understanding. I am lead to believe by all I've read that at small font sizes ClearType doesn't respect the font weight and spacing properly in GDI ClearType, while it's respected much better on the WPF ClearType, making it much closer to MacOS X sub-pixel rendering.

The problem, as Joel points out, is that users don't like change. They end up considering WPF text as blurry, when a mac user would find them absolutely appropriate. Should Microsoft backtrack on their ClearType WPF work? The problem would be that to respect the pixel grid, you need a pixel grid, and in WPF you don't have one. Ouch.

One thing I forgot to mention as well is that some of your blurry text comes from using Menus and other windows with transparency. This will disable ClearType altogether and switch to gray-scale rendering. When you wonder which mode each window is, make a screen capture and zoom, you'll see the red and green artifacts. And that includes every menu you use in your application (how's that for a hidden gem).

[Update]

While browsing wikipedia I stumbled on the following text.

Increasingly, new approaches to screen rendering have reduced the importance of extensive TrueType hinting. Apple's rendering approach on Mac OS X ignores almost all the hints in a TrueType font, while Microsoft's ClearType ignores many hints, and works best with "lightly hinted" fonts.

One additional potential explanation of the difference between GDI and quartz sub-pixel rendering.

Technorati Tags: wpf, cleartype

Managing managers

I've worked on many projects in my career, and just like my style of development has changed, so have my managers. This is not a tutorial on how to manage your manager, as I don't believe you should. No, this is more of an open letter to those managers that seem to enjoy making your life hard when it could be so simple.

I've been advocating Agile methodologies for years, I've seen big design first fail miserably and tdd and xp succeed fantastically. I've also seen the reverse, with very well organized big design first projects, often on complicated enterprise-scale systems, with many developers split into smaller teams that then adopt whichever methodology they are comfortable with.

But the worst cause of failure has always been this strange beast that is the methodology-agnostic manager. By manager I mean anything from Project Manager to Dev lead to self-appointed boss, and by methodology-agnostic I mean someone not following a proven way to ensure you can actually deliver software. This can range from Scrum (that I'm covering here) to various waterfalls to ones grown in house and prooved by time. I'll call him Ralph for the purpose of this rant. And Ralph usually say any of the following.

We'll just drop the unit tests.

This usually comes with We'll document [remember, the unit tests are your main source of documentation!] and the very satisfactory I planned a handover phase on the day you finish your contract. It usually come with a justification: We can fix the bugs after the release date, so we'll be on target.

You just cornered yourself into certain failure, Ralph. This may do for v1, you may survive it for v2 with a lot of time and a lot more money, but by v3 you've failed. Let me repeat that: You will not survive beyond v2 if you do not bet on quality over quantity.

Just do it

That one is a bit borderline. There is always pressure to deliver software, and cutting corners is a fact of life for any software project. After all, shipping is choosing. However (you've seen that one coming didn't you?) this means cutting functionality to ensure quality.

Ralph has a tendency to be terrified at the idea of managing the customer. A new project starts, and sometimes the marketing guy over-promised. Sometimes, the customer has described what they want, and we've provided a rough estimate at the beginning of the project. Sometimes, the business asked, got told yes and now Ralph does not want to go back saying it's not possible to deliver.

The reality is that if you can't deliver, you won't. Trying to hide that fact is ridiculous, you'll end up cutting features or being late anyway. It's better to catch these things early, involve the customer in deciding where to re-prioritise. Otherwise Ralph will make the choices for the business, and probably ask his development team to work over-hours. At that point you've failed.

The other reality is that predicting how long software will take is a dark art that is bound to be inaccurate. I'd probably say that as the complexity of a project increases, the probability of reaching an initial estimate drops to 1 in 5 for any reasonably complex scenarios. You can be a bit off or a lot off, but you won't get what you initially planned.

Ralph loves his excel and this does not fit with his model, and that's understandable. I know the complexity of the tasks I'm doing right now, got a few ideas about the ones in the future, but I have no way to predict how it will all work in 6 months. And I've never seen a single project that, 6 months on (or even 3 months on), still has the same requirements, the same deliverables planned and the same functionality in. Business requirements change and they often get shaped as the software evolves.

What do you do then? The same as you did, you give a rough estimate on how long to deliver the core functionality you describe in use-cases or user stories. Then you shape your project on each sprint to split between the minimum needed to deliver that functionality and the refactoring / framework-ization you deem necessary.

When hard choices are to be made, you'll often find they are about which bugs to fix or which refactoring or new code you want, rather than actual user stories being fulfilled. And that's manageable.

There's a school of thoughts in XP that advocates delivering the minimum amount of functionality and refactoring until you get a good enough solution. However, this is only applicable to applications with good code coverage (> 80%). Parts of your application built without TDD or without tests are probably impossible to put under tests easily, and refactorings become increasingly dangerous and expensive. In those instances you're probably better off rewriting from the ground-up. Remember, your prototype is *not* your v1.

This won't get used for long

Ah this one is by far my favourite of all. The first rule is that the prototype is your v1. It *will* end up being used, and will take ages to be decommissioned, if it ever is. Your v1 will try and make this usable, but you'll be stuck with the original design. The customer will not understand why your v1 cost so much more for a few improvements over the prototype. Ralph will not have the courage to say that the prototype didn't cut the mustard.

The only way to sooth the process is to hammer to Ralph: The prototype is not your v1. I know it's in prod and it works but it is *not* a v1. You can ensure it doesn't get used further by putting time bombs, erasing all the data every day and changing where your app is deployed on every release.

Just quick fix damn it!

This is also a common pitfall. It is perfectly reasonable in some instances to walk around an issue you cannot resolve. You document it and discuss it after investigating the problem and finding the root cause.

This is where Ralph comes in. Usually you describe in your daily that an issue has arisen and you need to investigate. Ralph sees a risk, and immediately will start pressuring for any investigative work to be stopped and a fix to be found. Under some circumstances it's reasonable, under most implementing a work-around for an unknown root cause will just remove the symptom without knowing the illness. You just left for later a big giant loophole in your code-base. It is sometimes the right thing to do but rarely.

Scrum doesn't work

This one is fascinating. Scrum is a fabulous methodology, but just like it is very difficult not to stop doing test-driven development under pressure, it seems terribly difficult for the newly appointed scrum master not to fall back into bad habits: micro-management and authority.

All these things happen because Ralph feels he is the one giving orders and splitting the tasks, and as such needs to keep control. This does not work. Let me repeat that. This does not work. A scrum master is there to relieve any impediments on writing software. The product owner decides what gets priority. The development team is self-organizing, because the development team is the one building the software. The devs build it, the scrum master doesn't. Ralph doesn't. So Ralph shouldn't decide, and that's a good thing.

It is good to loose control over the minor details. I'll offer you two reasons.

First, only the developers understand the complexity of writing code for the project on the platform they target. If Ralph has a development background, it was probably years ago and chances are, because he's a project manager now he's not writing any code. He can't decide on how things get done because he doesn't understand them. Then Ralph spends an awful amount of time and effort trying to understand the details of each bit of functionality to regain control. This is the worse scenario as any development planning, meeting and decision has to involve him. Ralph is now needy and has become an impediment.

Second, Ralph will work more efficiently with his colleagues if he's not constantly fighting for power. The product owner decides where the priorities are and which direction the dev team will adopt. Remember, the product owner is the voice of the customer. It's whoever decides what gets in the software. Ralph doesn't use the software, he's not the user. He doesn't talk to the users, that's the product owner. As such he has no idea what bits of functionality the customer wants to implement first. And he has nearly no control over developers because he's not the one building the software and is not involved in writing the code. If Ralph continues to try bossing people around about the code, it will end up impeding the dev team's capability to deliver.

Of course sometimes you end up with the absolute worst case scenario where the scrum master and the product owner titles are both given to Ralph. That's a recipe for disaster.

The daily stand up

This is way too common. Ralph has other issues to discuss, ranging from adding coloured cells to his excel spreadsheet to being explained a component's functionality. So you end up with No we'll do this meeting first and then we'll do the stand-up. You can see the face of developers when they realize they're going to sit around a table and watch a discussion that has very low impact on them they didn't intend on participating in: after all they came for a daily stand-up. Or you end up with we don't need to stand-up for stand-up meetings, a corollary where the purpose of the stand-up is not understood so the format is not respected.

So here's my rule of thumbs to know when you're screwing up: If you hear a developer moaning before the meeting because he knows it's going to end up taking ages; if you sit down; if the discussion is between two people and the others watch: you just ruined your stand-up meeting and the rhythm that's supposed to be associated with it. You got everybody bored and lost an hour of their time.

One last thing to any scrum master out there. You're supposed to track down how many tasks get completed, how many new ones get added, and with those numbers do your estimations on the likelihood of completing your sprint. If you don't do that, the nice cards on the board probably don't help anyone. Analyzing those tasks is what will let Ralph do his spreadsheet, if he's not following them he's not doing his job properly.

The planning meeting

Well this one is easy, it takes time, we write down all the new tasks, we update the product backlog, decide what we want to achieve for the next iteration based on the prioritisation of tasks agreed with the product owner.

However, old habits die hard, and it is overwhelmingly difficult for Ralph to accept that the planning is there to plan the current sprint, not to answer the question of how many months the whole thing is going to take. It gives a good idea of where you are and how likely you'll match your target, but it is in no way there to let Ralph write his excel spreadsheet for hours and hours. It's Ralph's job to understand what is going on and report to the business, it is not the developers'. Ralph, you're the one in charge of excel, embrace the process and keep the planning you sell to the business to yourself.

The post-mortem

I like calling it the post-mortem but there are many other names for it, from wrap-up to release party to the more common sprint retrospective. This is the moment where you analyze why you've not matched your targets, what you've done well and how you can improve the process.

This goes bad when the blame game starts being played. The only way to get something efficient out of a post-mortem is for everybody to understand and agree that estimates are exactly that, estimates. This also means that you accept that you're late and can explain what were the reasons, ranging from your lack of skills to dependencies on other components. No one should get penalized for encountering problems.

The half scrum

Another common misconception is that you can adopt a few bits of Scrum and be successful. I've never seen a project with a bit of scrum in there succeed. Neither have I seen a big plan project succeed without a very strong structure to compensate for the unreliable nature of software development. All in all, because Ralph can't decide which bits of a methodology he's going to use, he uses a bit of everything. This is a recipe for disaster and confusion.

The split responsabilites

This one is tricky. One of the most important bits is the work split. Having each developer work on a different component is a normal occurrence. However, you'll often find that on small teams, it is more efficient to work altogether on one thing rather than split each developer on each project. Assigning ownership to part of the projects to one-man team destroys communication between developers, reduces the rate at which they can code, and ends up more often than not with an inability to deliver anything in a sprint: You do half of two tasks instead of finishing one task well.

Ralph often likes to separate his eggs and have each developer in charge of one bit, so he can go and manage each of them individually. It becomes much easier for his excel spreadsheet. This silo development undermines the concept of working as a team, and often comes from Ralph trying to regain control on a project he has no visibility on. As we've seen before, visibility is there, but Ralph refuses to change his ways and embrace the reason behind agile. This is once again a recipe for disaster.

Conclusion

Here's a recap of all the bad habits Ralph has:

• He gives up code quality to try and achieve deliveries
• He's taking away from the business the choice of what gets delivered and the prioritisation of those elements
• He lives in fear of telling the business how things are really going
• He believes it will be easier to deliver now and refactor later (the prototype is your v1)
• He does not care about finding the cause of an issue and focuses only on the work-around
• He can and need to understand and decide how the software is written and who writes what
• He has to give orders (it's often the case that Ralph is the project manager but not your line manager...)
• He doesn't respect the daily meeting
• He wants to fill his 12 month plan and doesn't track the sprint backlog (the tasks on the whiteboard)
• He wants the help of developers to do his 12-month plan, because the tasks are not enough (after all he doesn't track them)
• He believes the plan is the solution to everything
• He believes he's good enough to reinvent a methodology by mixing bits and pieces
• He believes in splitting people to split tasks and regain control (silo development)

This Ralph is a disastrous scenario and very rarely will you encounter people showing all those signs. But if you try to deliver software in this day and age, and you can see in your project management more than a few of those points, I would advise extreme caution, as I believe you'll end up failing on your deliveries and ending up with a unmotivated team.

On a bright note, Ian Cooper told me once something that I kept at heart: everybody tries to do the right thing. Not knowing and fearing are normal things, failing is part of learning, and people willing to evolve overcome those limitations. Great managers are the ones that take the knowledge from their coworkers, embrace that change and adapt to and work with and for the people they manage.

TechEd: Reflection, code generation etc.

I'm sitting in Roy Osherove's session on lightweight code generation. Roy is a formidable speaker, but there's probably a bit too much content and some areas are, I believe, not explained in enough depth to be approachable by neophytes.

Overall, a very good session with a lot of deep content, and a great overview of what is new in .net 2.0. However a lot of the focus is on solving real problems through these technologies, and a good dose of DLR and expression trees discussion would probably have been a great and smooth transition to the rest of what we've seen this whole week.

And Roy is singing I don't want to blog about it. What an extraordinary end to a very good talk!

Technorati Tags: TechEd-Developers, TechEd

TechEd will be over very soon...

I'm not taking the plane until tomorrow, but it's very sad to see everyone starting to leave. The exhibition center is closed. Only one talk remain where I'll go and have a chat with Roy Osherove.

What to do tonight? I've not managed to learn about any parties, maybe tonight will be the first night where I can see Barcelona. Or maybe not.

To be fair, I am tired, my brain is close to exhaustion with all the talks, arguments and chatting that's been going through, and I want to go back to my partner and have a relaxing weekend. Which means suffering the plane tomorrow morning, an experience I'm pretty sure I won't enjoy: I have a ticket for the evening but *need* to arrive in the middle of the afternoon in London.

Oh the fun!

Technorati Tags: TechEd-Developers, TechEd

TechEd: SeaDragon and PhotoSynth

This is a private session I ended up going to because of my craving for orange juice..

PhotoSynth is in Silverlight, but it's not clear which version it will be shipped with. It's a multi scale system, the size of objects is not important. The maximum info needed at any moment is whatever is on the screen. It supports pictures, text or even a fractal.

Another demo is an application of the scaling with an issue of the guardian, zooming out to show al pages and zoom in into a corner to see a very close text that you normally would need a microscope to see on a real printed animation.

This works with the hardware of the macine because the GPU is used. [EDIT: I thought silverlight was only software, or is there something more in the pieline?]

Another demo is a bunch of photos from flicker of Notre Dames, and generate a 3d reconstruction from that picture, and where all the cameras were positioned. You can then do a photo to photo navigation.

It's all about large navigations of photos and 3d, and SeaDragon became Photosynth. The system starts from images, extract features, then those feature get matched, and the matching turns into a 3d scene.

There's some work that ha been done in the summer with NASA. A picture of the shuttle display recrangles that are features: scale, orientation and corners.

Descriptors are informative about what the content is, but minimally informative about resolution, point of view, cropping, etc. If we extract the same features for a number of images, we manage to match them. All the features get in a feature index, look for nearest neighbors, and that gives matches between several pictures. Now that you  know that two points on two pictures match and you do that across pictures, you can find out where the camera would have been and where the point would have been.Each point in the 3d visualization is one of these match features.

The system used to take several days, and we're getting ready to release in the spring community photosynth which will let a single user capture a 3d environment on his own. You start being able to capture a real part of the world. What it will allow is for users to make their own synths and share them, and link them together. When you link it you can navigate seamlessly from one synth to the next. We're going to see overlapping synths.

We've also been exploring public collections.

One of the secrets in photosynth is that it's almost 3d. That Synth was done yesterday after 5 mnutes of crawling. Some of these transitions are less than ideal ones. If you notice, the reason is because all these photos are projected on flat planes. It doesn't correspond to the geometry of that place. It's in between 2d and 3d. Something that a lot of peole have asked us to do real 3d reconstruction. There's been a flurry of research around it, and some of our colleagues have done is fabulous. There's an experimental viewer made oer the summer, with a project with the BBC. It's reconstructed 3d, we're not projecting on flat planes anymore we're projecting onto 3d models reconstructed from these pictures.

Another demo used in Live Labs is very close to the birds eye morphine in virtual earth. In virtual earth there are models of the city, photogrammetry, gps, etc. In this case we don't use any models, they come from the photos.There are still some errors but we're not finished with

We're working on integration with Virtual Earth because it relates to the interplay between top down and bottom up representations.

Technorati Tags: TechEd, TechEd-Developers

TechEd: The irresistible Forces and the Moveable Objects

Pat is a fantastic speaker, as always. However as always, the content can be followed by slides as well as by listening. For some reason, all the rules about not reading your slides and not putting too much content on them, he doesn't follow. And by some speaker magic, he succeeds brilliantly. Goes to show that, to each their own style, as long as you captivate the audience!

The topics remind me a lot of what I've heard from many previous conferences, Microsoft or not: Just imagine when everything you know will have changed so much you won't be able to apply any of what you know now. The big topics are storage, massive parallelisation of work, the death of the hard drive, etc.

What is less ground-breaking is one of the demoed video. It looks terribly similar to a digital home video that was available with the second edition of Bill Gate's book. Interactive TV, interactive Phone and interactive Car. Except where you used to hear windows you now hearthe software + services. Oh dear!

To be fair, the fact that these videos are vision of where Microsoft wants to go means that while marketing memes and corporate memos come and go, the company has kept one way forward and keep on pushing it forward.

But the most important thing to get from the talk is the idea that consistency is not as important as you think. It's a subject a lot of people from Amazon, eBay and other companies that scale massively have learnt and are now starting to promote. It's less important to be right than it is to be consistent.

In the end, I would have to wonder if a talk that apply to a handful of companies worldwide was really what should've been the general highlight of the last day.

Technorati Tags: TechEd-Developers, TechEd

TechEd: What's new in PowerShell V2

[EDIT: I'm thrilled by this presentation, can't wait to see the new bits!]

Jeffrey P. Snover [EDIT: Posted on this on his blog]

Before we start talking about v2, lets' talk about v1. The response has been phenomenal. We had a discussion with the Marketing guys. I explained all these things that happened in the past and it'll be a long time before we reach a million download. Was I wrong! We had more than 1.5 Million downloads in the first year.

We have tools shipping today using PowerShell, exchange, MOM, etc. We have more than 20 partners producing powershell inside. We're now a CEC 2009, which means any microsoft server software has to be  powershell compatible.

Why the success? You can see the tasks for exchange with VBScript, the productivity for powershell is phenomenal. A customer had a 481 lines of script he replaced with one line thanks to WMI and html formatting.

The bad: Remoting is Cmdlet specific. There's no uniformity you can rely upon. Theres a great scripting language but you can't write Cmdlets in PowerShell. Our scripting environment developement is notepad and ps-debug, which is a pretty lame experience. Finally our API does not address common GUI needs. The normal model is admin tool built on top of powershell. The people doing that end up rewriting the same thing so we are doing it.

The big things for v2 are: Universal code execution model, GUI over PowerShell, Production Scripting, and community feedback.

So what is universal code execution model? Any code that runs should be able to run on one or more machines, LAN or WAN, restricted or not restricted environments, foreground and background, in short or long connections, using impersonation, and initiated by user input or by events.

The code execution model: Invoke-Expression works with computer names, the runspace, the throttle, the credentials, the ShellName, and connection information.

If you want to run it in a backgroun, you can use start-psjob, and the parameters are the same

When we talk about remoting, the scenarios we nail are fan-out 1 to many. The goal is to reduce the TCO. Second is Interactive, and i don't want to use telnet. We're not trying to replace telnet. We're not doing the graphics telnet. Next, Many to 1. If you write a service, you dont want to create a new process for each user, that would blow up. I want to listen on a port with WS-Management. Then we want the GUI over CLI.

Mobile object model. I want to run some code over there, I bring the objects back here but they're exchange objects and we don't have the dll. The way we do that is standardizing the model, we take the type, we serialize in a property bag. It's a stable cross-machine cross-time. Get an object, serialize. It takes one of the 22 types, if it's not there you call ToString. You can specify which serialization you want. Those objects we serialize everytime we reach a RunSpace boundary.

As soon as something crosses, we serialize, and when we reach the destination runspace, we deserialize. We store the original type and we store where it came from. I have a bunch of machines, i do a getFreeDiskSpace and you get the objects back.

[EDIT: Demo of using Invoke-expression].

Background jobs: we had an object called psjob, you can provide a runspace to it, or not. If you don't provide everything we'll run it locally. We require the latest version of WS-Management (1.1), this is included in Vista SP1 but there's no download, so you need SP1. We think the background jobs should be in process, but in this CTP it goes through the remoting.
Jobs can contain child jobs that can contian child jobs.

Restricted Runspaces are environents you can host to run potentially hostile requests safely. You get control of which language features are available, which commands can be executed, and there's white listing of commands.

There's a whole new set of hosting APIs. YOu can only issue one request per run-space. RunSpace pooling, which Exchange did, is a pool of runspaces and execute the job against that pool.

There are a new set of hosting APIs that bypass the parser completely. You can provide parameters, commands, to build a pipeline. There's also a typed invoke, and redo invoke.

[EDIT: Demo of the new hosting API.]

Production scripting: We have a grpahical PowerShell, Script Cmdlets, Script Internationalization/Data Language- You go to the web and solve the problem by downloading a script. It's 3 lines of code. Now imagine it's 3000 lines, it's gonna take a while before you put it into production. Well PowerShell can guarantee it's only data. You look at the lines of code that are not data to evaluate if you run. In scripts you want to highlight which parts of a script you don't need to delve into; Debugging; and finally packages and modules.

Graphical PowerShell is done in WPF it seems.

Script Cmdlets supports most everything a .net Cmdlet can do. You add attribute. You also have the $Cmdlet variable that you get in .net as your base class.

The engine does a ton of work for you. You change from the fucntion name to the Cmdlet. You can add a Mandatory attribute, a Position, an Alias, ValidatePattern etc.

The data language is a limited subset of languages, with comparison operatirs, if / elseif / else, well defined convertfrom cmdlets, limited variables, comments and pipelines, literals.

Data msgs { convertfrom-stringdata 'commands' }

then you import the localized data with Import-LocalizedData msgs.

We have a new command called PSBreakPoint. They can be set on lines, line number / column number, functions, variable read/write, or a breakpoint any time this command is called.By default we hit a breakpoint and break. Alternatively, you can provide an Action scriptblock to execute when you hit the breakpoint. We also have get-PSCallStack. It's a placeholder, but it will be great. In V2 it will be an array of stack information, with location, scope etc.

We're making progress in including community feedback, but we are working on the big bits first.

We now have a splatting operator. We also added a -split and a -join. We added a type for ADSISearcher.

 

Technorati Tags: TechEd-Developers, TechEd

No more content for today...

Sadly I've not found conference rooms with power plugs, so no more content for today. I'll probably do a quick update tonight of what was said but it may be fairly short.

Off to see IanG for his WPF talk.

TechEd: Networking in Windows Vista and Windows Server 2008t

.Why do we need to discuss the new network stack? We want faster applications, we want to connect to anyone else, and we want a simpler, direct model without yet another server. For all these reasons we need the next generation TCP/IP.

There are some issues with Vista network issue. The Multimedia Scheduler service.  Mark Russinovitch published an article about it [EDIT: Here's the link to the article about the interaction with networking and multimedia scheduler]. One hopes this will be fixed by SP1.\

The new tcp/ip stack is a huge project for Microsoft becase it's the first-time microsoft re-write since the early 90's. It was an outdated technology. NG TCP/IP ships with Vista and a slightly different in Server 2008.

The major improvements include receive window auto-tuning, Compound TCP for very fast LANs, ECN - a way for routers to let the client knows when it's overloaded to reroute the data, better support for lossy network - RFC 2582, 2883, 3517, 4138, and some great IPv6 network with Neighbour Unreachability Detection.

There are other benefits too:  much simpler API, security enhancements with API filtering and monitoring, support for stack offload, and multiprocessor scalability, which was not possible before because of NDIS 5.1 limitations.

There are also a few conveniences: there's no restart, there's auto-configuration and self-tuning of IPv4, and policy-based QoS. There's for users roaming support in IPv4 and (better) in IPv6, home network support has been simplified, and there's more efficient multicasting.

So far, it may change in the future, but so far we have full resistance to all TCP/IP level DoS attacks, and IPv6 for more security. You also have the concept of routing compartments.

Let me try and make you a bit enthusiastic about IPv6. We are running out of address. Don't believe me, believe these numbers. ARIN announced on 21th May 2007 that we will run out in 2010. RIPE 55 Meeting took place two weeks ago said we have 2-4 years before running out, and Mr Vint Cert said that on BBC news too.

Theoretically, we could live without ip addresses. We could have several levels of NATs but it blocks us from performing p2p networks. NAT makes peer-to-peer very difficult, because of security. We need to find a balance. Seucirty of IPv4 doesn't exist. IPSec is optional, not so in IPv6. IPv6 provides significantly better QoS than IPv4. Routing tables will shrink with IPv6. Mobility does not work in IPv4. Finally device autoconf doesn't work so well. Some technologies has been backported, but it doesn't work very well.

6 benefits of IPv6: Address depletion solved, ent2end p2p communication restored, mobile and roaming connectivity, international mis-allocation, autoconf.

Terminology, we still talk about Hosts, we have LAN segments, Links, Subnets... In IPv4 a subnet is restricted to a few devices within the same network. Any enterprise network has a wardrobe of devices to solve the subnet. You don't have subnets communicating with other subnets communicating thanks to a Cisco router.

We have 128 bits for our addresses. We usually split it with 64 bits for subnet IDs, and 64 bits for interface ID.

IPv6 is expressed as 8 blocks of hexadecimal 16bits components.

A great thing for IPv6 is that there are classes instead of subnet masks: The CIDR notation.

There are 3 classes of address: Unicast, Multicast, Anycast. There are no more broadcast address. IPv4 wasn't designed for networks with very large amount of machines within the same submask. Now it's gone, you'll get a performance boost. Instad of broadcasting, there's a bunch of neighbor networking protocols.

There's a new unspecified address, and there's a new Loopback: ::1. There's the concept of a well known address. DNS Servers are supposed to be FEC0::FFF::1 FEC0::FFFF::2 and FFEC0::FFFF::3. As a developer, be aware that you should be able to accept v6 addresses.

Configuration can use DHCP, where the address is generated. What they don't need to perform anymore is the address generation. The other thing is that addresses expire. That gives us back some efficeincy and clears the address space.

Addresses expire, and one of the key things in IPv6 is that you have multiple address. You have global addresses, one expired, one in the future, some local addresses have local addresses, or several link local addresses. It's the key change: in IPv4, when designed, all the processing was on routers. In IPv6 we change the balance, nodes have enough power to do some networking process. That lets us have autonomous networks, by moving.

Mobility is the coolest thing, it's so simple and powerful it's beautiful. When a device tries to connect, it can't get any neighbors. Then it creates a local address. If the router contributes, you can change global addresses but your home address will make sure data keeps on flowing.

VIsta has what is known as a Dual-Layer architecture, isntead of a Dual-Stack implementation,w hich meant that the ipv6 TCP and UDP was grouped together and completely separate from ipv4.

Interoperability is probably a key issue for anything being introduced. IPv6 was built-in for interoperability: ISATAP, 6to4, 6over4, Teredo, portProxy. Teredo is very important because it bootstraps the world from IPv6 to IPv4. However it is not without problems.

That brings us to peer-to-peer networking. P2P goal is to enable direct communication between applications without relying on centralized  servers.  You could keep commnciating with everyone by keeping connections opened. You need to prevent DoS, Secure with crypto, resolve changin addresses, maintain dynamically changing group of nodes, and communicate with a subnet.

The p2p in vista is based in IPv6. It's installed by default in Windows Vista. At heart you have low-level and high level APIs.

The concept of a cloud is a group of peers that can communciate with each others. There's a global cloud with everybody, and a link local cloud, or several private global clouds.

A peer is a machine runnnign the application that connects to other. It has a Name and a PNRP ID, and a Certified Peer Address.

[EDIT: This is overall a big presentation about clouds, chrod based networks, graphs, etc. I worked for too many years with gnutella / gpulp / jxta to contiue blogging about the presentation. Battery is nearly dead anyway.]

Technorati Tags: TechEd-Developers, TechEd

TechEd: Threat Modeling

As an industry, we're good at looking at code compared to looking at security. You can run all the tools you want, you'll find security bug but you won't uncover any design bugs: they're the hardest to fix.

At the very high level, threat modeling is about identifying threats, mitigate them and move on. A couple of years ago, the threat modeling information we were giving was pretty lousy, you couldn't use it much if you were not a security expert. That's what we've been focusing on over the last few years.

When the Windows Mobile team did their first Threat Model, they had to answer a question: Is a stolen mobile devide still in the threat model. If the bad guy has the device, are you still trying to mitigate? They decided to try and give a fight, and include it.

For Windows Server 2003, we've asked the team to think about an admin browsing on a Domain Controller. If you compromise a DC, you can compromise the whole environment. The question always come to mobile code, and we decided to deactivate all mobile code.

Finally, you need to define who your users are. If your user is an IT guy rather than a mother, they're different people. When you present a mitigation to a user, will they understand it?

You have to model your application. We use Data Flow Diagrams for that. You model data, which is where most of the security issues come from these days.

Most whiteboard architectures are like DFD. [EDIT: Won't reproduce the diagrams, the process is defined on Larry's blog.]

When DFD talk about processes, they're not .exe, they're whatever process data.

You need to define in your DFD trust boundaries. You need to decide if two processes, when they talk together, if they trust each others. In DFD it's explicit.

For Vista we decided to add this functionality to run some processes with lower privileges, like IE runs in protected mode, and there's a trust boundary between IE and the OS. A process boundary may be between the system process and the user, as you don't trust data coming from the user. Finally Kernel transition is also a trust boundary.

Theres several types of DFDs. Context diagrams give a high-level content. In level 0, you dig into more details. I've rarely seen much deeper than Level 1. If you go over, you're probably in analysis paralysis.

One feature that never made into Vista was the Castle project: it was where workgroups meet the domains. In a DC the domain authenticate you. In the home, you have several computers but no domain. If you have 5 computers and no DC, how do you synchronize files, passwords etc across machines. So we came with the idea of a castle, a machine that was responsible for synchronizing the passwords.

Local user ---> Castle service -/-/-> Remote castle

The level 0 DFD contains many services. The shell runs as the user and communicate with the castle service: There's a trust boundary.

The castle service runs as SYSTEM because it has to manipulate the SAM. Config data in the registry is accessed by the Castle service but there's no trust boundary. And finally the machine boundary.

Purists would say that a service should be a complex service and be described in more level, but if it all runs within the same process boundary, having a simple process is good enough.

Each DFD is susceptible to certain kind of threats: STRIDE. [EDIT: again, Larry has the description. Saves me some typing]

Example of repudiation. My brother in law lied when sending a check. He said it was sent, I thought he was lying. I got the proof when the check arrived, because they were all post-marked. You use a disinterested and trusted third party.

Each element in the DFD is subject to attack. How you attack them depends on the element type. You attack data stores differently than you do with processes. How do you protect these elements?

You need to determine threats. You have primary threats and secondary threats.

If you have an external entity (a user for example), you're subject to spoofing. It's subject to repudiation. Data flow and data store can all be tampered. The vast majority of DOS come from processes, rarely against data stores. Finally processes are subject to all of the STRIDE attacks.

Secondary threats come from threat trees. They've been around for a long time. They're a bunch of pre-conditions that can lead to a threat. Unless you're a security expert, you do not know what the pre-conditions are. We made away with threat trees, and work on canonical threats.

For every primary threat, we build the threat tree. At the top of the root you have the prime threat. For example spoofing a user, there are 4 ways of doing it. For the first way, you have several reasons, and several causes which lead you to the leaf nodes in the tree. You can ask very interesting questions about the leaf nodes. [EDIT: Example about user spoofing].

The example of analyzing a bank log-in system. The connection is SSL, the password has to be strong. However the cookie was predictable, as it was i+1 for each new authentication. Suddenly by predicting the cookie you can hijack someone else's session.

We have in the SDL book all the threat trees and all the questions you can ask about the tree nodes.

Every single information disclosure threat can become a privacy issue. If you have an PII you can get a lawsuit. Any data store can be attacked.

Now you need to calculate the risk of every of these threats. We used to use numbers to calculate risks. but they're always wrong. Plus, the more degrees of freedom you have in the calculation, you can easily fudge the numbers so you don't have to fix the bugs. The more numbers, the easiest.

We used to use DREAD. I don't like it. Even David posted a weblog about it.

Calculating risk with heuristics with a simple rules of thumb: we remind you of that with the MSRC bulletin rankings. The problem with numbers is that if the number is under a level, subjectivity will permit you not to fix it, whereas the MSRC you have no way of doing that. You ask is it remote / local, deos it crash the machine, is it a server or a client product. The different levels are all described in the book.

Now you need to mitigate the threats. You have a bunch of options: leave as-is, it's low priority, trying to fix it may delay the product etc; remove the product like we did with castle; remety with technoogy counter measure; warning users, but i don't believe in the prompt, you push the security of the system on the user instead of taking ownership of the problem, its not a good solution if it's your only mitigation.

Each STRIDE threat can be mitigated. Spoofing, you have authentication. The authentication technologies you use are different: Amazon.com uses SSL. They use encryption, authentication and integrity. Tampering, you can have integrity: Digital signatures, hashes, etc. Repudiation threat is mitigated with non-repudiation, strong third parties etc. Information disclosure: You have confidentiality, like ACLs. DOS, you have availability. Elevate of Privilege you have authorization.

[EDIT" Won't take notes on the castle example, so i can enjoy it. Maybe Michael could publish that on his blog as an example!]

Technorati Tags: TechEd, TechEd-Developers

Skipping the first session

Even though I went to bed early, still woke up exhausted, so I'll give a miss to the first session. Such a beautiful day, it's a crime to not have some of those sessions outside.

Technorati Tags: TechEd

TechEd reactions

Here's a few things that went under my radar in the blogosphere about teched:

• Gill gives an overview of his first day, and think Barcelona is dirty at first sight
• Joao talks about the Influencer Community Camps pre-conference. He also talks about the CSD influencers dinner on monday, but I'm afraid I was not invited, neither was I on tuesday for the MVP (as you know, I'm not an MVP :)). But tomorrow I'll attend the MS Influencers Party, Wednesday the country party, and Thursday I'm meeting up with some people from the Windows Live team. So much to do, so little time!
• Sascha talks about the World of Warcraft IDE microsoft will release with their partner as an open source project based on Visual Studio shell that was demo'd during the keynote. Go and check it out. And Ronan let us know he'll be in the Ask The Experts for Team System.
• Andrei talks about the Microsoft Sync announcement, personally the one area I'm the most thrilled about, so thrilled the SDK is already installed and I'm coding away :)

Overall, not much noise so far. Maybe the PDC next year will be louder! More content to follow, and tomorrow pictures and videos!

Technorati Tags: TechEd, TechEd-Developers

TechEd: Security Development Lifecycle: Development Practices

[EDIT: By Michael Howard, definitely a legend in my book!]

Until 8 years ago, i was completely focused on security features. But a small group of us, around 1999, decided to split from the Windows group and focus on how to design software.

Security people suck. They only see things in black and white, you can either have secure or not secure. The real life doesn't work like that. Theres plenty of shades of gray, and security people don't see gray. What they're good at is to tell you when youre wrong, but they won't tell you how to do things.

We added the NX to WindowsXP SP2. The first wave of AMD devices supported the option of not executing data, as most security vulnerabilities get introduced as data that then get executed.

When something is not on by default, people don't turn it on. Also, if it's not on by default it won't get as much testing.

When we activated NX by default, Flash, Acrobat, Java VM don't work. So we had to choose between not running by default and not get used, or be on by default and nothing works. As a security guy I said break the applications. But my manager said that didn't work! So we decided to have NX on by default only for opt-in processes.

The one thing I learnt is that there are shades of gray.

References to books for Writing secure code for Windows Vista and Writing secure code.

So what is the SDL? It's a bunch of requirements and recommendations, and the post release process. It's taking the lifecycle and improve the security by reducing vulnerabilities and reducing the severity of the vulnerabilities.

Lot of people think you can improve the quality by adopting some process, you'll improve the security of the software: there's no evidence! You can't think of general quality stuff, you need to focus on security itself.

The matrix I use is the amount of bulletins post-shipping. Since Vista, we have fixed 70 vulnerabilities in XP, 33 in Vista. In a perfect world it would be 0, but I'm happy with the 50% reduction. IIS6 only had 3 bulletins, sql server had one 4 years ago. Even the analysts start recognizing this.

The SDL is made of two big chucks: Requirements and recommendations (must and should).

Some functionality was invented a long time ago and was fine 20 years ago. But today it doesn't work because of threats. For example CRT (C runtime). You can't write new code that uses the old APIs, 120 functions. You can include banned.h which works with gcc and others, or use VS2005 that puts a warning.

In the SDL, if you put forward a requirement, you need to be prescriptive about what the fix is. For the CRT, we recommend two libraries: Safe CRT and strsafe. VS also automatically replace the banned functions if it knows the destination buffer size.

One of my favourite tool is SAL, Standard Annotation Language. It helps static analysis tools. The shell team decided to annotate the shell headers. He ran the tools we have at microsoft again and found 100 bugs. It lets you annotate your header files with how attributes relate to each others, if they're optional, in, out, etc...

Integer arithmetic issues can be resolved in several ways. First, use the latest C++ compiler, it protects you against the operator::new overflows. It crashes your process but that's better than running code injection vulnerabilities. Second, review your memory allocation: Is there any calculation to calculate a buffer size? Does the bad guy controls one or more operations? Finally, you can use IntSafe (C) and SafeInt (C++).

Next thing is the no weak crypto. A lot of algorythm that were ok a few years ago are not ok anymore. No new code can use MD4 and MD5 (collision of 1 in a million in MD5), SHA1 is showing signs of weakness, so it's banned too. Use SHA2. Can't use DES or triple DES - use AES. RC4 is not allowed because it's a stream cipher, no symetric keys < 128bits, no RSA keys, no weak random number generation, no embedded secrets and be crypto agile: don't define which crypto you use in your code. You should also not have weak ACLs: we found a bug a couple of years ago where a vendor was intalling their device driver in ProgramFiles. They changed the ACL to Everyone full control. Michael's advice: don't touch ACLs. The defaults are very good, don't change them.

We also have requirements about compiler switches. The GS flag adds random numbers to the stack: you need to guess the cookie we've set because we check it, and if it's not what we expect we crash the process. When the blaster worm happened, win2000 and XP were infected. Windows Server 2003 wasn't infected but the coding was there. We detected the buffer overflow at runtime and crashed. It was affected but not infected. For some arguments we copy the data earlier in the stack. In Vista, we compile with the orcas compiler.

Safe exception handler would have prevented CodeRed. The PE header has the address of the exception handler. At runtime the OS checks the exception against the list in the PE header. NX is the No Execute, ad DynamicBase is base address randomization for OS code. Finally, /Strict MIDL generates safer IDLs.

Although tools are incredibly useful, they don't make the code secure. Bad code and bad developers won't be saved by tools. There shoudl be nothing special about security, each developer should include it as part of getting the job done.

In Vista there's an option to detect any heap corruption and kill you on the spot.

The only scenario in which reliability guys have a conflict with security guys is when we bluescreen the machine if the security event log is full.

There are also web-specific guidelines. A lot of people write a lot of rubbish code. Use the Xss library, don't let SQL injection happen. Deny access to the underlying tables, only allow views and stored procedures.

Gadgets are Html / javascript / Silverlight. If your gadget renders untrusted data, you can call anything. Any exploit code can run. The first thing you should do is validate all your incoming data, and code review for insecure use of innerHtml, document.write, and Eval (which is one letter away from evil and there's a reason for that).

Inspect your Gadget from Michale Howard & David Ross.

Technorati Tags: TechEd, TechEd-Developers

TechEd: Metropolis - Interchangeability of Operations

[EDIT: Unbelievable but you have to sneak in the conference rooms to get access to the one power point that's hidden under traps. Hopefully no one will ask me to unplug myself because of Health and Safety but you never know... An IT conference without power plugs, what a great idea!]

I've used the term Metropolis for a set of talks. Look at Cities, Transportation, manufacturing and try and find some trends. From looking at cities we can learn a lot about what will happen in IT.

At a high level, IT shops map to cities. You have evolution that occurs in isolation, and then things get connected, which causes change. Factories and buildings are comparable to applications. Each application is part of an IT shop. Retail maps to business process. It's about tying together deliveries that come from different origins.

Manufacturing goods and structured data and operations are similar. That what we see as data structures and operations are made to fit better together.

I don't believe it's an analogy, it's not helping to think about a subject, it's the same drive behind both technologies.

What are the challenges in SOA? Explicit boundaries; Autonomy - you can replace your service without impacting the other services.

How do you perform actions across these boundaries. What trust do you use? SOAs are behaving the way you see people in business behave in the real world. Its an opening up of the autonomy and explicit boundaries.

Those that know of me know I've been doing transactions for a long time; but I've grown to believe that transactions across boundaries are very fragile. I don't think services as boundary of trusts will engage into transactions with other services. Some say two-phase transactions will happen across services, some say they shouldn't because of trust issue. For this talk, each service will have a transaction but will not hold locks for another service.

When you look at manufacturing, there was a transition from hand-crafted to automated. A machine creates the part. The shape and form was so crude with the machine that you had to manually change them before assembling. Interchangeability was driven by being able to exchange pieces between two broken guns on a battlefield to make one that work.

In computing we're massively stuck because we don't do very well on how to exchange services. The important points that you need to take from this transaction:

• Autonomy
• Agreement is different: Attempt and confirm/cancel
• Interchangeability: One operation is as good as the next. Fullfill my requirement, be it that you use A or B. You execute your operation on a pool of services.
• Semantics: How do you make the request to rearrange and reorder. You need Precision but you don't want the Intricacy.
• Variety: It can be made out of interchangeable of the operations. If you put different goods in WalMart, the combination is astonishing, but they come from a limited number of pools.

Let's see the history of manufacturing. If you go back to the early nineteenth century, you'd order one thing and you'll get one gun. You have a lot of craftsmen, each of them builds a complete item. But each person made a different gun because it was based on their own approach. Completing anything needed fitting. Soft parts were stamped because they couldn't be exchanged.

The American way of manufacturing comes from shortage of labour. They started building machines to do the work. The problem was that the machine created parts were still inaccurate. Because of that, they need adjustment. Fitting was still required.

With the armory system of manufacturing. LeBlanc proposed interchangeable small parts but it was dropped, but it was brought back in America, or at least the idea. By creating Jugs, Gauges, and the precision work John Hall, the price of manufacturing the pieces increased because you had to be picky with the pieces you get. You had interchangeability but the economics didn't work.

From armories to bicycles to sewing machines, all the way to cars with Ford, impacts what we do today. The disassembly plant: the meat arrived in Chicago, and each butcher was responsible for cutting one piece of meat. There was very high throughput. In assembly, Ford moved the parts to the people rather than the people to the parts.

The Ford factory made sure there was no fitting, every part had to be fully designed, custom machines were made, so all the pieces could be interchangeable. The machine was giving the precision and the accuracy. Even the building was shaped to optimize the production. But it became impossible to change the car design as you would now have to knock the building down.

Interchangeability is great but there are other requirements! Which brings us to General Motors. GM gradually introduced the concept of a yearly model, which introduced the requirement of being agile. You suddenly need multi-purpose machine tools. For this you had to be able to re-arrange the components of the machine to create a new piece. Mechanization, interchangeability and rapid changeover to new models.

Let's talk about the American system of transaction processing. You start your activities, be it a human or a machine, and it goes to your database, you calculate ad you return your answer. But you have to do fitting inside the transaction. There's no boundaries when you update two databases within a transaction. Within each transaction you need to fit the work. But it gives you volume.

We need transactional "machine tools". No precision tools: our current tools can be used in different ways. Different programmers get different results. They're wonderful for labour savings, but not to separate their behaviour. Today, we have applications that have external behavior that are similar but the inside is different. If we are to take our applications apart, we need to make the internals of our applications interchangeable.

Services are connected by messaging. You don't know how the operation is executed, but you know the contract of the messages and the contract of the order in which messages are exchanged. Interaction is based on business functionality. The services share operations, not data. Sometimes there's reference data. It's like a department store catalog, you use that data to connect to your business.

What about optimistic concurrency control? Can you do optimistic concurrency work with your bank? No, because there's a security boundary. You need to think about trust.

Autonomy means independent control. My local business logic decides how i change my local data. I decide what changes. If you want a change, ask me to do a business operation.

Long-running work. Services don't share transactions. How do they cooperatively make decisions? how do independent businesses take decisions? There's tentative operations, with reservations, cancelable orders, or confirmation. If you buy a house in the states, as a buyer you give everything to the Escrow company. The bank, the buyer, the seller, everyone puts their trust in the escrow company. However the sell is only a reservation until you know if you will get it. The only guarantee is that you'll get your money back if the deal doesn't work out.

Coordinating n-systems requires that at least n-1 accept uncertainty. You system needs to be in a confused state. In 2-phase commit, the database maintains locks. With cancelable operations, uncertainty is the reservation you have for a hotel room. There's an intrinsic conflict between consistency and availability. Two Phase commit is the anti-availability protocol.

Because operations are cancelable, you can reorder them. Your cancellation is not an undo but an action you receive to cope with the failure. So what are the semantics for cancellation / confirmation?

Cancellation is about coping with the operation not being done. You accept the right to cancel. Confirmation is the guy that has the right to cancel saying that he won't, for example hotel room confirmations in the morning. Airline companies clear up their overbooking once the plain leave.

To make your operation cancelable, you want to reserve its effect until the confirmation. If an operation is unique, you must lock its effect until you cancel it. You need to make a decision between provisioning and overbooking. How do you manage the pool of resources?

One operation in a class is the same as another. If you reserve a king-size room, it could be any of the rooms. Interchangeability comes by increasing commonality.

It's a pain to offer cancelable operations. But you want to do it because your customers need that in order to function correctly. Its annoying that the customer can be fickle but it's the reality of a complex application.

We used to make decisions atomically by committing or rolling back your transactions. You want to now think about how things compose, but for that you need to have each service being part of a long running work.

In the armory system of transaction processing, you have a complete solution when you assemble small services. In manufacturing you need precision. You need to understand the constraints of your operation. We've made components that were multi-purpose. What you need to do is to have the least amount of functionality to make them interchangeable.

Resource-oriented data lives longer but is changed by long-running operations, like bank records. You have activity-oriented data, that gets created when you start doing a job, and retires when the operation is complete, like an order.

These two classes of data have different characteristics.

In both cases, the data is encapsulated within the service, and there's no optimistic concurrency control externally. You mediate access through business logic.

Let's take a Foo operation. First case, the data is isolated. If I change something about someone's order, I only change a bit of data. But for Joe's order, that may modify my resource-oriented data by changing my inventory. Activity oriented data is used for a single long-running activity and won't span several. You can create a tentative operation, it's easy to track down in case the operation is canceled, but nothing is shared so there's no impact. What happens if you share data?

It becomes harder in resource-oriented services. What is interacting with what? They make commitments and these commitments need to be tracked. You have an association between activity data, the reservation, and the resource oriented data, the list of hotel rooms that are available every day.

Specificity is not interchangeability, Variety is the enemy of interchangeability.

There's a lot of work around DSLs. You express the intent of the business domain with a domain language. That's a great foundation to think about the issues that have been raised.

This is heavy-duty reconfigureable manufacturing machines. You don't need much skills to use DSLs if they're constrained enough.

Service operations are like standard parts, whereas tentative operations compose. That lets you do standard software components and use them across several products. All the products look different on the outside but they're all the same inside.

We should learn about that manufacturing part, and do some rambling philosophy.

Atomic transactions are singularities. New challenges happen as you spread work across space and time. Interchangeability helps relax space and time. You can interleave and reorder. Variety means you have lots of options and choices, but you can't keep stocks. Interchangeability needs precision rather than variety.

EDIT: Some form of question about Wal-Mart using Child labour and how it relates to services. Sometimes reality is better than science fiction!

Update 26th November 2007: http://blogs.msdn.com/pathelland/archive/2007/11/25/presentation-of-metropolis-interchangeability-of-operations-at-teched-emea-in-barcelona.aspx

Technorati Tags: TechEd, TechEd-Developers