Managing managers

I've worked on many projects in my career, and just like my style of development has changed, so have my managers. This is not a tutorial on how to manage your manager, as I don't believe you should. No, this is more of an open letter to those managers that seem to enjoy making your life hard when it could be so simple.

I've been advocating Agile methodologies for years, I've seen big design first fail miserably and tdd and xp succeed fantastically. I've also seen the reverse, with very well organized big design first projects, often on complicated enterprise-scale systems, with many developers split into smaller teams that then adopt whichever methodology they are comfortable with.

But the worst cause of failure has always been this strange beast that is the methodology-agnostic manager. By manager I mean anything from Project Manager to Dev lead to self-appointed boss, and by methodology-agnostic I mean someone not following a proven way to ensure you can actually deliver software. This can range from Scrum (that I'm covering here) to various waterfalls to ones grown in house and prooved by time. I'll call him Ralph for the purpose of this rant. And Ralph usually say any of the following.

We'll just drop the unit tests.

This usually comes with We'll document [remember, the unit tests are your main source of documentation!] and the very satisfactory I planned a handover phase on the day you finish your contract. It usually come with a justification: We can fix the bugs after the release date, so we'll be on target.

You just cornered yourself into certain failure, Ralph. This may do for v1, you may survive it for v2 with a lot of time and a lot more money, but by v3 you've failed. Let me repeat that: You will not survive beyond v2 if you do not bet on quality over quantity.

Just do it

That one is a bit borderline. There is always pressure to deliver software, and cutting corners is a fact of life for any software project. After all, shipping is choosing. However (you've seen that one coming didn't you?) this means cutting functionality to ensure quality.

Ralph has a tendency to be terrified at the idea of managing the customer. A new project starts, and sometimes the marketing guy over-promised. Sometimes, the customer has described what they want, and we've provided a rough estimate at the beginning of the project. Sometimes, the business asked, got told yes and now Ralph does not want to go back saying it's not possible to deliver.

The reality is that if you can't deliver, you won't. Trying to hide that fact is ridiculous, you'll end up cutting features or being late anyway. It's better to catch these things early, involve the customer in deciding where to re-prioritise. Otherwise Ralph will make the choices for the business, and probably ask his development team to work over-hours. At that point you've failed.

The other reality is that predicting how long software will take is a dark art that is bound to be inaccurate. I'd probably say that as the complexity of a project increases, the probability of reaching an initial estimate drops to 1 in 5 for any reasonably complex scenarios. You can be a bit off or a lot off, but you won't get what you initially planned.

Ralph loves his excel and this does not fit with his model, and that's understandable. I know the complexity of the tasks I'm doing right now, got a few ideas about the ones in the future, but I have no way to predict how it will all work in 6 months. And I've never seen a single project that, 6 months on (or even 3 months on), still has the same requirements, the same deliverables planned and the same functionality in. Business requirements change and they often get shaped as the software evolves.

What do you do then? The same as you did, you give a rough estimate on how long to deliver the core functionality you describe in use-cases or user stories. Then you shape your project on each sprint to split between the minimum needed to deliver that functionality and the refactoring / framework-ization you deem necessary.

When hard choices are to be made, you'll often find they are about which bugs to fix or which refactoring or new code you want, rather than actual user stories being fulfilled. And that's manageable.

There's a school of thoughts in XP that advocates delivering the minimum amount of functionality and refactoring until you get a good enough solution. However, this is only applicable to applications with good code coverage (> 80%). Parts of your application built without TDD or without tests are probably impossible to put under tests easily, and refactorings become increasingly dangerous and expensive. In those instances you're probably better off rewriting from the ground-up. Remember, your prototype is *not* your v1.

This won't get used for long

Ah this one is by far my favourite of all. The first rule is that the prototype is your v1. It *will* end up being used, and will take ages to be decommissioned, if it ever is. Your v1 will try and make this usable, but you'll be stuck with the original design. The customer will not understand why your v1 cost so much more for a few improvements over the prototype. Ralph will not have the courage to say that the prototype didn't cut the mustard.

The only way to sooth the process is to hammer to Ralph: The prototype is not your v1. I know it's in prod and it works but it is *not* a v1. You can ensure it doesn't get used further by putting time bombs, erasing all the data every day and changing where your app is deployed on every release.

Just quick fix damn it!

This is also a common pitfall. It is perfectly reasonable in some instances to walk around an issue you cannot resolve. You document it and discuss it after investigating the problem and finding the root cause.

This is where Ralph comes in. Usually you describe in your daily that an issue has arisen and you need to investigate. Ralph sees a risk, and immediately will start pressuring for any investigative work to be stopped and a fix to be found. Under some circumstances it's reasonable, under most implementing a work-around for an unknown root cause will just remove the symptom without knowing the illness. You just left for later a big giant loophole in your code-base. It is sometimes the right thing to do but rarely.

Scrum doesn't work

This one is fascinating. Scrum is a fabulous methodology, but just like it is very difficult not to stop doing test-driven development under pressure, it seems terribly difficult for the newly appointed scrum master not to fall back into bad habits: micro-management and authority.

All these things happen because Ralph feels he is the one giving orders and splitting the tasks, and as such needs to keep control. This does not work. Let me repeat that. This does not work. A scrum master is there to relieve any impediments on writing software. The product owner decides what gets priority. The development team is self-organizing, because the development team is the one building the software. The devs build it, the scrum master doesn't. Ralph doesn't. So Ralph shouldn't decide, and that's a good thing.

It is good to loose control over the minor details. I'll offer you two reasons.

First, only the developers understand the complexity of writing code for the project on the platform they target. If Ralph has a development background, it was probably years ago and chances are, because he's a project manager now he's not writing any code. He can't decide on how things get done because he doesn't understand them. Then Ralph spends an awful amount of time and effort trying to understand the details of each bit of functionality to regain control. This is the worse scenario as any development planning, meeting and decision has to involve him. Ralph is now needy and has become an impediment.

Second, Ralph will work more efficiently with his colleagues if he's not constantly fighting for power. The product owner decides where the priorities are and which direction the dev team will adopt. Remember, the product owner is the voice of the customer. It's whoever decides what gets in the software. Ralph doesn't use the software, he's not the user. He doesn't talk to the users, that's the product owner. As such he has no idea what bits of functionality the customer wants to implement first. And he has nearly no control over developers because he's not the one building the software and is not involved in writing the code. If Ralph continues to try bossing people around about the code, it will end up impeding the dev team's capability to deliver.

Of course sometimes you end up with the absolute worst case scenario where the scrum master and the product owner titles are both given to Ralph. That's a recipe for disaster.

The daily stand up

This is way too common. Ralph has other issues to discuss, ranging from adding coloured cells to his excel spreadsheet to being explained a component's functionality. So you end up with No we'll do this meeting first and then we'll do the stand-up. You can see the face of developers when they realize they're going to sit around a table and watch a discussion that has very low impact on them they didn't intend on participating in: after all they came for a daily stand-up. Or you end up with we don't need to stand-up for stand-up meetings, a corollary where the purpose of the stand-up is not understood so the format is not respected.

So here's my rule of thumbs to know when you're screwing up: If you hear a developer moaning before the meeting because he knows it's going to end up taking ages; if you sit down; if the discussion is between two people and the others watch: you just ruined your stand-up meeting and the rhythm that's supposed to be associated with it. You got everybody bored and lost an hour of their time.

One last thing to any scrum master out there. You're supposed to track down how many tasks get completed, how many new ones get added, and with those numbers do your estimations on the likelihood of completing your sprint. If you don't do that, the nice cards on the board probably don't help anyone. Analyzing those tasks is what will let Ralph do his spreadsheet, if he's not following them he's not doing his job properly.

The planning meeting

Well this one is easy, it takes time, we write down all the new tasks, we update the product backlog, decide what we want to achieve for the next iteration based on the prioritisation of tasks agreed with the product owner.

However, old habits die hard, and it is overwhelmingly difficult for Ralph to accept that the planning is there to plan the current sprint, not to answer the question of how many months the whole thing is going to take. It gives a good idea of where you are and how likely you'll match your target, but it is in no way there to let Ralph write his excel spreadsheet for hours and hours. It's Ralph's job to understand what is going on and report to the business, it is not the developers'. Ralph, you're the one in charge of excel, embrace the process and keep the planning you sell to the business to yourself.

The post-mortem

I like calling it the post-mortem but there are many other names for it, from wrap-up to release party to the more common sprint retrospective. This is the moment where you analyze why you've not matched your targets, what you've done well and how you can improve the process.

This goes bad when the blame game starts being played. The only way to get something efficient out of a post-mortem is for everybody to understand and agree that estimates are exactly that, estimates. This also means that you accept that you're late and can explain what were the reasons, ranging from your lack of skills to dependencies on other components. No one should get penalized for encountering problems.

The half scrum

Another common misconception is that you can adopt a few bits of Scrum and be successful. I've never seen a project with a bit of scrum in there succeed. Neither have I seen a big plan project succeed without a very strong structure to compensate for the unreliable nature of software development. All in all, because Ralph can't decide which bits of a methodology he's going to use, he uses a bit of everything. This is a recipe for disaster and confusion.

The split responsabilites

This one is tricky. One of the most important bits is the work split. Having each developer work on a different component is a normal occurrence. However, you'll often find that on small teams, it is more efficient to work altogether on one thing rather than split each developer on each project. Assigning ownership to part of the projects to one-man team destroys communication between developers, reduces the rate at which they can code, and ends up more often than not with an inability to deliver anything in a sprint: You do half of two tasks instead of finishing one task well.

Ralph often likes to separate his eggs and have each developer in charge of one bit, so he can go and manage each of them individually. It becomes much easier for his excel spreadsheet. This silo development undermines the concept of working as a team, and often comes from Ralph trying to regain control on a project he has no visibility on. As we've seen before, visibility is there, but Ralph refuses to change his ways and embrace the reason behind agile. This is once again a recipe for disaster.

Conclusion

Here's a recap of all the bad habits Ralph has:

• He gives up code quality to try and achieve deliveries
• He's taking away from the business the choice of what gets delivered and the prioritisation of those elements
• He lives in fear of telling the business how things are really going
• He believes it will be easier to deliver now and refactor later (the prototype is your v1)
• He does not care about finding the cause of an issue and focuses only on the work-around
• He can and need to understand and decide how the software is written and who writes what
• He has to give orders (it's often the case that Ralph is the project manager but not your line manager...)
• He doesn't respect the daily meeting
• He wants to fill his 12 month plan and doesn't track the sprint backlog (the tasks on the whiteboard)
• He wants the help of developers to do his 12-month plan, because the tasks are not enough (after all he doesn't track them)
• He believes the plan is the solution to everything
• He believes he's good enough to reinvent a methodology by mixing bits and pieces
• He believes in splitting people to split tasks and regain control (silo development)

This Ralph is a disastrous scenario and very rarely will you encounter people showing all those signs. But if you try to deliver software in this day and age, and you can see in your project management more than a few of those points, I would advise extreme caution, as I believe you'll end up failing on your deliveries and ending up with a unmotivated team.

On a bright note, Ian Cooper told me once something that I kept at heart: everybody tries to do the right thing. Not knowing and fearing are normal things, failing is part of learning, and people willing to evolve overcome those limitations. Great managers are the ones that take the knowledge from their coworkers, embrace that change and adapt to and work with and for the people they manage.

TechEd: Reflection, code generation etc.

I'm sitting in Roy Osherove's session on lightweight code generation. Roy is a formidable speaker, but there's probably a bit too much content and some areas are, I believe, not explained in enough depth to be approachable by neophytes.

Overall, a very good session with a lot of deep content, and a great overview of what is new in .net 2.0. However a lot of the focus is on solving real problems through these technologies, and a good dose of DLR and expression trees discussion would probably have been a great and smooth transition to the rest of what we've seen this whole week.

And Roy is singing I don't want to blog about it. What an extraordinary end to a very good talk!

Technorati Tags: TechEd-Developers, TechEd

TechEd will be over very soon...

I'm not taking the plane until tomorrow, but it's very sad to see everyone starting to leave. The exhibition center is closed. Only one talk remain where I'll go and have a chat with Roy Osherove.

What to do tonight? I've not managed to learn about any parties, maybe tonight will be the first night where I can see Barcelona. Or maybe not.

To be fair, I am tired, my brain is close to exhaustion with all the talks, arguments and chatting that's been going through, and I want to go back to my partner and have a relaxing weekend. Which means suffering the plane tomorrow morning, an experience I'm pretty sure I won't enjoy: I have a ticket for the evening but *need* to arrive in the middle of the afternoon in London.

Oh the fun!

Technorati Tags: TechEd-Developers, TechEd

TechEd: SeaDragon and PhotoSynth

This is a private session I ended up going to because of my craving for orange juice..

PhotoSynth is in Silverlight, but it's not clear which version it will be shipped with. It's a multi scale system, the size of objects is not important. The maximum info needed at any moment is whatever is on the screen. It supports pictures, text or even a fractal.

Another demo is an application of the scaling with an issue of the guardian, zooming out to show al pages and zoom in into a corner to see a very close text that you normally would need a microscope to see on a real printed animation.

This works with the hardware of the macine because the GPU is used. [EDIT: I thought silverlight was only software, or is there something more in the pieline?]

Another demo is a bunch of photos from flicker of Notre Dames, and generate a 3d reconstruction from that picture, and where all the cameras were positioned. You can then do a photo to photo navigation.

It's all about large navigations of photos and 3d, and SeaDragon became Photosynth. The system starts from images, extract features, then those feature get matched, and the matching turns into a 3d scene.

There's some work that ha been done in the summer with NASA. A picture of the shuttle display recrangles that are features: scale, orientation and corners.

Descriptors are informative about what the content is, but minimally informative about resolution, point of view, cropping, etc. If we extract the same features for a number of images, we manage to match them. All the features get in a feature index, look for nearest neighbors, and that gives matches between several pictures. Now that you  know that two points on two pictures match and you do that across pictures, you can find out where the camera would have been and where the point would have been.Each point in the 3d visualization is one of these match features.

The system used to take several days, and we're getting ready to release in the spring community photosynth which will let a single user capture a 3d environment on his own. You start being able to capture a real part of the world. What it will allow is for users to make their own synths and share them, and link them together. When you link it you can navigate seamlessly from one synth to the next. We're going to see overlapping synths.

We've also been exploring public collections.

One of the secrets in photosynth is that it's almost 3d. That Synth was done yesterday after 5 mnutes of crawling. Some of these transitions are less than ideal ones. If you notice, the reason is because all these photos are projected on flat planes. It doesn't correspond to the geometry of that place. It's in between 2d and 3d. Something that a lot of peole have asked us to do real 3d reconstruction. There's been a flurry of research around it, and some of our colleagues have done is fabulous. There's an experimental viewer made oer the summer, with a project with the BBC. It's reconstructed 3d, we're not projecting on flat planes anymore we're projecting onto 3d models reconstructed from these pictures.

Another demo used in Live Labs is very close to the birds eye morphine in virtual earth. In virtual earth there are models of the city, photogrammetry, gps, etc. In this case we don't use any models, they come from the photos.There are still some errors but we're not finished with

We're working on integration with Virtual Earth because it relates to the interplay between top down and bottom up representations.

Technorati Tags: TechEd, TechEd-Developers

TechEd: The irresistible Forces and the Moveable Objects

Pat is a fantastic speaker, as always. However as always, the content can be followed by slides as well as by listening. For some reason, all the rules about not reading your slides and not putting too much content on them, he doesn't follow. And by some speaker magic, he succeeds brilliantly. Goes to show that, to each their own style, as long as you captivate the audience!

The topics remind me a lot of what I've heard from many previous conferences, Microsoft or not: Just imagine when everything you know will have changed so much you won't be able to apply any of what you know now. The big topics are storage, massive parallelisation of work, the death of the hard drive, etc.

What is less ground-breaking is one of the demoed video. It looks terribly similar to a digital home video that was available with the second edition of Bill Gate's book. Interactive TV, interactive Phone and interactive Car. Except where you used to hear windows you now hearthe software + services. Oh dear!

To be fair, the fact that these videos are vision of where Microsoft wants to go means that while marketing memes and corporate memos come and go, the company has kept one way forward and keep on pushing it forward.

But the most important thing to get from the talk is the idea that consistency is not as important as you think. It's a subject a lot of people from Amazon, eBay and other companies that scale massively have learnt and are now starting to promote. It's less important to be right than it is to be consistent.

In the end, I would have to wonder if a talk that apply to a handful of companies worldwide was really what should've been the general highlight of the last day.

Technorati Tags: TechEd-Developers, TechEd

TechEd: What's new in PowerShell V2

[EDIT: I'm thrilled by this presentation, can't wait to see the new bits!]

Jeffrey P. Snover [EDIT: Posted on this on his blog]

Before we start talking about v2, lets' talk about v1. The response has been phenomenal. We had a discussion with the Marketing guys. I explained all these things that happened in the past and it'll be a long time before we reach a million download. Was I wrong! We had more than 1.5 Million downloads in the first year.

We have tools shipping today using PowerShell, exchange, MOM, etc. We have more than 20 partners producing powershell inside. We're now a CEC 2009, which means any microsoft server software has to be  powershell compatible.

Why the success? You can see the tasks for exchange with VBScript, the productivity for powershell is phenomenal. A customer had a 481 lines of script he replaced with one line thanks to WMI and html formatting.

The bad: Remoting is Cmdlet specific. There's no uniformity you can rely upon. Theres a great scripting language but you can't write Cmdlets in PowerShell. Our scripting environment developement is notepad and ps-debug, which is a pretty lame experience. Finally our API does not address common GUI needs. The normal model is admin tool built on top of powershell. The people doing that end up rewriting the same thing so we are doing it.

The big things for v2 are: Universal code execution model, GUI over PowerShell, Production Scripting, and community feedback.

So what is universal code execution model? Any code that runs should be able to run on one or more machines, LAN or WAN, restricted or not restricted environments, foreground and background, in short or long connections, using impersonation, and initiated by user input or by events.

The code execution model: Invoke-Expression works with computer names, the runspace, the throttle, the credentials, the ShellName, and connection information.

If you want to run it in a backgroun, you can use start-psjob, and the parameters are the same

When we talk about remoting, the scenarios we nail are fan-out 1 to many. The goal is to reduce the TCO. Second is Interactive, and i don't want to use telnet. We're not trying to replace telnet. We're not doing the graphics telnet. Next, Many to 1. If you write a service, you dont want to create a new process for each user, that would blow up. I want to listen on a port with WS-Management. Then we want the GUI over CLI.

Mobile object model. I want to run some code over there, I bring the objects back here but they're exchange objects and we don't have the dll. The way we do that is standardizing the model, we take the type, we serialize in a property bag. It's a stable cross-machine cross-time. Get an object, serialize. It takes one of the 22 types, if it's not there you call ToString. You can specify which serialization you want. Those objects we serialize everytime we reach a RunSpace boundary.

As soon as something crosses, we serialize, and when we reach the destination runspace, we deserialize. We store the original type and we store where it came from. I have a bunch of machines, i do a getFreeDiskSpace and you get the objects back.

[EDIT: Demo of using Invoke-expression].

Background jobs: we had an object called psjob, you can provide a runspace to it, or not. If you don't provide everything we'll run it locally. We require the latest version of WS-Management (1.1), this is included in Vista SP1 but there's no download, so you need SP1. We think the background jobs should be in process, but in this CTP it goes through the remoting.
Jobs can contain child jobs that can contian child jobs.

Restricted Runspaces are environents you can host to run potentially hostile requests safely. You get control of which language features are available, which commands can be executed, and there's white listing of commands.

There's a whole new set of hosting APIs. YOu can only issue one request per run-space. RunSpace pooling, which Exchange did, is a pool of runspaces and execute the job against that pool.

There are a new set of hosting APIs that bypass the parser completely. You can provide parameters, commands, to build a pipeline. There's also a typed invoke, and redo invoke.

[EDIT: Demo of the new hosting API.]

Production scripting: We have a grpahical PowerShell, Script Cmdlets, Script Internationalization/Data Language- You go to the web and solve the problem by downloading a script. It's 3 lines of code. Now imagine it's 3000 lines, it's gonna take a while before you put it into production. Well PowerShell can guarantee it's only data. You look at the lines of code that are not data to evaluate if you run. In scripts you want to highlight which parts of a script you don't need to delve into; Debugging; and finally packages and modules.

Graphical PowerShell is done in WPF it seems.

Script Cmdlets supports most everything a .net Cmdlet can do. You add attribute. You also have the $Cmdlet variable that you get in .net as your base class.

The engine does a ton of work for you. You change from the fucntion name to the Cmdlet. You can add a Mandatory attribute, a Position, an Alias, ValidatePattern etc.

The data language is a limited subset of languages, with comparison operatirs, if / elseif / else, well defined convertfrom cmdlets, limited variables, comments and pipelines, literals.

Data msgs { convertfrom-stringdata 'commands' }

then you import the localized data with Import-LocalizedData msgs.

We have a new command called PSBreakPoint. They can be set on lines, line number / column number, functions, variable read/write, or a breakpoint any time this command is called.By default we hit a breakpoint and break. Alternatively, you can provide an Action scriptblock to execute when you hit the breakpoint. We also have get-PSCallStack. It's a placeholder, but it will be great. In V2 it will be an array of stack information, with location, scope etc.

We're making progress in including community feedback, but we are working on the big bits first.

We now have a splatting operator. We also added a -split and a -join. We added a type for ADSISearcher.

 

Technorati Tags: TechEd-Developers, TechEd

No more content for today...

Sadly I've not found conference rooms with power plugs, so no more content for today. I'll probably do a quick update tonight of what was said but it may be fairly short.

Off to see IanG for his WPF talk.

TechEd: Networking in Windows Vista and Windows Server 2008t

.Why do we need to discuss the new network stack? We want faster applications, we want to connect to anyone else, and we want a simpler, direct model without yet another server. For all these reasons we need the next generation TCP/IP.

There are some issues with Vista network issue. The Multimedia Scheduler service.  Mark Russinovitch published an article about it [EDIT: Here's the link to the article about the interaction with networking and multimedia scheduler]. One hopes this will be fixed by SP1.\

The new tcp/ip stack is a huge project for Microsoft becase it's the first-time microsoft re-write since the early 90's. It was an outdated technology. NG TCP/IP ships with Vista and a slightly different in Server 2008.

The major improvements include receive window auto-tuning, Compound TCP for very fast LANs, ECN - a way for routers to let the client knows when it's overloaded to reroute the data, better support for lossy network - RFC 2582, 2883, 3517, 4138, and some great IPv6 network with Neighbour Unreachability Detection.

There are other benefits too:  much simpler API, security enhancements with API filtering and monitoring, support for stack offload, and multiprocessor scalability, which was not possible before because of NDIS 5.1 limitations.

There are also a few conveniences: there's no restart, there's auto-configuration and self-tuning of IPv4, and policy-based QoS. There's for users roaming support in IPv4 and (better) in IPv6, home network support has been simplified, and there's more efficient multicasting.

So far, it may change in the future, but so far we have full resistance to all TCP/IP level DoS attacks, and IPv6 for more security. You also have the concept of routing compartments.

Let me try and make you a bit enthusiastic about IPv6. We are running out of address. Don't believe me, believe these numbers. ARIN announced on 21th May 2007 that we will run out in 2010. RIPE 55 Meeting took place two weeks ago said we have 2-4 years before running out, and Mr Vint Cert said that on BBC news too.

Theoretically, we could live without ip addresses. We could have several levels of NATs but it blocks us from performing p2p networks. NAT makes peer-to-peer very difficult, because of security. We need to find a balance. Seucirty of IPv4 doesn't exist. IPSec is optional, not so in IPv6. IPv6 provides significantly better QoS than IPv4. Routing tables will shrink with IPv6. Mobility does not work in IPv4. Finally device autoconf doesn't work so well. Some technologies has been backported, but it doesn't work very well.

6 benefits of IPv6: Address depletion solved, ent2end p2p communication restored, mobile and roaming connectivity, international mis-allocation, autoconf.

Terminology, we still talk about Hosts, we have LAN segments, Links, Subnets... In IPv4 a subnet is restricted to a few devices within the same network. Any enterprise network has a wardrobe of devices to solve the subnet. You don't have subnets communicating with other subnets communicating thanks to a Cisco router.

We have 128 bits for our addresses. We usually split it with 64 bits for subnet IDs, and 64 bits for interface ID.

IPv6 is expressed as 8 blocks of hexadecimal 16bits components.

A great thing for IPv6 is that there are classes instead of subnet masks: The CIDR notation.

There are 3 classes of address: Unicast, Multicast, Anycast. There are no more broadcast address. IPv4 wasn't designed for networks with very large amount of machines within the same submask. Now it's gone, you'll get a performance boost. Instad of broadcasting, there's a bunch of neighbor networking protocols.

There's a new unspecified address, and there's a new Loopback: ::1. There's the concept of a well known address. DNS Servers are supposed to be FEC0::FFF::1 FEC0::FFFF::2 and FFEC0::FFFF::3. As a developer, be aware that you should be able to accept v6 addresses.

Configuration can use DHCP, where the address is generated. What they don't need to perform anymore is the address generation. The other thing is that addresses expire. That gives us back some efficeincy and clears the address space.

Addresses expire, and one of the key things in IPv6 is that you have multiple address. You have global addresses, one expired, one in the future, some local addresses have local addresses, or several link local addresses. It's the key change: in IPv4, when designed, all the processing was on routers. In IPv6 we change the balance, nodes have enough power to do some networking process. That lets us have autonomous networks, by moving.

Mobility is the coolest thing, it's so simple and powerful it's beautiful. When a device tries to connect, it can't get any neighbors. Then it creates a local address. If the router contributes, you can change global addresses but your home address will make sure data keeps on flowing.

VIsta has what is known as a Dual-Layer architecture, isntead of a Dual-Stack implementation,w hich meant that the ipv6 TCP and UDP was grouped together and completely separate from ipv4.

Interoperability is probably a key issue for anything being introduced. IPv6 was built-in for interoperability: ISATAP, 6to4, 6over4, Teredo, portProxy. Teredo is very important because it bootstraps the world from IPv6 to IPv4. However it is not without problems.

That brings us to peer-to-peer networking. P2P goal is to enable direct communication between applications without relying on centralized  servers.  You could keep commnciating with everyone by keeping connections opened. You need to prevent DoS, Secure with crypto, resolve changin addresses, maintain dynamically changing group of nodes, and communicate with a subnet.

The p2p in vista is based in IPv6. It's installed by default in Windows Vista. At heart you have low-level and high level APIs.

The concept of a cloud is a group of peers that can communciate with each others. There's a global cloud with everybody, and a link local cloud, or several private global clouds.

A peer is a machine runnnign the application that connects to other. It has a Name and a PNRP ID, and a Certified Peer Address.

[EDIT: This is overall a big presentation about clouds, chrod based networks, graphs, etc. I worked for too many years with gnutella / gpulp / jxta to contiue blogging about the presentation. Battery is nearly dead anyway.]

Technorati Tags: TechEd-Developers, TechEd

TechEd: Threat Modeling

As an industry, we're good at looking at code compared to looking at security. You can run all the tools you want, you'll find security bug but you won't uncover any design bugs: they're the hardest to fix.

At the very high level, threat modeling is about identifying threats, mitigate them and move on. A couple of years ago, the threat modeling information we were giving was pretty lousy, you couldn't use it much if you were not a security expert. That's what we've been focusing on over the last few years.

When the Windows Mobile team did their first Threat Model, they had to answer a question: Is a stolen mobile devide still in the threat model. If the bad guy has the device, are you still trying to mitigate? They decided to try and give a fight, and include it.

For Windows Server 2003, we've asked the team to think about an admin browsing on a Domain Controller. If you compromise a DC, you can compromise the whole environment. The question always come to mobile code, and we decided to deactivate all mobile code.

Finally, you need to define who your users are. If your user is an IT guy rather than a mother, they're different people. When you present a mitigation to a user, will they understand it?

You have to model your application. We use Data Flow Diagrams for that. You model data, which is where most of the security issues come from these days.

Most whiteboard architectures are like DFD. [EDIT: Won't reproduce the diagrams, the process is defined on Larry's blog.]

When DFD talk about processes, they're not .exe, they're whatever process data.

You need to define in your DFD trust boundaries. You need to decide if two processes, when they talk together, if they trust each others. In DFD it's explicit.

For Vista we decided to add this functionality to run some processes with lower privileges, like IE runs in protected mode, and there's a trust boundary between IE and the OS. A process boundary may be between the system process and the user, as you don't trust data coming from the user. Finally Kernel transition is also a trust boundary.

Theres several types of DFDs. Context diagrams give a high-level content. In level 0, you dig into more details. I've rarely seen much deeper than Level 1. If you go over, you're probably in analysis paralysis.

One feature that never made into Vista was the Castle project: it was where workgroups meet the domains. In a DC the domain authenticate you. In the home, you have several computers but no domain. If you have 5 computers and no DC, how do you synchronize files, passwords etc across machines. So we came with the idea of a castle, a machine that was responsible for synchronizing the passwords.

Local user ---> Castle service -/-/-> Remote castle

The level 0 DFD contains many services. The shell runs as the user and communicate with the castle service: There's a trust boundary.

The castle service runs as SYSTEM because it has to manipulate the SAM. Config data in the registry is accessed by the Castle service but there's no trust boundary. And finally the machine boundary.

Purists would say that a service should be a complex service and be described in more level, but if it all runs within the same process boundary, having a simple process is good enough.

Each DFD is susceptible to certain kind of threats: STRIDE. [EDIT: again, Larry has the description. Saves me some typing]

Example of repudiation. My brother in law lied when sending a check. He said it was sent, I thought he was lying. I got the proof when the check arrived, because they were all post-marked. You use a disinterested and trusted third party.

Each element in the DFD is subject to attack. How you attack them depends on the element type. You attack data stores differently than you do with processes. How do you protect these elements?

You need to determine threats. You have primary threats and secondary threats.

If you have an external entity (a user for example), you're subject to spoofing. It's subject to repudiation. Data flow and data store can all be tampered. The vast majority of DOS come from processes, rarely against data stores. Finally processes are subject to all of the STRIDE attacks.

Secondary threats come from threat trees. They've been around for a long time. They're a bunch of pre-conditions that can lead to a threat. Unless you're a security expert, you do not know what the pre-conditions are. We made away with threat trees, and work on canonical threats.

For every primary threat, we build the threat tree. At the top of the root you have the prime threat. For example spoofing a user, there are 4 ways of doing it. For the first way, you have several reasons, and several causes which lead you to the leaf nodes in the tree. You can ask very interesting questions about the leaf nodes. [EDIT: Example about user spoofing].

The example of analyzing a bank log-in system. The connection is SSL, the password has to be strong. However the cookie was predictable, as it was i+1 for each new authentication. Suddenly by predicting the cookie you can hijack someone else's session.

We have in the SDL book all the threat trees and all the questions you can ask about the tree nodes.

Every single information disclosure threat can become a privacy issue. If you have an PII you can get a lawsuit. Any data store can be attacked.

Now you need to calculate the risk of every of these threats. We used to use numbers to calculate risks. but they're always wrong. Plus, the more degrees of freedom you have in the calculation, you can easily fudge the numbers so you don't have to fix the bugs. The more numbers, the easiest.

We used to use DREAD. I don't like it. Even David posted a weblog about it.

Calculating risk with heuristics with a simple rules of thumb: we remind you of that with the MSRC bulletin rankings. The problem with numbers is that if the number is under a level, subjectivity will permit you not to fix it, whereas the MSRC you have no way of doing that. You ask is it remote / local, deos it crash the machine, is it a server or a client product. The different levels are all described in the book.

Now you need to mitigate the threats. You have a bunch of options: leave as-is, it's low priority, trying to fix it may delay the product etc; remove the product like we did with castle; remety with technoogy counter measure; warning users, but i don't believe in the prompt, you push the security of the system on the user instead of taking ownership of the problem, its not a good solution if it's your only mitigation.

Each STRIDE threat can be mitigated. Spoofing, you have authentication. The authentication technologies you use are different: Amazon.com uses SSL. They use encryption, authentication and integrity. Tampering, you can have integrity: Digital signatures, hashes, etc. Repudiation threat is mitigated with non-repudiation, strong third parties etc. Information disclosure: You have confidentiality, like ACLs. DOS, you have availability. Elevate of Privilege you have authorization.

[EDIT" Won't take notes on the castle example, so i can enjoy it. Maybe Michael could publish that on his blog as an example!]

Technorati Tags: TechEd, TechEd-Developers

Skipping the first session

Even though I went to bed early, still woke up exhausted, so I'll give a miss to the first session. Such a beautiful day, it's a crime to not have some of those sessions outside.

Technorati Tags: TechEd

TechEd reactions

Here's a few things that went under my radar in the blogosphere about teched:

• Gill gives an overview of his first day, and think Barcelona is dirty at first sight
• Joao talks about the Influencer Community Camps pre-conference. He also talks about the CSD influencers dinner on monday, but I'm afraid I was not invited, neither was I on tuesday for the MVP (as you know, I'm not an MVP :)). But tomorrow I'll attend the MS Influencers Party, Wednesday the country party, and Thursday I'm meeting up with some people from the Windows Live team. So much to do, so little time!
• Sascha talks about the World of Warcraft IDE microsoft will release with their partner as an open source project based on Visual Studio shell that was demo'd during the keynote. Go and check it out. And Ronan let us know he'll be in the Ask The Experts for Team System.
• Andrei talks about the Microsoft Sync announcement, personally the one area I'm the most thrilled about, so thrilled the SDK is already installed and I'm coding away :)

Overall, not much noise so far. Maybe the PDC next year will be louder! More content to follow, and tomorrow pictures and videos!

Technorati Tags: TechEd, TechEd-Developers

TechEd: Security Development Lifecycle: Development Practices

[EDIT: By Michael Howard, definitely a legend in my book!]

Until 8 years ago, i was completely focused on security features. But a small group of us, around 1999, decided to split from the Windows group and focus on how to design software.

Security people suck. They only see things in black and white, you can either have secure or not secure. The real life doesn't work like that. Theres plenty of shades of gray, and security people don't see gray. What they're good at is to tell you when youre wrong, but they won't tell you how to do things.

We added the NX to WindowsXP SP2. The first wave of AMD devices supported the option of not executing data, as most security vulnerabilities get introduced as data that then get executed.

When something is not on by default, people don't turn it on. Also, if it's not on by default it won't get as much testing.

When we activated NX by default, Flash, Acrobat, Java VM don't work. So we had to choose between not running by default and not get used, or be on by default and nothing works. As a security guy I said break the applications. But my manager said that didn't work! So we decided to have NX on by default only for opt-in processes.

The one thing I learnt is that there are shades of gray.

References to books for Writing secure code for Windows Vista and Writing secure code.

So what is the SDL? It's a bunch of requirements and recommendations, and the post release process. It's taking the lifecycle and improve the security by reducing vulnerabilities and reducing the severity of the vulnerabilities.

Lot of people think you can improve the quality by adopting some process, you'll improve the security of the software: there's no evidence! You can't think of general quality stuff, you need to focus on security itself.

The matrix I use is the amount of bulletins post-shipping. Since Vista, we have fixed 70 vulnerabilities in XP, 33 in Vista. In a perfect world it would be 0, but I'm happy with the 50% reduction. IIS6 only had 3 bulletins, sql server had one 4 years ago. Even the analysts start recognizing this.

The SDL is made of two big chucks: Requirements and recommendations (must and should).

Some functionality was invented a long time ago and was fine 20 years ago. But today it doesn't work because of threats. For example CRT (C runtime). You can't write new code that uses the old APIs, 120 functions. You can include banned.h which works with gcc and others, or use VS2005 that puts a warning.

In the SDL, if you put forward a requirement, you need to be prescriptive about what the fix is. For the CRT, we recommend two libraries: Safe CRT and strsafe. VS also automatically replace the banned functions if it knows the destination buffer size.

One of my favourite tool is SAL, Standard Annotation Language. It helps static analysis tools. The shell team decided to annotate the shell headers. He ran the tools we have at microsoft again and found 100 bugs. It lets you annotate your header files with how attributes relate to each others, if they're optional, in, out, etc...

Integer arithmetic issues can be resolved in several ways. First, use the latest C++ compiler, it protects you against the operator::new overflows. It crashes your process but that's better than running code injection vulnerabilities. Second, review your memory allocation: Is there any calculation to calculate a buffer size? Does the bad guy controls one or more operations? Finally, you can use IntSafe (C) and SafeInt (C++).

Next thing is the no weak crypto. A lot of algorythm that were ok a few years ago are not ok anymore. No new code can use MD4 and MD5 (collision of 1 in a million in MD5), SHA1 is showing signs of weakness, so it's banned too. Use SHA2. Can't use DES or triple DES - use AES. RC4 is not allowed because it's a stream cipher, no symetric keys < 128bits, no RSA keys, no weak random number generation, no embedded secrets and be crypto agile: don't define which crypto you use in your code. You should also not have weak ACLs: we found a bug a couple of years ago where a vendor was intalling their device driver in ProgramFiles. They changed the ACL to Everyone full control. Michael's advice: don't touch ACLs. The defaults are very good, don't change them.

We also have requirements about compiler switches. The GS flag adds random numbers to the stack: you need to guess the cookie we've set because we check it, and if it's not what we expect we crash the process. When the blaster worm happened, win2000 and XP were infected. Windows Server 2003 wasn't infected but the coding was there. We detected the buffer overflow at runtime and crashed. It was affected but not infected. For some arguments we copy the data earlier in the stack. In Vista, we compile with the orcas compiler.

Safe exception handler would have prevented CodeRed. The PE header has the address of the exception handler. At runtime the OS checks the exception against the list in the PE header. NX is the No Execute, ad DynamicBase is base address randomization for OS code. Finally, /Strict MIDL generates safer IDLs.

Although tools are incredibly useful, they don't make the code secure. Bad code and bad developers won't be saved by tools. There shoudl be nothing special about security, each developer should include it as part of getting the job done.

In Vista there's an option to detect any heap corruption and kill you on the spot.

The only scenario in which reliability guys have a conflict with security guys is when we bluescreen the machine if the security event log is full.

There are also web-specific guidelines. A lot of people write a lot of rubbish code. Use the Xss library, don't let SQL injection happen. Deny access to the underlying tables, only allow views and stored procedures.

Gadgets are Html / javascript / Silverlight. If your gadget renders untrusted data, you can call anything. Any exploit code can run. The first thing you should do is validate all your incoming data, and code review for insecure use of innerHtml, document.write, and Eval (which is one letter away from evil and there's a reason for that).

Inspect your Gadget from Michale Howard & David Ross.

Technorati Tags: TechEd, TechEd-Developers

TechEd: Metropolis - Interchangeability of Operations

[EDIT: Unbelievable but you have to sneak in the conference rooms to get access to the one power point that's hidden under traps. Hopefully no one will ask me to unplug myself because of Health and Safety but you never know... An IT conference without power plugs, what a great idea!]

I've used the term Metropolis for a set of talks. Look at Cities, Transportation, manufacturing and try and find some trends. From looking at cities we can learn a lot about what will happen in IT.

At a high level, IT shops map to cities. You have evolution that occurs in isolation, and then things get connected, which causes change. Factories and buildings are comparable to applications. Each application is part of an IT shop. Retail maps to business process. It's about tying together deliveries that come from different origins.

Manufacturing goods and structured data and operations are similar. That what we see as data structures and operations are made to fit better together.

I don't believe it's an analogy, it's not helping to think about a subject, it's the same drive behind both technologies.

What are the challenges in SOA? Explicit boundaries; Autonomy - you can replace your service without impacting the other services.

How do you perform actions across these boundaries. What trust do you use? SOAs are behaving the way you see people in business behave in the real world. Its an opening up of the autonomy and explicit boundaries.

Those that know of me know I've been doing transactions for a long time; but I've grown to believe that transactions across boundaries are very fragile. I don't think services as boundary of trusts will engage into transactions with other services. Some say two-phase transactions will happen across services, some say they shouldn't because of trust issue. For this talk, each service will have a transaction but will not hold locks for another service.

When you look at manufacturing, there was a transition from hand-crafted to automated. A machine creates the part. The shape and form was so crude with the machine that you had to manually change them before assembling. Interchangeability was driven by being able to exchange pieces between two broken guns on a battlefield to make one that work.

In computing we're massively stuck because we don't do very well on how to exchange services. The important points that you need to take from this transaction:

• Autonomy
• Agreement is different: Attempt and confirm/cancel
• Interchangeability: One operation is as good as the next. Fullfill my requirement, be it that you use A or B. You execute your operation on a pool of services.
• Semantics: How do you make the request to rearrange and reorder. You need Precision but you don't want the Intricacy.
• Variety: It can be made out of interchangeable of the operations. If you put different goods in WalMart, the combination is astonishing, but they come from a limited number of pools.

Let's see the history of manufacturing. If you go back to the early nineteenth century, you'd order one thing and you'll get one gun. You have a lot of craftsmen, each of them builds a complete item. But each person made a different gun because it was based on their own approach. Completing anything needed fitting. Soft parts were stamped because they couldn't be exchanged.

The American way of manufacturing comes from shortage of labour. They started building machines to do the work. The problem was that the machine created parts were still inaccurate. Because of that, they need adjustment. Fitting was still required.

With the armory system of manufacturing. LeBlanc proposed interchangeable small parts but it was dropped, but it was brought back in America, or at least the idea. By creating Jugs, Gauges, and the precision work John Hall, the price of manufacturing the pieces increased because you had to be picky with the pieces you get. You had interchangeability but the economics didn't work.

From armories to bicycles to sewing machines, all the way to cars with Ford, impacts what we do today. The disassembly plant: the meat arrived in Chicago, and each butcher was responsible for cutting one piece of meat. There was very high throughput. In assembly, Ford moved the parts to the people rather than the people to the parts.

The Ford factory made sure there was no fitting, every part had to be fully designed, custom machines were made, so all the pieces could be interchangeable. The machine was giving the precision and the accuracy. Even the building was shaped to optimize the production. But it became impossible to change the car design as you would now have to knock the building down.

Interchangeability is great but there are other requirements! Which brings us to General Motors. GM gradually introduced the concept of a yearly model, which introduced the requirement of being agile. You suddenly need multi-purpose machine tools. For this you had to be able to re-arrange the components of the machine to create a new piece. Mechanization, interchangeability and rapid changeover to new models.

Let's talk about the American system of transaction processing. You start your activities, be it a human or a machine, and it goes to your database, you calculate ad you return your answer. But you have to do fitting inside the transaction. There's no boundaries when you update two databases within a transaction. Within each transaction you need to fit the work. But it gives you volume.

We need transactional "machine tools". No precision tools: our current tools can be used in different ways. Different programmers get different results. They're wonderful for labour savings, but not to separate their behaviour. Today, we have applications that have external behavior that are similar but the inside is different. If we are to take our applications apart, we need to make the internals of our applications interchangeable.

Services are connected by messaging. You don't know how the operation is executed, but you know the contract of the messages and the contract of the order in which messages are exchanged. Interaction is based on business functionality. The services share operations, not data. Sometimes there's reference data. It's like a department store catalog, you use that data to connect to your business.

What about optimistic concurrency control? Can you do optimistic concurrency work with your bank? No, because there's a security boundary. You need to think about trust.

Autonomy means independent control. My local business logic decides how i change my local data. I decide what changes. If you want a change, ask me to do a business operation.

Long-running work. Services don't share transactions. How do they cooperatively make decisions? how do independent businesses take decisions? There's tentative operations, with reservations, cancelable orders, or confirmation. If you buy a house in the states, as a buyer you give everything to the Escrow company. The bank, the buyer, the seller, everyone puts their trust in the escrow company. However the sell is only a reservation until you know if you will get it. The only guarantee is that you'll get your money back if the deal doesn't work out.

Coordinating n-systems requires that at least n-1 accept uncertainty. You system needs to be in a confused state. In 2-phase commit, the database maintains locks. With cancelable operations, uncertainty is the reservation you have for a hotel room. There's an intrinsic conflict between consistency and availability. Two Phase commit is the anti-availability protocol.

Because operations are cancelable, you can reorder them. Your cancellation is not an undo but an action you receive to cope with the failure. So what are the semantics for cancellation / confirmation?

Cancellation is about coping with the operation not being done. You accept the right to cancel. Confirmation is the guy that has the right to cancel saying that he won't, for example hotel room confirmations in the morning. Airline companies clear up their overbooking once the plain leave.

To make your operation cancelable, you want to reserve its effect until the confirmation. If an operation is unique, you must lock its effect until you cancel it. You need to make a decision between provisioning and overbooking. How do you manage the pool of resources?

One operation in a class is the same as another. If you reserve a king-size room, it could be any of the rooms. Interchangeability comes by increasing commonality.

It's a pain to offer cancelable operations. But you want to do it because your customers need that in order to function correctly. Its annoying that the customer can be fickle but it's the reality of a complex application.

We used to make decisions atomically by committing or rolling back your transactions. You want to now think about how things compose, but for that you need to have each service being part of a long running work.

In the armory system of transaction processing, you have a complete solution when you assemble small services. In manufacturing you need precision. You need to understand the constraints of your operation. We've made components that were multi-purpose. What you need to do is to have the least amount of functionality to make them interchangeable.

Resource-oriented data lives longer but is changed by long-running operations, like bank records. You have activity-oriented data, that gets created when you start doing a job, and retires when the operation is complete, like an order.

These two classes of data have different characteristics.

In both cases, the data is encapsulated within the service, and there's no optimistic concurrency control externally. You mediate access through business logic.

Let's take a Foo operation. First case, the data is isolated. If I change something about someone's order, I only change a bit of data. But for Joe's order, that may modify my resource-oriented data by changing my inventory. Activity oriented data is used for a single long-running activity and won't span several. You can create a tentative operation, it's easy to track down in case the operation is canceled, but nothing is shared so there's no impact. What happens if you share data?

It becomes harder in resource-oriented services. What is interacting with what? They make commitments and these commitments need to be tracked. You have an association between activity data, the reservation, and the resource oriented data, the list of hotel rooms that are available every day.

Specificity is not interchangeability, Variety is the enemy of interchangeability.

There's a lot of work around DSLs. You express the intent of the business domain with a domain language. That's a great foundation to think about the issues that have been raised.

This is heavy-duty reconfigureable manufacturing machines. You don't need much skills to use DSLs if they're constrained enough.

Service operations are like standard parts, whereas tentative operations compose. That lets you do standard software components and use them across several products. All the products look different on the outside but they're all the same inside.

We should learn about that manufacturing part, and do some rambling philosophy.

Atomic transactions are singularities. New challenges happen as you spread work across space and time. Interchangeability helps relax space and time. You can interleave and reorder. Variety means you have lots of options and choices, but you can't keep stocks. Interchangeability needs precision rather than variety.

EDIT: Some form of question about Wal-Mart using Child labour and how it relates to services. Sometimes reality is better than science fiction!

Update 26th November 2007: http://blogs.msdn.com/pathelland/archive/2007/11/25/presentation-of-metropolis-interchangeability-of-operations-at-teched-emea-in-barcelona.aspx

Technorati Tags: TechEd, TechEd-Developers

Even on Keynotes people don't set their fonts properly

I know it's not the aim to get Visual Studio or Blend visible, but for heaven's sake, LET ME SEE WHAT YOU DO and increase your font size. Silly!

Technorati Tags: TechEd

TechEd - Keynote

Just arrived in the room for the keynote. Of course as it usually happens, all the devices I carefully put on charge last night were not charged, which include camera and camcorder. Expect multimedia content tomorrow instead :(

The room features two screens on which bee videos are being broadcasted, as well as live video from two artists doing some real-time tagging. It all sounds very social network. The theme is orange and yellow, very cyber-opera! And for one the wireless network seems to be sustaining the load quite well!

[13:59] S. Somasegar gets on stage.

TechEd gives a chance to interact with some of you. A lot of you have had a crucial role in delivering .net 3.5, through feedback, technology previews, bugs, scenarios you highlighted and taking a bet on those technologies. For this I'd like to take the opportunity of saying thank you.

When I went for dinner on Saturday my colleagues organized, two things made it an experience: the food of course, but also the environment and how it was presented. Alike, Differentiated user experiences are what will drive the next generation of applications.

It's two years ago that we've launched Visual Studio 2005. As of today, there are 1 million professional developers using vs2005, and that's a phenomenal number. As well, we had over 17 million downloads, a mind-blogging response. At the other hand of the spectrum, we wanted to get into the lifecycle tool space. Over a fourth of our developers are using the Team System tools. But our developers are a community. A few years ago, if you asked a question you would get 30 or 40% chance of getting an answer. Today it's 80%.

Microsoft has always been since 1975 a platform company. Every piece of software is always thought about as having a platform component and a user interface component. The best thing we can do is to create a modular and extensible platform. We want to make sure we have the most vibrant developer and partner communities.

My 11yo does not know programming but spend a fair amount of time online, she's on facebook, she wants to customize her home-page, and share what she's creating. She wants to create an online experience even though she's not coding. My wife is an accountant, she doesn't code as part of her day to day work, but to automate some tasks she sometimes code. You have different experiences for different platforms for different users.

That's why we have different tools and different platforms.

As we speak, the product team is putting the final touches to .net Fx 3.5, which is the next big step for productivity. You should be able to use your .net programming when you do client, services, cross-browser web through silverlight. We already have .net 3.0 that we released in Vista with WPF, WCF, WF... AddIns in WPF, WCF protocol additions, designers being integrated in Visual Studio.

But my favorite is Language Integrated query, LINQ. You had the data world, and the programming world, and they were different things. You have to understand the data language, the constraints and understand the semantics to write your application. We said, let's reuse your skills, what if you could use the same query system as a first class citizen for your objects, for your data.

Silverlight is the new platform. We shipped the first version earlier this year. and we'll push further with the version integrating .net

While we finish the framework, we also finish Visual Studio. For the first time we're supporting multi-targeting. For the first time yu can have one toolset and build applications that target 2.0, 3.0 and 3.5. Looking at the feedback people are very happy we're going that path.

If you're building an AJAX platform, it's integrated in VS2008. If you want Vista technologies, .net 3, we have designers. It's also easier to target applications for office 2007. You used to have to buy VSTO and you get all the tools. We got feedback that it was more complicated than it needed to be, so we took the feedback and integrated VSTO in VS2008 Professional.

We have a good story for VS with server, client, office, cross-browser. We've integrated database tools as well.

And finally we released earlier this year Expression Studio to enable a smooth workflow between Expression and Visual studio.

We also want to continue delivering more prescriptive guidance. Patterns & Practices will continue and we added the S+S (Software+Services) solutions, so it's easier for you to build yoru applications.

MSDN used to be a publishing platform where we pushed articles. It was a good start but msdn should be a community platform. The community can come together through forums and be self-sufficient. We now have MSDN Wiki, took our documentation and published it and added a wiki to annotate content and add content. Just like we've translated visual studio in many more languages. Using automated translation we provide a base for content that the community can build on. Were working on the Portuguese documentation.

Currently we're working on MSDN Code Gallery. There's no easy way to share a sample code. MSDN Code gallery will let you do that. We'll release a beta sometime in the next month, and Live in the next year, so you can come and share your code samples Code gallery. I want to turn MSDN into a community platform and we're well on the way.

Today we're making available a technology preview for Microsoft Sync Framework. Take data synchronization to a new level: it's a framework to enable synchronization for your application, for peer2peer and offline capabilities. Take a look at it and give us a feedback. [EDIT: Is that the return of the sync work from WinFS?]

Software+Services Blueprints is the first guidance we release for standard scenarios. I'm a big user of Outlook 2007. My wedding anniversary is coming next month. Would it be nice if there could be an add-in that integrates the wedding date, buy on its own the present based on analyzing my previous spending, on its own without no intervention buy and deliver the present? [EDIT: What the frell?]

The Blueprint is a bunch of prescriptive guidance and building blocks

As we speak today, later this month we'll release the RTM [EDIT: I Wonder where the RC1 went? Or do they think beta 2 was good enough?], and we'll ship soon after that.

Tony Goodhew is working for the Prodct marketing team is going to do a demo of the cool stuff you can do with Visual Studio 2008.

[EDIT: I won't retranscript the demo, I doubt there's much inherent value for my readers in describing someone showing technology that's already available to people].

I hope you had a play with the beta. We've shown you what you can do with it. Let's see a video of what really happened to get Visual Studio 2008.

[EDIT: Video is slightly funny, I'll let to others the pleasure of commenting on it. Let's just say it emphasize the fact that VS2008 was built by VS2008.]

In 2005 we took a bit longer than we wanted to deliver, in hindsight, you did a good choice delivering it. On the way there was a whole bunch of tide overs, and we only managed to do what we did because we had visiblity on our project, where things needed help, more resources or scoping.

We have more than 200 partners devliering mroe than 2000 products. We're making changes to our partner programs. Some of our partners wnat to target other platforms, we're removing the restriction on using our tools to write for other platforms. We're also making the Visual Studio source code to our partners, and debug your add-in. Dan Fernandez will show you some of the products that can be done.

[EDIT: Demo of an open source project for World of Warcraft add-ons for designing worlds, it was built in a few weeks by a few developers. Show demo of LUA editing].

With Express editions we wanted to expand our audience. Popfly is a visual mashup system: as an open beta last motnh, there's been 100k customers.

With Popfly, you can publish your web page to popfly. Popfly explorer and visual studio integration will help with that.

[DEMO: Visual Web Developer and Popfly explorer beta: It's available at http://www.popfly.ms/Overview/Explorer.aspx).

Technorati Tags: TechEd, WOW

Arrived in Barcelona - The goods

Got in Barcelona alright. Had a goo time so far. Big bag of good with yet another unwearable t-shirt. Am I the only skinny person attending these conferences? An XL t-shirt when you wear XS is ludicrous.

Here's a picture with all this content. Look at the size of this tshirt, it's mental! It's probably going to join the rank of numerous ones that I use exclusively for DIY.

The bag is cool though, very good quality, and such an obvious tool to make your colleagues jealous when you go back to the office. I'll keep my Muji though, I like my bags like I like my laptops: slim.

Technorati Tags: Teched-Developers, TechEd-Developers

Getting ready for TechEd

• Configuration of Windows Home Server for remote access - checked
• Full backup of the laptop on DVD - checked
• Spare external hard drive - checked
• International access for the 3g card - checked
• 1080i camcorder - checked
• Digital camera - checked
• Lab machines assigned DNS and RDP - checked
• Online Check-in - checked

Still to do (or why i won't get much sleep)

• Finish the automatic encoding / transfer of Hi-def videos to my media server
• Finish my DV application
• Find some use for all this video equipment (maybe a daily video blog of what I saw....)
• Finish rewriting some Windows Live Writer plugins work for real-time blogging
• Finishing installing the SDKs I may need to have fun coding when there.
• Try to find out who's going and where are the good parties.

May I mention I'm thrilled? :)

Technorati Tags: TechEd

Working with Sql user instances

Just a quick note, as I've not been very active on this blog for the last few weeks. I still have a lot of content in the pipeline so expect a new tips'n'tricks article next week. I've been preparing some stuff to publish at TechEd as well, so if you are going to be around let me know I'll show you some fun stuff.

Also I'll be releasing a tool to help work with sql express user instances and attached database files that I've been working on. Hopefully you'll find this useful.

Stay tuned...

Microsoft and the Magic sweet spot

As I was coding away on the train today, for a few minutes my mind blanked while watching the landscapes Virgin offer to us regular travelers. The did an amazing job at decorating the length of the track, to the point where you'd nearly get fooled into thinking there are actually villages, cows and other things between London and Manchester. As it happens sometimes, I started thinking about my early days as a fresh and learning developer, many many years ago.

Against the normal academic training some IT people receive, I evolved as a developer from the top down. I didn't do University and got no degree in IT. I actually got as far as studying law. I learnt everything by starting to write HTML web sites back in 95, then started JavaScript in 96 and VB in 97, and discovered JScript.net and C# when I got through a friend an early preview of what was to become .net (NGWS anyone?). And I'll admit that while I understand how x86 works and how pointers work, I am absolutely unable to write a single like of C++ or any other esoteric close to the metal languages (I hope esoteric here is not taken as a challenge of C++ usefulness, as I'm sure the gurus amongst you will have understood). And it's only after all these years of coding that I start understanding calling conventions, Win32, etc.

It all got me thinking about how you understand a language and a platform. Until you reach the thorough knowledge that only years of experience will bring you, you often ignore the why question, preferring the answer to the how: you care about how you achieve a result, with the acceptance that you do not understand why it needs to be done in a specific way.

For example, in JavaScript, you'd add a method to an object type by adding it to its prototype property. You know that instance methods and type methods are two different beasts intuitively, because var $cube = new Cube(); $cube.getValue = function () { return 2; } will not work for another Cube you'd create later on. This empirical knowledge is surface magic for however long you'll take to dig deeper, learn and discover what a prototype is and how it relates to objects. At this stage, if things go wrong you'll probably have to learn more things. And it's an open ended confusing thing because both $cube.getValue and $cube.prototype.getValue can exist.

Another example comes from the WPF world.. Whenever you create a custom library of Controls for WPF, you learn about themes and how styles and templates get applied to your controls. You know that the themes being selected for a control are the end of the line customizations you provide, and you know you have to declare an attribute at the assembly level, and we're still very much in the domain of empirical knowledge. But you also know that the fallback when you don't provide themes is a file in the themes folder called general.xaml. You know the how but the why has no answer. It is a decision the framework designers took, and you have no way of changing it or influencing it in any way. This by design rule is crystal magic. The behavior cannot be changed, and once you learn how to do it, the why question is irrelevant. And if it doesn't work, there's no additional knowledge cost, because you don't need to learn anything else.

My point in all that is that these two magical elements have conflicting developer audiences. More Junior developers (and some senior developers I've worked with in the past), let's call them Mortimer, are happy with crystal magic, because it massively reduces the surface of the API you need to learn. More senior developers that have a will to understand (anyone that uses reflector as his main documentation tool would fall in this category), let's call them Einstein, tolerate surface magic because they can crack the surface, dig deeper and know all the intimate details of how the surface is achieved. They don't necessarily like crystal magic because it just works and they have no control over it, but as it's a closed ending, there's nothing that can be done about it, so they move on.

Everything would be for the best if surface magic worked for Mortimer. APIs like WCF provide a very simple surface magic, with only ServiceContract, OperationContract and DataContract concepts to learn. With the tooling support for WCF services in visual studio, the surface is small enough for most developers to start being productive very quickly. The reality strikes as soon as something unexpected happens: passing too big of an object graph, or worse, a cyclic reference within the DataContracts. Suddenly, Mortimer faces understanding notions such as bindings, behaviors, configuration files, and many other concepts that multiplies exponentially the API surface they need to understand before they can fix a problem. You just designed an API that targets Mortimer when everything works, but that requires Einstein to fix if anything goes wrong.  That said, tooling support is appropriate for being productive, which should ensure WCF adoption, so we're pretty close to the sweet point.

WPF on the other hand has a bit of both. Xaml is easy enough for Mortimer: it's understandable and can be fixed by Mortimer, because most of the bindings between DependencyObject and Xaml follows a crystal magic design. The property store will map Xaml and DependencyProperties based on the name, and nothing you want or can do will change this. You also very much pay as you go when you learn Xaml, because you can start with simple object graphs, and discover attached properties at a later time. You can use them without understanding them, and often you'll be able to fix issues by learning a bit about the Binding syntax. On the other hand, it's a very large platform and requires much more time than Mortimer would spend learning a new technology. Areas like Bindings are complex when they don't work, outputting errors to the debug window is confusing when you don't know where to look, the coexistence of logical and visual trees is difficult to grasp for winforms developers, etc.  Tooling support should help, but from what I see in beta 2, I don't believe the sweet spot will be hit with this version of Visual Studio.

And finally PowerShell is a deliciously efficient crystal magic solution. Things work as expected, pipes and formats work by magic, using a defined behavior that will not change. The framework is static enough that people are confident that the knowledge they learn using the script engine will be valid whichever Commandlet they call. The built-in variables work automatically. $_ will always have the same meaning, $$ too.

What Microsoft doesn't have anymore is a completely crystal magic solution for development. ASP was very much crystal magic, ColdFusion was too, back in the CFML days. Early days VB had few extensibility points which let developers grasp the platform more easily. PHP still has a bit of that feel, just like many other frameworks that limit choice and extensibility to increase the pace at which their developers become experts. When most of your developers know 80%+ of your API, you know you're on the right track. Anything under that and you face a problem: You're segmenting your market between Mortimer and Einstein. Your Mortimer has to get more knowledgeable to survive your technology update, your Einstein will just keep doing his expert job. The low end of the Mortimer market is gone to join the ranks of other easier and simpler technologies.

The sweet spot is not to try and provide one API for everyone, but to provide one crystal magic API for the quick and easy, and a completely different solution for Einstein and upgraded Mortimer.

Getting dynamically typed languages with the DLR is going the right direction. But now, this is a call for action to you Microsoft: Provide us with simple tools with a lot of magic to gain back the people gone to LAMP. Give us a forms design tool that maps any control on a form if the form is named after a table and a control is named after a field in that table. Give us a web tool that simply easy work with no mvc, no presenter, and a simple script language to inject a bit of server side code where it makes sense.

And remember that a crystal magic API is one that you *do not extend*. Not everyone needs a framework.

P.S.: This is just my opinion on why we're loosing so many people to php and other simpler technologies. I have a lot of designer friends that just plain refuse to touch any Microsoft technologies, because PHP is so simple. I'm probably not the target audience for any of this. Any resemblance with current or previous coworkers is purely intentional. No guarantees implied or granted. Do not use in a nuclear facility. Etc.

Technorati Tags: rant, .net, frameworks

The tale of the annoying read-only ConfigurationElement

One of the piece of code I disseminate on each project I work on is a lightweight factory builder. The idea is that you declare which interfaces are implemented by which types, or constructed by which builder. It's perfect for replacing all the types you abstract when unit testing, and put the responsibility of the dependency injection on the configuration file.

Here's a small example of how one would declare an interface as having a concrete implementation.

<factories>
  <factory interface-type="CaffeineIT.Blog.IConfigProvider, CaffeineIT.Blog.Demo"
           concrete-type="CaffeineIT.Blog.SettingsConfigProvider, CaffeineIT.Blog.Demo" />
</factories>

Very straightforward. From your code you initialize the type like so.

var myConfig = FactoryActivator.CreateInstance<IConfigProvider>();

The problem when unit testing my configuration sections is that they're read-only by default. That means I need to do some black magic swapping config files around when running my tests.

Well, it's not very well documented, but with the help of reflector, you discover the secret so many have kept to themselves. To make your ConfigurationElement read-write, override the SetReadOnly method and do nothing.

protected override void SetReadOnly()
{
    //do nothing!
}

I don't know about you but I feel like I'm cheating.

Technorati Tags: .net, .config

WPF Tips'n'Tricks #8: Use your code-behind for binding

In the View-ViewModel-DataModel pattern that has been documented a fair bit for WPF, it is often the case that you want to have a testable class that wraps your data to expose an object model that will make binding from Xaml code easier. However, when writing prototyping or simply answering messages on the Msdn forums, as well as for the code examples on my blog, it's often the case that you want to have a sort-of ViewModel that you want to quickly hack together.

The solution is to simply leverage the code-behind of your window! Let's create a simple property on our window code-behind.

    public partial class MainWindow : Window

    {

        public MainWindow()

        {

            InitializeComponent();

        }

 

 

        public bool IsFlagSet

        {

            get { return (bool)GetValue(_isFlagSetProperty); }

            set { SetValue(_isFlagSetProperty, value); }

        }

        public static readonly DependencyProperty _isFlagSetProperty =

            DependencyProperty.Register("IsFlagSet", typeof(bool), typeof(MainWindow), new FrameworkPropertyMetadata(false));

 

    }

And now in the Xaml code you just need to bind the DataContext property to the class itself.

<Window x:Class="CaffeineIT.Blog.XamlCollections.MainWindow"

  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"

  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"

  DataContext="{Binding RelativeSource={RelativeSource Self}}">

    <TextBlock>The flag value is:

        <TextBlock Text="{Binding Path=IsFlagSet}"/>

    </TextBlock>

</Window>

Voila! You can now bind straight to the properties defined on your Window class! But as the complexity of your project increases, you'll want to unit test the ViewModel, and for this you'll have to separate the code in a separate class. The way I choose to do it is to embed the object directly inside the Xaml code. With this approach, you'll have the a ViewModel class. Here I implement it by inheriting from DependencyObject to keep the DP we defined earlier, as it provides support for validate and coerce callbacks, but this is absolutely not necessary. You're more than welcome to implement your property through accessors and INotifyPropertyChanged.

    public class NewWindowViewModel : DependencyObject

    {

        public bool IsFlagSet

        {

            get { return (bool)GetValue(_isFlagSetProperty); }

            set { SetValue(_isFlagSetProperty, value); }

        }

        public static readonly DependencyProperty _isFlagSetProperty =

            DependencyProperty.Register("IsFlagSet", typeof(bool), typeof(NewWindowViewModel), new PropertyMetadata(false));

    }

And now I can declare this object in my Xaml.

<Window x:Class="CaffeineIT.Blog.ViewModel.NewWindow"

  xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"

  xmlns:x="http://schemas.microsoft.com/winfx/2006/xaml"

  xmlns:vm="clr-namespace:CaffeineIT.Blog.ViewModel" DataContext="{DynamicResource ViewModel}">

    <Window.Resources>

        <vm:NewWindowViewModel x:Key="ViewModel" />

    </Window.Resources>

    <TextBlock>The flag value is:

        <TextBlock Text="{Binding Path=IsFlagSet}"/>

    </TextBlock>

</Window>

I find the code very tidy. Through this mechanism, you can also bind properties from your view to the ViewModel. Imagine: View updates ViewModel updates View... You can abstract all the logic of your view in your ViewModel in an oh-so-clean way it makes me salivate every time. You can then leverage commands to encapsulate actions that modifies the ViewModel which update the View. No more tier leakage!

Technorati Tags: wpf, .net, .net3, .net35

.net source code to be released

From ScottGu, the source code for the .net framework libraries is going to be released, and on a source code server. This is thrilling news. Makes me regret spending all this time within WPF internals through reflector! Does that include comments in the source files?