How to deal with a stuck or infinitely queued build

Topic(s)
Audience

Everyone

BuildIcon_Large

Team Foundation Build can be a difficult beast, but not usually because of itself. Although people do experience problems when adding solutions that have been around for a while with no build run, it really does do a fantastic job.

On occasion I have seen what I call a “Stuck Build” which is a build that never completes, this tends to only happen on builds setup on large codebases that have never been built before. It also usually also occurs at the point in the build after everything has been built and Team Foundation Build is trying to upload the data it has collected to Team Foundation Server.


Developers should always make sure that any builds they queue complete in a timely fashion.

I queued a build at 3:51. It is still there at 4:21. It seems stuck.

clip_image001

-George Gong, SSW

If you see a Queued build that never completes then there is probably a stuck build somewhere. George;s build has not even run yet, it is still waiting in the queue.

I created a new build for SSW.Website and now it stays in the queue for over 30 minutes.

clip_image002[4]

But the TFS build controller is ok.

clip_image004[4]

clip_image006[4]

I have just installed the VS2010 RTM and Team Explorer. Did I miss something?

-Brite Cheng, SSW

Again, this build is queued, it never started. It is queued as there is another build running. Evan though we now have more than one build server queues will always exist. All it takes is for one more build than we have capacity for to be sent to the controller at the same time. Someone will always have to wait. But they should not have to wait for long. We should probably have something line http://buildmonitor.codeplex.com/ setup so we can quickly look and see what builds are running J but in the mean time you can run the “Build Notifications” application to see what builds are running.

clip_image002
Figure: Where to find the build monitor on the Start menu

Sometimes a build can get Stuck. They usually get stuck when they are uploading assets into TFS. This could be a communication problem or purely a size problem, or a little of both. One thing to look at is how many data collectors you have running during the build and maybe reduce them to get a consistent pass.

You can log onto the server and kill the running build, but make sure it is stuck first… No build should take longer than 30 minutes (Unless it always does).

I would expect any developer to follow the process that comes next:

How to Debug a stuck build:

In this case there is a stuck build of SSW.Framework running:

image

Figure: Build list

I can tell that it is stuck by looking at the running time:

clip_image006
Figure: A very stuck build

After cancelling the build, it did not free up the Build Service (stuck) so I need to restart the build service.

Launch the Admin Console on the build server and restart the build service

clip_image007

Figure: Restart the build service

Then we need to queue another build.

clip_image009

Figure: build is now successful

Done

 

Technorati Tags: ,

Create a conversation around this article

Share on Facebook
Share on Twitter
Share on Linkdin

Read more

Martin Hinshelwood
For a long time now I have been searching for that perfect domain that epitomised the vision, the why, of what I am trying to achieve with my customers and the industry at large. Now I have found it in http://nkdagility.com
Martin Hinshelwood
At the MVP Summit I was appalled by the number of people who asked questions about new features for supporting hierarchical tasks! I shared a disgusted look with Peter Provost and we had a quick (and I mean really quick) conversation that resulted in this post. it really comes down …
Martin Hinshelwood
In my journey of delivering an immersive Product Development Mentor Program over the last eight weeks, a compelling narrative unfolded that beautifully illustrates the essence and true strength of Scrum. This story, rooted in the practical application of Scrum through Minecraft, unveils the depth of adaptability and resilience that Scrum …
Martin Hinshelwood
The Boards in Azure DevOps are a powerful tool that your teams can leverage to enable transparent visualization of the current state of value delivery.  However, the inclusion of Blocked columns can stealthily erode the very foundations of efficiency these boards are meant to uphold. By obfuscating the state of …