a·gen·tic a·gil·i·ty

Test Track Pro and the case of the missing data

Explains issues with Test Track Pro data queries in TFS Integration, highlighting problems with date filters causing missing records and suggesting hourly filtering as a fix.

Published on
4 minute read
Image
https://nkdagility.com/resources/h4j0yIumt7h
Subscribe

Test Track Pro and the case of the missing data

As you may know, I have been having lots of problems with creating a Test Track Pro Adapter for the TFS Integration Platform. You may have been following my trials and tribulations in trying to get the data through.


It looks as if someone dropped the ball at Seapine when they wrote the query code for TTP. When you query the server to get a list of data the only control you have over the number of records that you get back is to use a “filter”. These filter are pre-setup and I have on on the production TTP system that will give me all of the data that I need to migrate. The problem is that it take around 15 minutes for the query to return the 3000+ records.

So, I first run the query as is and retrieve all of the records and suffer the long load.

Test Track Pro and the case of the missing data

Figure: Initial Query loads the entire data set

Test Track Pro and the case of the missing data

Figure: This produces a large data set, but check the top

Once the initial run is done, there is really no need to load the entire data set, just to find the two or three or sum records that have changed since the last run. I decided, just to be safe, to load the data from the last day, some ~25 records, as that is almost the balance between load and loss.

So lets add an additional filter for data in the last month…

Test Track Pro and the case of the missing data

Figure: Only get data in the last Day

With my recent debugging efforts I was maybe looking more closely at things than I would normally, and I noticed that the most recent changes were not being displayed.

Test Track Pro and the case of the missing data

Figure: Where has my data gone

If you look closely you will see that the top two records have disappeared. What the heck!

If I change the query to be the last 24 hours instead of the last day then the data miraculously appears.

Test Track Pro and the case of the missing data

Figure: Last 24 hours works

In fact I just checked again and the data is now there! Oh, it has just gone midnight in Boston (where the TTP server is) and the data now appears.

You may now ask, “What is the problem? The data will eventually get into the list?”… well, by that time it is too late. Let me explain.

As part of the TFS Integration Platform there is something called a “High Water Mark” and in this case it is a Date. This stores the date of the last run of data so we can check which records to add and which to edit.

 1Me._highWaterMarkDelta.Reload()
 2'------------------------------------------------------
 3Dim context As TtpContext = GetTtpContext()
 4Dim raw As List(Of TtpDefectMigrationItem) = GetTtpRawData(context, viewName)
 5TraceManager.TraceInformation("Located {0} raw updates since {1} in {2} seconds", raw.Count, _highWaterMarkDelta.Value, Now.Subtract(_tstart).TotalSeconds)
 6Dim deltaNew = (From ri In raw Where ri.CreatedOn.CompareTo(_highWaterMarkDelta.Value) > 0).ToList
 7TraceManager.TraceInformation("Located {0} deltas as NEW in {1} seconds", deltaNew.Count, Now.Subtract(_tstart).TotalSeconds)
 8deltaNew = GetDeltaWorkflow(context, deltaNew)
 9TraceManager.TraceInformation("Updated {0} deltas with workflow in {1} seconds", deltaNew.Count, Now.Subtract(_tstart).TotalSeconds)
10Dim changesNew As List(Of ChangeGroup) = GetChangeGroupsForAdds(deltaNew)
11TraceManager.TraceInformation("Created {0} add change groups in {1} seconds", changesNew.Count, Now.Subtract(_tstart).TotalSeconds)
12For Each c In changesNew
13    c.Save()
14Next
15TraceManager.TraceInformation("Saved {0} add change groups in {1} seconds", changesNew.Count, Now.Subtract(_tstart).TotalSeconds)
16Dim deltaEdit = (From ri In raw Where ri.ModifiedOn.CompareTo(_highWaterMarkDelta.Value) > 0 And Not ri.CreatedOn.CompareTo(_highWaterMarkDelta.Value) > 0).ToList
17TraceManager.TraceInformation("Located {0} deltas as EDIT in {1} seconds", deltaEdit.Count, Now.Subtract(_tstart).TotalSeconds)
18deltaEdit = GetDeltaWorkflow(context, deltaEdit)
19TraceManager.TraceInformation("Updated {0} deltas with workflow in {1} seconds", deltaEdit.Count, Now.Subtract(_tstart).TotalSeconds)
20Dim changesEdit As List(Of ChangeGroup) = GetChangeGroupsForEdits(deltaEdit)
21TraceManager.TraceInformation("Created {0} edit change groups in {1} seconds", changesEdit.Count, Now.Subtract(_tstart).TotalSeconds)
22For Each c In changesEdit
23    c.Save()
24Next
25TraceManager.TraceInformation("Saved {0} edit change groups in {1} seconds", changesEdit.Count, Now.Subtract(_tstart).TotalSeconds)
26'------------------------------------------------------
27Me._highWaterMarkDelta.Update(DateTime.Now)
28Me._changeGroupService.PromoteDeltaToPending()

Figure:

As you can see on lines 6 and 16 we do a query based on the High Water Mark to make sure that we only import data once, and that we can tell the difference between add and update.

So, if you are using “last day” in your query then by the time the data is returned in the query then all the dates are before the high water mark. Not good as all of this data gets missed and you will not get any updates after the first run.

The solution is to use the number of “hours” since, rather than days…

Troubleshooting Software Development
Subscribe

Related blog

No related videos found.

Connect with Martin Hinshelwood

If you've made it this far, it's worth connecting with our principal consultant and coach, Martin Hinshelwood, for a 30-minute 'ask me anything' call.

Our Happy Clients​

We partner with businesses across diverse industries, including finance, insurance, healthcare, pharmaceuticals, technology, engineering, transportation, hospitality, entertainment, legal, government, and military sectors.​

Big Data for Humans Logo

Big Data for Humans

Teleplan Logo

Teleplan

Ericson Logo

Ericson

Cognizant Microsoft Business Group (MBG) Logo

Cognizant Microsoft Business Group (MBG)

Slicedbread Logo

Slicedbread

Jack Links Logo

Jack Links

Hubtel Ghana Logo

Hubtel Ghana

Trayport Logo

Trayport

Capita Secure Information Solutions Ltd Logo

Capita Secure Information Solutions Ltd

Graham & Brown Logo

Graham & Brown

MacDonald Humfrey (Automation) Ltd. Logo

MacDonald Humfrey (Automation) Ltd.

SuperControl Logo

SuperControl

Epic Games Logo

Epic Games

Workday Logo

Workday

Deliotte Logo

Deliotte

ProgramUtvikling Logo

ProgramUtvikling

Emerson Process Management Logo

Emerson Process Management

Philips Logo

Philips

New Hampshire Supreme Court Logo

New Hampshire Supreme Court

Department of Work and Pensions (UK) Logo

Department of Work and Pensions (UK)

Washington Department of Transport Logo

Washington Department of Transport

Ghana Police Service Logo

Ghana Police Service

Washington Department of Enterprise Services Logo

Washington Department of Enterprise Services

Royal Air Force Logo

Royal Air Force

Cognizant Microsoft Business Group (MBG) Logo

Cognizant Microsoft Business Group (MBG)

Brandes Investment Partners L.P. Logo

Brandes Investment Partners L.P.

MacDonald Humfrey (Automation) Ltd. Logo

MacDonald Humfrey (Automation) Ltd.

Akaditi Logo

Akaditi

Emerson Process Management Logo

Emerson Process Management

Xceptor - Process and Data Automation Logo

Xceptor - Process and Data Automation