Skip to main content
SUBMITTED

Source Row Counts

  • January 2, 2019
  • 1 reply
  • 18 views

Forum|alt.badge.img

We all know that Discovery Hub is extremely reliable at moving rows from one database to another.  However, proving that to other folks can be a challenge.  Also, scripts, validation rules, incremental loading, history rules, and primary keys can sometimes cause a destination and a source table to come "out of sync," which is sometimes intended, sometimes not.  In either case, this desync makes it harder to validate that the load is working as intended. 

Adding an extra count option to the execution package - a count rows in the source table - would a small thing to add, but an extremely powerful validation tool.  This capability would make all kinds of tasks, from basic development to upgrading the software, much easier to validate. Ideally, this count would take into account data selection rules and data source settings, so we'd be able to quickly compare what we expected to get vs. what we actually got. 

Thanks!

1 reply

Forum|alt.badge.img

Agree this is useful; however some things to consider:

* Without proper transaction isolation the source row count and the number of rows retrieved may already be out of sync

* Doing the row count on the source may be a performance hit, this needs to be optional.