*! version 2.3 by Mead Over 20May2021

help for cf2Version 2.3, 20May2021

Compare the variables in two datasets, extension of Stata's cf

Syntax

cf2
varlist
using
filename
,
sortvars(varlist)
[
options
]

Description

cf2 is an extension of Stata's cf command. Like cf, cf2
compares variables in a data file in memory with those in a "using" data file on disc.

Stata's cf command has two deficiencies. First,
when the two data sets have differing numbers of observations, Stata's cf
exits without providing any other information.
Second, Stata's cf fails when the only difference between two data sets
is the way they are sorted.

cf2 first provides a tabulation of the degree to which observations match
in the two datasets. Then, for the subset of observations in the two datasets with matching
values of the sorting variables, cf2 executes Stata's cf on all or a
user-selected subset of the variables in the -master- dataset.
For those variables, cf2 reports discrepancies or "mismatches" between the two datasets.

Like Stata's assert and confirm commands, both the cf command
and this program, cf2, will exit with an error and halt program
execution if there are any discrepancies on compared observations.
However, cf2's continue option prevents this behavior,
enabling execution to terminate normally despite mismatches
of either observations or variables.

Like Stata's cf command, cf2 has no effect on data in memory or on
the data in the using data file.

Options

sortvars(varlist) list of variables which
together uniquely identify all the observations in both the -master-
and the -using- datasets. A required option. See help for isid.

all displays the result of the comparison for each variable in
varlist. Unless all is specified, only the results of the
variables that differ are displayed.

verbose as in cf, the verbose option
gives a detailed listing, by variable, of each observation that differs.
This option also directs cf2 to produce
descriptive statistics of the unmatched observations of the varlist variables and,
if the other conditions specified under
maxrows(integer) are met,
a twoway frequency tabulation of the un-matched observations.

continue allows cf2 to terminate without error
despite unmatched observations or variables. Specifying this option also
assures that cf2's stored results are available.

listpct(integer) Directs cf2 to list the specified percentage
of observations that are present in only one of the two files being compared.
Default is zero, i.e. none of the unmatched observations are listed.
Accepts integer values in the range zero to 100. Does not affect the
listing of observations present in both files but different, which is controlled
by the verbose option.*

maxrows(integer) When observations are present in
one of the two datasets, but not the other, and when there are at least two
sort variables specified by the option sortvars(varlist),
this option directs cf2 to produce a two-way frequency
tabulation of the observations
with a row for each value of the first sort variable
and a column for each value of the second sort variable.
This option requires the user to also specify the option
verbose.
This tabulation is suppressed if the number of unmatched values of
the first sort value is less than the ceiling set
by maxrows(), which defaults to a maximum of 20 rows.*

* Options omitted from the cf2 dialog.

Examples

Setup is similar to that in the help file for cf

. sysuse auto

. drop gear_ratio

. replace mpg = 20 in 1/2

. replace rep78 = 6 in 3

. * The following line differs from the help file for Stata's cf

. drop if strpos(make,"Datsun")

. save mycf, replace

. sysuse auto

Using Stata's cf command
to compare the variables in memory with the variables in
mycf.dta
does not work
when the two data sets have different numbers of
observations. The following command:

. cf _all using mycf

produces the following results and halts execution:

master has 74 obs., using 70
r(9);

Instead using cf2 to compare the variables in memory with the variables in
mycf.dta

. cf2 _all using mycf, sortvars(make)

produces the following results and, when there are mismatches, also halts execution:

Obs |
matching |
between |
-master- & |
-using- | Freq. Percent Cum.

+

MasterOnly | 4 5.41 5.41
Both | 70 94.59 100.00

+

Total | 74 100.00

Executing STATA's -cf- command on only the 70 observations in both files:
-> cf make price mpg rep78 headroom trunk weight length turn displacement gear_ratio foreign using `"mycf"',

mpg: 2 mismatches
rep78: 1 mismatches
gear_ratio: does not exist in using
r(9);

The option verbose gives a detailed listing of the differences
and the option continue allows cf2 to terminate normally
despite differing numbers of observations and mismatches,
storing information in Stata's return space.

. cf2 _all using mycf, sortvars(make) verbose continue

Descriptive statistics on the 4 observation(s) only in the -master- data

Variable | Obs Mean Std. Dev. Min Max

+

make | 0
price | 4 6006.5 1573.115 4589 8129
mpg | 4 25.75 6.291529 21 35
rep78 | 4 4.25 .5 4 5
headroom | 4 2.125 .4787136 1.5 2.5

+

trunk | 4 7.5 1 6 8
weight | 4 2355 302.2692 2020 2750
length | 4 172.25 8.180261 165 184
turn | 4 34.75 2.5 32 38
displacement | 4 117.25 24.985 85 146

+

gear_ratio | 4 3.67 .1639106 3.54 3.89
foreign | 4 1 0 1 1

mpg: 2 mismatches
obs 1. 22 in master; 20 in using
obs 2. 17 in master; 20 in using
rep78: 1 mismatches
obs 3. . in master; 6 in using
gear_ratio: does not exist in using

Because the option continue has allowed cf2 to terminate
without error, subsequently issuing the command
return list reveals the following information
available to the user as stored results:

. return list

scalars:
r(N_mismatchs) = 3
r(N_master_only) = 4
r(N_using_only) = 0
r(N_both) = 70

macros:
r(Nsum) : "3"

Compare the mpg and foreign variables in memory with those
variables in mycf.dta. (Results omitted)

. cf2 mpg foreign using mycf, sortvars(make)

Same as above, but give a detailed listing of the differences. (Results omitted)

. cf2 mpg foreign using mycf, sortvars(make) verbose

Same as above, but list all specified variables, even if there are no
differences. (Results omitted)

. cf2 mpg foreign using mycf, sortvars(make) verbose all

Stored results

When cf2 terminates without error or the option continue is specified,
cf2 stores the following in r():

Macros

r(Nsum) number of differences (string)

Scalars

r(N_mismatchs) number of differences (real)

r(N_master_only) number of observations only in -master- dataset

r(N_using_only) number of observations only in -using- dataset

r(N_both) number of observations in both datasets on which cf is executed

r(listpct) Percentage of unmatched observations listed if different from zero

Author

Mead Over,
Center for Global Development, Washington, DC 20036 USA. Email:
MOver@CGDev.Org if you observe any
problems.