The purpose of this module is to help a developer discover
why records are being mysteriously deleted or overwritten.
It does this by logging the key fields and other important
information when a specific datafile is saved or deleted.
This solution is only suitable for character-mode DataFlex.
This solution falls under the category of "desperate
times call for desperate measures". You really
should only do this if you have exhausted every other
avenue available to solve the problem. The good news
is that every time I've used this technique, I've been
able to find the problem as soon as a user caused the
problem to occur.
The overall scheme of this technique is to insert some
code into an FMAC macro that saves and deletes records
The code will log each time a record is created or updated
for a specific file. Because the code is inserted at
the macro level, every source file, module, package,
and include file anywhere in the application that uses
SAVE, SAVERECORD and DELETE on the monitored file will
have the transaction logged.
This solution is compatible with every known technique
for saving and deleting records:
- Save filename
- Save (integer)
- Save indirect_file.recnum
- SaveRecord filename
- SaveRecord (integer)
- SaveRecord indirect_file.recnum
- Delete filename
- Delete (integer)
- Delete indirect_file.recnum
Other macros, such as the ENTER macro, will need to
be slightly modified to log saves in those macros too.
If none of your programs use an ENTER macro, then you
can skip that part.
This solution is designed to require a minimum number
of changes to the source code. Three lines are required
to be inserted at the top of each file. The lines are
simple enough that a DataFlex program could be written
to do this insertion, or if you're handy with PERL,
AWK or your editor's macro language, you could use any
of those techniques to insert those lines too.
WHAT IS THE RESULT?
The log file will contain a list of SAVE and DELETE
transactions that occurred in chronological order. An
example of what the log will look like is:
invoice, S, 01/27/2004, 1315, 4115, jim,
12345, 5, 1500
In the above line, 'invoice' is the name of the program
that saved ('S') the record #1500 on 01/27/2004 at 1:13pm
(1315). The user's name was jim and the save occurred
in the invoice program on line 4115 (see invoice.prn
to correlate this number with an actual source code
line number). The record affected (its key fields) are
12345 and 5.
Later, if users complain that a record with a key 12345-5
is missing, you can search for it in the log file and
find the above record. Now that you know it was saved
to record 1500, you can then search for other transactions
that occurred to record 1500. You might find this line:
invmaint, D, 01/27/2004, 1317, 1277, sally,
12345, 5, 1500
It looks like Sally accidentally or otherwise deleted
that record at 1:17pm (1317) in the invmaint program.
The exact line number that caused the deletion is line
number 1277. You can use the INVMAINT.PRN file to find
out which source line was actually involved with the
deletion. Mystery solved.
Or maybe you'll see:
inv_lkup, S, 01/28/2004, 1015, 5523, joe,
12377, 2, 1500
That line says that the inv_lkup program overwrote
record 1500 with line item 12377-2 at 10:15am the next
day. I don't know why a lookup program would be doing
that, but line 5523 in the INV_LKUP.PRN file is the
place to start trying to figure out how it happened.
The key points here are that once you know exactly
which record stored the now missing line item, you can
then find out what other programs edited that record
and which one deleted or overwrote it.
First, make a backup of your flex.cfl, FMAC. and filelist.cfg
files. The FMAC. file is a text file (source file) that
contains the definitions of all of the DataFlex commands.
The flex.cfl file is the precompiled version of the
Next, make a backup of all your source and *.flx files.
You will be recompiling every program throughout your
application that opens the datafile you want to monitor.
Create a new datafile called TDM_LOG using the TDM_LOG.DEF
file provided in the .zip file.
The schema for the TDM_LOG file is:
After you've created the TDM_LOG file, modify the KEY1
and KEY2 fields to match the data type and length of
the key fields in the file you want to monitor. If the
file you want to monitor has more than 2 key fields,
just add them in using the same pattern as above. If
the file only has one key field, you can just leave
The log file doesn't need any indexes. Because of that,
you don't need to worry about the maximum number of
CHANGING THE FMAC
Open your FMAC file using your favorite editor.
In DataFlex 3.2, there are five commands that save
a record: SAVE, SAVERECORD, API_SAVE, OLD_SAVERECORD
and OLD_SAVE. I'll assume that you aren't using OLD_SAVERECORD
or OLD_SAVE. If you are, then you'll need to modify
those macros as well.
SAVE and SAVERECORD use API_SAVE and API_SAVERECORD
respectively. API_SAVE uses API_SAVERECORD (by using
SAVERECORD as a surrogate). API_SAVERECORD uses a command
called VARIABLE_FILE_COMMAND. That's the macro that
you will actually modify.
In the VARIABLE_FILE_COMMAND macro, add this line at
the top of the macro:
TDM_DELETE_LOG !1 !2
and add this line at the bottom of the macro:
TDM_SAVE_LOG !1 !2
The VARIABLE_FILE_COMMAND macro should look like:
TDM_DELETE_LOG !1 !2
// This supports both Indirect_file and Indirect_file.Recnum (JJT)
#IFSAME !2.RECNUM |FS0,0& |FN0,0& |FD0,0& |FS0,0&.RECNUM
!A  !1 |CI0 !3
#IFCLASS !2 SNDIECWVF
!A  !1 !2 !3
!A  !1 !2.FILE_NUMBER !3
TDM_SAVE_LOG !1 !2
In the ENTER macro, there is one location that needs
to be changed. Find the lines in the ENTER macro that
//[NOT ERR] DELETE !1 // delete records
!A [~|127] $C3 !1.RECNUM
Uncomment the first line and DELETE the second line
so that they look like this:
[NOT ERR] DELETE !1 // delete records
You need to delete the second line because it contains
a !A. The !A will cause the compiler to go bonkers when
it's inside of a comment, so you must delete the line
to prevent any compile-time problems.
Save the FMAC file and pack it using dfpack:
This will generate a new flex.cfl file. You can verify
that it worked by checking the time/date stamp on your
TESTING THIS SOLUTION
A sample program, along with the expected results,
is available in the TESTING subdirectory included in
the .ZIP file that came with this module. Once you've
modified the FMAC and installed the TEST1 datafile into
your filelist.cfg, you should be able to compile and
run the t1.src program without errors using the TDM_LOG.PKG
provided in the distribution .ZIP file. The distribution
TDM_LOG.PKG assumes that TEST1 is installed as datafile
#42. If you decide to install TEST1 someplace else,
you will need to follow the below instructions for changing
CHANGING OTHER MACRO FILES:
If there are custom macros in your project, you might
need to modify some of those files as well. Look for
any macro lines that use $C2 (save) or $C3 (delete)
and change those lines to use the new SAVE and DELETE
GET FILE INFORMATION:
Next, use your favorite programming editor to open
the .FD file for the datafile you want to monitor. If
the datafile's name is LINEITEM, you'll want to open
Look for the definitions of the datafile's key fields.
They will look something like:
#REPLACE LINEITEM.HEADER |FS142,1 #REPLACE LINEITEM.LINENUM
The first line says that LINEITEM.HEADER is defined
as an ASCII type (|FS), that it is file number 142 and
that it's the first field in the datafile.
Once you've located the key field information, open
the TDM_LOG.PKG file.
Search for "CHANGE-THIS-LINE" inside of the
TDM_LOG.PKG file. There are four lines that need to
The first line looks like:
DEFINE TDM_TARGET_FILE FOR 42 // CHANGE-THIS-LINE
Change the 42 in that line to the file number of the
file you want to monitor. Using the above example for
lineitem, you would change 42 to 142.
The next line that needs to be changed looks like:
[TDM_LOG_THIS] MOVE "XXX" TO TDM_LOG.USER
This line records the user's identification to the
log file. If you don't have user IDs or you don't care
about recording that, you can replace "XXX"
with "". A typical way this line would be
changed might look like:
[TDM_LOG_THIS] GET_ENVIRONMENT "USER_ID"
The important point here is that however you get the
user's ID, it must be guarded with the [TDM_LOG_THIS]
indicator to prevent that line from executing when it
doesn't need to execute. If you're not comfortable with
macro writing, then just leave this line alone and don't
worry about trying to get the user ID. It's more important
to find where the problem is occurring than finding
out who to blame. Also, modifying a macro with anything
more complicated than the above two examples can be
a very tricky business unless you really know what you're
The next line that needs to be changed looks like:
[TDM_LOG_THIS] MOVE |FS42,1 TO TDM_LOG.KEY1 // CHANGE-THIS-LINE
In that line, change the |FS42,1 to whatever your first
key field is. The above example is |FS142,1 so you'll
change the line to look like:
[TDM_LOG_THIS] MOVE |FS142,1 TO TDM_LOG.KEY1 // CHANGE-THIS-LINE
Similarly, the next line that needs to be changed looks
[TDM_LOG_THIS] MOVE |FS42,2 TO TDM_LOG.KEY2 // CHANGE-THIS-LINE
and you'll make a similar change of |FS42,2 to |FS142,2.
If you only had one key field, you will need to comment
the above line so that the line will still be there
when you want to monitor a different datafile.
If you have additional key fields, follow the above
pattern for as many key fields as you have.
Finally, the last line to change is:
[TDM_LOG_THIS] MOVE |FN42,0 TO TDM_LOG.REC // CHANGE-THIS-LINE
Change only the file number (42 in this example) to
whatever file number you used in the first change. Using
the above example, you will change this line to:
[TDM_LOG_THIS] MOVE |FN142,0 TO TDM_LOG.REC // CHANGE-THIS-LINE
because 142 is the file number for the lineitem file.
Save the TDM_LOG.PKG file.
MODIFY SOURCE FILES:
At the top of every source file, add these three lines.
These lines can appear before any image definitions,
so you can just blindly add them at the top of every
#REPLACE TDM_PROGRAM_NAME "invoice"
The first line has the name of the program and it's
the only line that will be different for every program.
The other two lines, the USE and OPEN lines, will be
the same in every program.
Recompile all of your programs and test them. As you
save and delete records in the file that's being monitored,
you should see lines being added to the TDM_LOG datafile.
There will be a slight performance penalty when using
this technique, even in programs that are recompiled
and don't even open the file being monitored. This is
because every save needs to check if it's being monitored
Restore the backed-up SRC and FLX files. Restore the
backed-up filelist.cfg, fmac and flex.cfl files too.
Delete the TDM_LOG.* files.
This solution is not compatible with alias files. If
the file has an alias file that is also making saves
and deletes, this solution can be modified to handle
it. I've personally never encountered that problem so
I've never had to code it.
This solution assumes that the program doesn't make
use of clever techniques that set the file-mode of all
opened files to be read-only. In particular, the TDM_LOG
file is opened at the top of the program and it must
be writable whenever a record in the monitored datafile
I hope you find this solution helpful in solving what
would otherwise be an almost impossible problem to find.
If you do successfully find a problem using this technique,
drop me an email and let me know.
to Log All Save and Delete Transactions Article
Curtis Krauskopf is a software
engineer and the president of The Database Managers (www.decompile.com).
He has been writing code professionally for over 25 years. His prior projects
include multiple web e-commerce applications, decompilers
for the DataFlex language, aircraft simulators, an automated Y2K conversion
program for over 3,000,000 compiled DataFlex programs, and inventory control projects.
Curtis has spoken at many domestic and international DataFlex developer conferences
and has been published in FlexLines Online, JavaPro
Users Journal and C++ Builder Developer's Journal.
Other Popular DataFlex topics at The Database Managers,