HOW TO FIND WHAT DOESNT EXIST?

[entingz2004]

<MY BIGGEST PROBLEM>



Please consider this:

I got 2000 text files, and each of them supposed to have a [[DONE]] tag at the first row, or at the beginning of the file. But sometimes, I forgot to put that "tag" before saving and already mixed it with other different files that I finished for that day before realizing it. It takes me a lot of time and sweat just to look for that text file, I tried the "find files" command, but I cant find which of the 2K files doesnt contain that tag. I've also tried using the regular expressions but I cant create an exact sequence of expressions or syntax.

The "Find Files" command always check every line of a certain file, I think I can fix this problem if only theres an option on which line of a text file should the command will work. I really find it very hard, because I use about 15-20 different tags, most of the time a text file contains 2 or more tags. The [[DONE]] tag only appear once, at the beginning of the file.

Examples of my tags are:

[[DRAWING]]
[[DIAGRAM]]
[[PHOTOGRAPH]]
[[STAMP]]

As you can see, those tags starts with a [[ (double open brakets) and ended with ]] (double closed brakets).

After the [[DONE]] tag I place a control number:

[[DONE]]GT5622363

(No space after the last ])

It also gives me a headache if the supplied control number is wrong and which supposed to be a sequencial (but it doesnt starts with 0, always starts with a certain combinations of number):

1st file of 2000 files: [[DONE]]GT56200363

2nd file of 2000 files: [[DONE]]GT56200364 (...and so on)

The first two letters are always fixed combinations.

Please help me out with this, I would be very greatful.

HELP ME!

Thank You...

[Bob Hansen]

Just a thought, not fully processed......:

Can you find and bookmark all the lines that do have the tags?

If Yes, then toggle the bookmarks to isolate the lines that are missing the tags. Can you now process those lines, using Replace to insert the necessary tags (may need to use Regular Expressions)?

Not a solution, but I hope this kicked up some new ideas ......

[talleyrand]

Been a while since I've had a fun question to answer. So unless Bob can get his thoughts to gel, here's a solution.

Install Python and you're good to go.

Copy this code and paste into Textpad.
Save it to something like C:\MyDirectory\SomeSubDirectory\kumusta\entingz2004.py
Update base directory's (baseDir) value
Update the file extension (currently *.txt)

C:\>python C:\MyDirectory\SomeSubDirectory\kumusta\entingz2004.py

You can also run it straight out of Texpad. Set the command to wherever you installed Python (probably C:\python23\python.exe)

Code: Select all

import glob
import os


def process(baseDir, keyword):
    """
    Process all files in a directory.  For those that contain our keyword in the
    first line of the file., we will examine them to see if they have the
    correct sequence.  For those that do not have our keyword, we will simply
    record that they do not.

    This is case sensitive and assumes there is at least one line in the file,
    we have sufficient privileges to open and read.
    """
    #create a dictionary
    _invalid = {}
    _sequenceIssues = {}

    _current = 0
    _controlNumber = 0

    #Update this line with the correct file extension
    # i.e. if your files have a .csv extension, change the txt to csv
    #  or if they all start with foo and have multiple extensions, update it to
    #   foo.*
    for _fileName in glob.glob1(baseDir, "*.txt"):

        _fullyQualified = os.path.join(baseDir, _fileName)
        #open a file for reading
        _f = open(_fullyQualified, 'r')
        _line = _f.readline()
        #read first line and see if it contains our keyword
        if (_line.find(keyword) == -1):
            #our keyword did not appear, so record the file name and continue processing
            _invalid[_fullyQualified] = ('does not contain %s ' % keyword)
        else:
            #It does contain the keyword, so strip what's left (and remove the line break)
            try:
                #attempt to make a number out of the rest
                _current = int(_line[len(keyword) + 2:-1])

                #See if the new value is one greater than the last thing we looked at
                if (_controlNumber + 1 <> _current):
                    #number are out of sync
                    _sequenceIssues[_fullyQualified] = 'Should have a control number of %s but instead found %s' \
                        % (str(_controlNumber +1), _current)
                #Set the last number seen to the current number
                _controlNumber = _current
            except ValueError:
                #we were unable to parse the text into a number
                _sequenceIssues[_fullyQualified] = 'Does not appear to have a valid number.  Value = ' \
                    + _line[len(keyword) + 2:-1]

    return _sequenceIssues, _invalid


def main():

    #Update this value with the correct location of the files
    baseDir = r"C:\MyDirectory\SomeSubDirectory\kumusta"

    #Update this if you would like to find a different keyword at the beginging
    keyword = "[[DONE]]"

    #Call our function and assign results
    sequence, invalid = process(baseDir, keyword)

    #iterate through the dictionary and print results to screen

    print "The following files were missing %s in the first line" % keyword
    for key in invalid.keys():
        print key, invalid[key]


    print '\nThe following files had sequence issues'

    #This shows the ones that are not in sequence
    for key in sequence.keys():
        print key, sequence[key]

if __name__ == '__main__':
    main()

[entingz2004]

Kababayan!

Thanks for that...but is there no any other easy way? I mean, it would take me a lot of time talking to my boss just to let him approve me to install that Python because we're not familiar with that software.

I hope its possible to the Textpad alone...

-----

Bob, bookmarks cannot be used for searching...

-----

Any Textpad genius out there???

Oh well, thanks guys...I'll check regularly on this thread, I hope that someone could get some ideas.

[Bob Hansen]

1. Put a unique number at the beginnning of each line.
2. Use Find to Mark ALL line that have code at front
3. Use Copy Other to copy all the "Code lines into a new temp document.
4. Use Delete all Boomarked lines
5. Insert Code in front of all remaining lines
6. Copy original lines from temp file and append to lines just coded
7. Sort lines by beginning number.

May have to juggle some columns for the CODE and the unique numbers., todo the final sorting.,but I think this may work for you.

[robi2106]

I found the unix utility "grep" available as a windows .exe is best for this sort of operation. The function of the utility would be as follows:

[the directory for the files]:\>grep -L "[DONE]" *.*

The "-L" arguments will only display file name that are missing the line of text inside the quotes. This does a surprising amout of work. For example I pipe the result of one grep into the next to chain commands together. For example, a search for all files missing the [DONE] tag that have an id number in them starting with sh2489, searching in all sub directories:

grep -i -d=recurse "sh2489" *.* | grep -L "[DONE]"

Here (for your reference) is the usage statement for grep.exe.

c:\>grep --help
Usage: grep [OPTION]... PATTERN [FILE] ...
Search for PATTERN in each FILE or standard input.

Regexp selection and interpretation:
-E, --extended-regexp PATTERN is an extended regular expression
-F, --fixed-regexp PATTERN is a fixed string separated by newlines
-G, --basic-regexp PATTERN is a basic regular expression
-e, --regexp=PATTERN use PATTERN as a regular expression
-f, --file=FILE obtain PATTERN from FILE
-i, --ignore-case ignore case distinctions
-w, --word-regexp force PATTERN to match only whole words
-x, --line-regexp force PATTERN to match only whole lines

Miscellaneous:
-s, --no-messages suppress error messages
-v, --revert-match select non-matching lines
-V, --version print version information and exit
--help display this help and exit

Output control:
-b, --byte-offset print the byte offset with output lines
-n, --line-number print line number with output lines
-H, --with-filename print the filename for each match
-h, --no-filename suppress the prefixing filename on output
-q, --quiet, --silent suppress all normal output
-a, --text do not suppress binary output
-d, --directories=ACTION how to handle directories
ACTION is 'read', 'recurse', or 'skip'.
-r, --recursive equivalent to --directories=recurse.
-L, --files-without-match only print FILE names containing no match
-l, --files-with-matches only print FILE names containing matches
-c, --count only print a count of matching lines per FILE

Context control:
-B, --before-context=NUM print NUM lines of leading context
-A, --after-context=NUM print NUM lines of trailing context
-C, --context[=NUM] print NUM (default 2) lines of output context
unless overriden by -A or -B
-NUM same as --context=NUM
-U, --binary do not strip CR characters at EOL (MSDOS)
-u, --unix-byte-offsets report offsets as if CRs were not there (MSDOS)

If no -[GEF], then `egrep' assumes -E, `fgrep' -F, else -G.
With no FILE, or when FILE is -, read standard input. If less than
two FILEs given, assume -h. Exit with 0 if matches, with 1 if none.
Exit with 2 if syntax errors or system errors.

Report bugs to <bug-gnu-utils@gnu.org>.

jason

[entingz2004]

Thanks!

Im working on my station, what if the files are stored in the server?...How can I do that?

[robi2106]

Thanks!

Im working on my station, what if the files are stored in the server?...How can I do that?

If both are windows systems you can mount the drive & folder where the files live, or if the server is a different OS that cannot be mounted directly, consider a reference by absolute path; "\\TheServerName\That_Share\the_data_dir\*.*" or something to that affect.

The absolute path reference may require authentication no matter what OS it is running.

Also, the link I provided was for the UNIX source for grep. Just search for "windows grep utility" or "windows grep binary" and you will find what you need.

I use an app specifically for this sort of operation, called WinGrep, but this is not a free application. It is, however, a very very good application for recursive search and replace sort of operations.
http://www.wingrep.com

One variety of this utility that is free to use, and free as in speech, is available below. It is a native Win32bit compiled binary of the Open Source unix grep. I can't remember if this is the exact binary I use.
http://unxutils.sourceforge.net/UnxUtils.zip

Hope that helped.

jason

[qrouton]

For those who run into this problem in the future...

If you want to do this (or something like it) in TextPad I would suggest:

Assuming that these are true:

- The missing tag is the first word of the first line of a document
- The letter "o" is unique to the tag that is missing
- The document window (f11) is open
- POSIX is turned on under Configure -> Preferences -> Editor

1. Open all documents and perform a global replace on all documents changing the newline (\n) character to a unique character like "~". This will make all documents a single line. Save all documents with <ctl><shift>s to clear the asterisk on the file names.

2. Next use the search string:
^\[\[[^o]*\]\]
and replace it (on all documents) with something like
Marker--&

This will mark all changed documents with the asterisk (in the document selector window) making it easy to identify those documents you need to modify.

If the same string is to be inserted in all documents, use that string in place of "Marker--" and be sure to add the
&" as well as the unique character which replaced the newline character in step 1.

3. Make your changes and then perform a global change on all documents to return your newline characters for the unique character you selected for the first step.

Regards,
Kenton

[robi2106]

For those who run into this problem in the future...
If you want to do this (or something like it) in TextPad I would suggest:

A possible problem with this is that you must modify a file in order to find it. Another problem is that large data sets, like hundreds of files, may not easily be viewed in the GUI. You first have to open up the file in order to see if it has the problem. Even if you do a global search for the incorrect phrase with POSIX regular expressions turned on, you still have to open the file first (unless I am missing some other functionality).

When opening up large files, or many files and performing search / replace operations or large files or large numbers of files, TextPad may require huge amounts of system RAM and take prohibitively long. I have performed Search / Replace operations on a dozen files, each ~10Mbyte or so. These search / replace operations sometimes took 15 minutes. Then again I was performing complex searching for repeated patterns and eliminating repeated characters, etc so my experience may not be typical.

jason