search a string AND NOT another on the same line

[pfranken]

(I'm unfamiliar with the Regular Expressions and need some help.)

I need so search a tree and find all files which have lines that contain the string ".nsf" but that do NOT have the string "/boz/" preceeding the othr string on the same line.
Ideally the search should limit the not part to the preceeding part that is also part of the reference to the same file.

In normal language :
I have a website with 5000+ files. A lot of references in the website are to Lotus Domino files (the ".nsf" extension). The entire Domino tree has been grafted onto a new subdirectory "boz". So all references in the files had to be altered.
I've done that by using search-replace for all subdirs that were in the original root. But I must make sure that I've not missed any. So I need to find all references that are not moved to the new "boz" branch.

All help is appreciated,
Peter
Netherlands

[MudGuard]

This is not possible with regexes.

Use http://home.snafu.de/tilman/xenulink.html to check links in webpages (or any similar tool)

[pfranken]

First of all, since I'm migrating from Windows to Unix I cannot check for broken links due to Case mismatches until after the move.
Second, the new structure is different from the current one and I need to make sure that is going to work properly.
Third, when using a Notes/Domino system with a large nr of databases you get an incredible amount of links which are gerenated by the system on every domino page.
Fourth, I do not know of a link checker that can handle the security of the Domino system.

I already use Xenu and am pleased with it. But I want to try to get my web site as close to 100% ok directly after the move. Trying to fix any error as quickly as possible just after a move is not an option when you have 200+ very active users and a great nr of links to verify.

Maybe someone knows of another tool or good trick that I can use ?

[talleyrand]

Not my best work but I'm swamped. If you have any programming savvy, you can repeat the section in main for each part of your subtree. If I had the time, I'd look up how to do in Python but as I said, I'm hard-pressed for time. As a bonus, this code ought to work on either your Unix(c) or Windows machine. If you need help, and assuming Keith doesn't mind, feel free to post your questions.

Install Python and you're good to go.

Copy this code and paste into Textpad.
Save it to something like C:\pfranken.py
Update the path for baseDir, extension and possibly the keys

C:\>python pfranken.py

You can also run it straight out of Texpad. Set the command to wherever you installed Python (probably C:\python22\python.exe)

Code: Select all

import glob
import os

def findIt(directory, extension, key1, key2):
   """
   Return a dictionary where the key is the filename
      and the value will be a list of the offending lines
   key1 is the string that provides the initial match
   key2 is the string that is required to be there when key1 exists
   """
   #get a listing of all nsf files and directories

   l = [] #an empty list
   d = {} #an empty dictionary

   #Get a listing of all the files in a location matching the given pattern
   files = glob.glob1(directory, extension)

   for fileName in files:
      for currentLine in open(directory + os.sep + fileName, 'r').readlines():
         #currentLine is case sensitive.  To drop the case, uncomment the following line
         #currenLine.lower()
         if (currentLine.find(key1) >= 0):
            #key1 exists on the current line
            # so see if key2 exists as well
            if (currentLine.find(key2) == -1):
               #it doesn't exist so we add it to our list
               l.append(currentLine)

      #iterated through the file
      if (l):
         #there is something in the list, ergo add to our dictionary
         # this will perform a deep copy instead of a shallow
         d[fileName] = l[:]
         #clear the array
         l = []

   return d


def main():

   #do not put on the trailing [back]slash
   baseDir = r"f:\python\test"
   extension = "*.*"
   key1 = r".nsf"
   key2 = r"/boz/"

   #begin repeatable section
   baseDir = r"f:\python\test"

   d = findIt(baseDir, extension, key1, key2)

   if (d):
      print "The following files were found to contain " + key1 + " without containing " + key2

      for fileName in d.keys():
         print "In file " + baseDir + os.sep + fileName + " the following lines are suspect"
         for line in d[fileName]:
            print line
   #end repeatable section

if (__name__ == "__main__"):
   main()

[pfranken]

I'm speechless ........ This is almost exactly what I need.

Since I'm not a perl expert and since I've just seen python for the first time in my life ........ could you change the code to walk the entire tree as seen from the basedir ? There are over 500 directories in the tree, so copy-paste-adapt is not a real option.

I've tried the code and it does the job perfectly. It makes my work a lot easier.

Great to find that there are still people on the internet that provide this level of help.

Thanks,
Peter

[talleyrand]

Grab a 2.3.x or higher version of Python. The ActiveState Python is still based on the 2.2 build but the os.walk call requires the 2.3 module. I'm more comfortable with the AS build for its COM wrappers. Be sure to change your tool to c:\python23\python.exe

Also note that if you uncomment the line to invoke Textpad, you will need to update the location if you install TP into a different place. HTH

Code: Select all

import glob
import os

def findIt(directory, extension, key1, key2):
   """
   Return a dictionary where the key is the filename
      and the value will be a list of the offending lines
   key1 is the string that provides the initial match
   key2 is the string that is required to be there when key1 exists
   """
   #get a listing of all nsf files and directories

   l = [] #an empty list
   d = {} #an empty dictionary

   #Get a listing of all the files in a location matching the given pattern
   files = glob.glob1(directory, extension)

   for fileName in files:
      #23 Oct 03 bjf - track the line number also
      line = 0
      for currentLine in open(directory + os.sep + fileName, 'r').readlines():
         line += 1
         #currentLine is case sensitive.  To drop the case, uncomment the following line
         #currenLine.lower()

         #23 Oct 03 bjf - trimmed the \n from current line
         currentLine = currentLine.rstrip()

         if (currentLine.find(key1) >= 0):
            #key1 exists on the current line
            # so see if key2 exists as well
            if (currentLine.find(key2) == -1):
               #it doesn't exist so we add it to our list
               l.append("Line " + str(line) + "->" + currentLine)

      #iterated through the file
      if (l):
         #there is something in the list, ergo add to our dictionary
         # this will perform a deep copy instead of a shallow
         d[directory + os.sep + fileName] = l[:]
         #clear the array
         l = []

   return d


def main():

   baseDir = r"f:\python\test"
   extension = "*.*"
   key1 = r".nsf"
   key2 = r"/boz/"

   l = []
   for root, dirs, files in os.walk(baseDir):
       l.append(findIt(root, extension, key1, key2))

   print "The following files were found to contain " + key1 + " without containing " + key2
   for d in l:
      for fileName in d.keys():
         #23 Oct 03 bjf
         #Uncomment the following line if you'd like TextPad to open all the files
#         os.system(r"C:\Progra~1\Textpa~1\textpad.exe -q -u " + fileName)

         #Use the following lines if you'd just like a print out of file and line number(s)
         print "In file " + fileName + " the following lines are suspect"
         for line in d[fileName]:
            print line

if (__name__ == "__main__"):
   main()

[pfranken]

Thanks for the help. But may I bother you some more please ?

The code seems unable to handle directory names containing spaces. Is that easily fixed ?

Ex :

Code: Select all

Traceback (most recent call last):
  File "U:\test2.py", line 73, in ?
    main() 
  File "U:\test2.py", line 58, in main
    l.append(findIt(root, extension, key1, key2)) 
  File "U:\test2.py", line 23, in findIt
    for currentLine in open(s, 'r').readlines(): 
IOError: [Errno 13] Permission denied: 't:\\htmlhosted\\boz\\Specials\\Darts kampioenschap 05 juni a.s._files'

Tool completed with exit code 1

Or even directories that have names like "www.google.com".

Thanks,
Peter