Replace correctly from character class search

mo · Post by mo » Sat Sep 13, 2003 8:51 pm

Regex seems to be the order of the day...on Bob Hansen's suggestion I got the O'Reily book and am getting into it, and it will no doubt teach me how to solve this problem, but not so far (up to page 33)...

I have a large dictionary set up in TextPad. In it I converted all the "&" characters to "and". However, the way I did this left me with a huge number of words joined together by "and" and it was several days before I realized my mistake so no "undo".

Bit by bit I have been correcting the matter. First by searching for "and" and substituting (one at a time) " and "; then I started noticing some features I could search and replace wholesale (meaning I can go through it by way of "replace next"): if the "and" is preceded by an "a" it's one of the situations. So I can replace by "a and "; this also is true if the "and" is preceded by a number or semicolon or end bracket. "8 and "; but I think I can do this with some exceptions by using "[aeiou0-9\);]and" but I cannot figure out how to replace the correct "a or e or i or o or u or 0 etc..." in the replace (or leave it untouched).

I think I understand about the text in () being saved and replaced by \1 but how do I get the appropriate character saved for the appropriate replace?

something;andsomething
somethingaandsomething
somethingeandsomething
somethingiandsomething
somethingoandsomething
somethinguandsomething
something1andsomething
something2andsomething
something)andsomething

Bob Hansen · Post by **Bob Hansen** » Sat Sep 13, 2003 9:04 pm

Hello mo......

I am confused about what you want to do. It sounds like you wnat to convert existing text into something else. Insert spaces? Delete "and"?

If this is the FROM list:

something;andsomething
somethingaandsomething
somethingeandsomething
somethingiandsomething
somethingoandsomething
somethinguandsomething
something1andsomething
something2andsomething
something)andsomething

Then what will the TO list look like?

s_reynisson · Post by **s_reynisson** » Sat Sep 13, 2003 9:23 pm

My first stab at this. Note the last line does not
produce a match and is left out. Using POSIX.

Code: Select all

find     ([aeiou0-9\);])and
replace  \1.and.                   Note that  .=space

something;andsomething 
somethingaandsomething 
somethingeandsomething 
somethingiandsomething 
somethingoandsomething 
somethinguandsomething 
something1andsomething 
something2andsomething 
something)andsomething
something and something

gives

something; and something 
somethinga and something 
somethinge and something 
somethingi and something 
somethingo and something 
somethingu and something 
something1 and something 
something2 and something 
something) and something
something and something

mo · Post by mo » Sat Sep 13, 2003 9:23 pm

EDIT out...you got it. I need to study the way you set up the saving brackets.

OK...I looked. I'm sure I tried it that way! haha.

second edit: which explains why it didn't work for me...not using POSEX.

well I tried it this way, for non-POSEX and it didn't work:

\([aeiou0-9\);]\)and

This might be a good place to ask why one would/should use POSEX over not and what the difference is.

EDIT 4?

yAGG! i copied and pasted, then modified with the \s...but there was a space after the "and " ... I even saw it as I copied...geting old, loosing it, be dead soon...thanks everyone, been nice knowig ya!

EDIT 5 -- all done! about one a second from the time of the last post.

Thanks very much...this stuff is very interesting! We're getting done in minutes what it would have taken days to do 15 years ago. That must be doing something to "Time" itself.

thanks,

mo

Bob Hansen · Post by **Bob Hansen** » Sat Sep 13, 2003 10:25 pm

Here is a solution with/without POSIX that allows for any single character, not just a predefined list.
=====================
Using POSIX

Search for:
(.)and

Replace with:
\1 and
===================
Not using POSIX:

Search for:
\(.\)and

Replace with:
\1 and
===================

Note that the last line on both examples includes a "space" character after the "and", so the last characters are: "and ".

This will allow for any character in front of "and", and you won't have to edit a list of search characters.

Advantages of POSIX vs. using TextPad default is that POSIX is easier to read. If you look at the help for POSIX you will see that almost all expressions are identical, but without the leading backslash in front of many symbols. This also allows you to cut from other POSIX Regex libraries and paste into TextPad.
==================================

It looks like TextPad default uses Basic Regular Expressions (BREs), and the POSIX syntax uses the Extended Regular Expressions (EREs). But that is not totally correct either. There also seems to be a mixture of flavors of Modern grep, Modern egrep. It looks like the POSIX expressions that are allowed are limited to matching default expressions. Other expressions in the POSIX standard P1003.2 are not supported at this time. Help file says the TextPad expressions are BASED on that standard, not fully compliant with it.

Whew.......it's never easy.

Bob Hansen · Post by **Bob Hansen** » Sat Sep 13, 2003 10:36 pm

For more info on POSIX, check out:
http://www.lynuxworks.com/products/posix/posix.php3

And here is an excerpt from that page:

Certified POSIX conformance exists when conformance is certified by an accredited, independent certification authority. For example, LynxOS has been certified conformant to POSIX 1003.1-1996 by Mindcraft, Inc. and tested against FIPS 151-2 (Federal Information Processing Standard).

POSIX compliance is a less powerful label, and could merely mean that a product provides partial POSIX support. "POSIX compliance" means that documentation is available that shows which POSIX features are supported and which are not.

Be wary of claims like POSIX operating system or 95% POSIX, which do not specify POSIX conformance.
Remember that POSIX compliance does not always mean that all POSIX-defined features are supported.
Always ask for proof of POSIX conformance
The IEEE stipulates that a conformance document must be made available for products which claim POSIX conformance.

Again, for clarification, TextPad makes no claim about being POSIX COMPLIANT or CONFORMANCE. It is "BASED ON" POSIX standards.

mo · Post by mo » Sat Sep 13, 2003 10:42 pm

Thanks Bob,

The reason I wanted to be more specific is that the characters "and" are quite common except following these characters. This way I got about 99% hits...exception viand and squander...couple of foreign words.

My first effort at this was to just search for "and" which is about what your solution does. This is a long dictionary!

mo

Bob Hansen · Post by **Bob Hansen** » Sat Sep 13, 2003 10:54 pm

Too late to go back to the original

and

Search for "&" and

Replace with " and "

Likely to be fewer "&" combined with other characters than "and".

mo · Post by mo » Sat Sep 13, 2003 11:27 pm

yeah...too late...it was days later that I saw what I had done. I was going to add to my "we're saving vast amounts of time" thought the thought that this is also a mistake I couldn't have made say 20-25 years ago...(15 years ago I think it was possible to make this kind of mistake and correct it this way...although the dictionary itself would have used up most of the storage space of most home computers.)

webmasta · Post by **webmasta** » Sun Sep 14, 2003 4:20 am

Something I can never quite wrap my head around.. regex in itself is a brick wall... why do different developers complicate regex by instituting a proprietory set of patterns.. seems like every editor I open is a different way to do it.... like re-inventing the wheel.

Simple solutions are good for dunces.. learn regex the right way and forget all that confusion the editors come with... at least that way theres books and tons of documentation on the web.

It aint easy... but it will work everywhere you go.

webM

Post by **bbadmin** » Sun Sep 14, 2003 9:16 am

TextPad's regular expressions always match the longest possible string, so you might catch most of them with this (POSIX syntax):

Code: Select all

Find what: ([^ ]+)and([^ ]+)
Replace with: \1 and \2

That successfully handles "somethingandviand" and "viandandsomething". To avoid splitting words such as "bandaid", and "handsome", you can require at least 2 characters each side of "and" using "{2,}", as in:

Code: Select all

([^ ]{2,})and([^ ]{2,})

As to why different editors use different syntax, I can only speak for TextPad: when it was first released in 1994, no standard existed, so we chose to base it on Unix extended regular expressions, as used by vi and egrep. We are aware that the industry is moving towards standardising on the ECMAScript RE syntax (aka JavaScript, which is similar to Perl's syntax), so read into that what you like.

Keith MacDonald
Helios Software Solutions

mo · Post by mo » Sun Sep 14, 2003 12:04 pm

Keith,

Thanks for this. (about half way through...break time)...interestingly, the search for at least 2 produces fewer bad hits but is harder to implement because of thefact that quite often it requires several additional clicks on a word to get to the next word, but since one is rolling along clicking to get to the next word (and the number of clicks to get there is inconstant) the tendancy is to click past the next word too. One on either side produces more bad hits but ends up being faster because of fewer mistakes like these.

Got to factor in human error in these things.

Couple of interesting traps:

husbandandwife and handsandfeet and handsand5fingers produce hits which one (well, ok, me) tends to immediately click replace.

Bob Hansen · Post by **Bob Hansen** » Sun Sep 14, 2003 3:11 pm

Don't forget Replace All and/or Mark All then followed with Goto Next Bookmark or Replace Next as possible ways to reduce the number of clicks to get to replace or get to the next matches

mo · Post by mo » Sun Sep 14, 2003 3:49 pm

Hello Bob,

I usually do use those options, but in this case it is a dangerous proposition. "and" is just too much a part of too many things.

I finished this up this morning. Not too bad. The best I could do in terms of speeding things up was to notice unique situations that were repeating and search and "replace all" those.

Thanks to everyone for all your help!