Template:Regex/doc

Purpose

This template helps field the details in the wikitext of any page on the wiki. Normally searches ignore non-alphanumeric characters, but regular expressions (regex) accept all characters, plus metacharacters.

This template acts as a doorway by helping to develop a database query before running it on the wiki, and it does this by way of a search link that can also be used to share such discoveries. This template can also be used to learn the regular expression syntax of this version of Cirrus Search. You could use a bare {{search link}} to do all this, but this template saves a lot of typing (see below), so you only need to focus on entering a regexp.

An important alternative to using this template is performing a search directly with insource:"quotes-delimited arguments". These find wikitext without resorting to the regex searches this template does with insource:/slash-delimited arguments/, (which is a common syntax for regex searches). See § About CirrusSearch below for a better understanding of when this template is not needed. See below for other search tools.

Regular expressions are little computer programs, so it is characteristic of regex searches that they must always be tested to achieve their potential precision and thoroughness. But only a few of these intensive searches are technically able to run at a time against the database. This template minimizes your footprint, and guarantees that you will never run an untested regexp on every namespace in the wiki, even if your default search would let you do that. Use of this template enables the smallest possible footprint by using filters to limit the search domain. The first domain it targets is its own page in an ad hoc sandbox. Once your regexp pattern is honed, you add a search domain, by setting |prefix=.

Parameters

Wonani vinandi: Help:Parameters

|pattern= or {{{1}}}	a regexp search pattern. Pattern is also the first positional parameter.
|prefix= or {{{2}}}	search domain. Prefix accepts a namespace number, or n for the current namespace, or : for mainspace, plus it has the usual prefix: meaning. Defaults to its current page (fullpagename) if a pattern is given alone.
|label= or {{{3}}}	search link label. Label is also a positional parameter.

Procedure

Decide whether you really need a thoroughly precise regexp search, or whether you can find the general wikitext of interest with a plain insource: filter. Examples of the plain insource: search are in § Parameters hastemplate and insource. In those cases, {{search link}} is sufficient, and sandboxing is not being suggested.

Namespace plus pagename equals fullpagename.

The procedure here is an iterative, read-evaluate-modify cycle.

1. Find an existing fullpagename with the wikitext instances you are interested in targeting. Or create one yourself, and save it to the database so the query will find it.
2. Open the wikitext, and enter a |pattern=. Prefix will be added later.
3. Show Preview. See the pattern in the newly created search link.
4. Click on the search link. Note the bold text in each match, the centered, complete query, and note the count off to the right.
5. Go back in your browser. Modify the regexp. Cycle. Or don't go back, you may need to majorly reset at the complete query.
6. Enter a |prefix=. Start with a namespace. At the complete query trim results via the first letter(s) of pagenames tacked onto the namespace's automatically-given colon.

Step 6 is the core provision of this template. Caveat emptor: if you change the target, you'll have to re-save it to the database. If you target it again immediately, you'll want to purge that target. You don't have to ever purge if you just change |pattern=. Note that you can target any single page using prefix:.

Developing regular expressions in an ad hoc sandbox

Regular expressions are little computer programs, so it is characteristic of regex searches that they must be written while studying the target data, and tested to achieve their potential precision and thoroughness. However, only a few of these intensive searches are technically able to run at a time against the database. A sandbox minimizes your footprint, and guarantees that you will never run an untested regexp on every namespace in the wiki, even if your default search would let you do that.

Although a normal search targeting the entire wiki will run quickly, a regexp search should target as few pages as possible by using filters in order to run quickly. A filter is part or whole of a database query. Filters include:

• word(s) or phrase
• intitle:
• incategory:
• hastemplate:
• prefix: (always at the end)
• linksto:
• namespace: (always at the beginning)
• insource:"word1 word2"
• insource:word

Order is not important because the search is optimized by the software before it is run.

To target just one page while experimenting with or developing a regex search, target a fullpagename. From the search box use the filter prefix:fullpagename. From the edit box (of any section of the page with the target data), you can always just write prefix:{{FULLPAGENAME}} and it will "expand" for you to the fullpagename. Although you can edit a history page, technically a "history page" is not a page (in the database), and so {{FULLPAGENAME}} there will point to the database version (not its own rendering). For the same reason, you cannot search for the wikitext on a page that is not already saved (to the database), although you can certainly change the search parameters again and again with no need to save them.

Fullpagename is namespace:pagename. Knowing this you can adjust your Prefix parameter. Although prefix can filter down to one page, it can filter up to a namespace, and it also accepts the beginning letter(s) of set of pagenames if you want to reduce the namespace search domain.

Regex sandboxing uses an ad hoc sandbox made by editing any page containing the target data, and using it as a "sandbox" (not editing it to save it). It then develops by using adding a search link that includes insource:/regexp/, with the filter prefix:{{FULLPAGENAME}} alongside.

Use of a sandbox enables the smallest possible footprint by using filters to limit the search domain. Once your regexp pattern is honed, you increase the search domain. A regex search is best run with filters, not alone even if it is a polished rexexp.

Sandboxing procedure

Rather than use the search box, where entering an equals sign and a pipe character, and "quotes around phrases" is a straightforward matter, it is still easiest to use a regex-based search-link template — {{regex}} or {{tlusage}} — on the page with sample data, because then you can focus on the target data there and on writing the regexp pattern. It is easier, that is, if you already understand how templates "escape" the pipe character and the equals sign. See Help:Template#Parameters for other important details.

The procedure here is an iterative, read-evaluate-modify cycle. Regex development requires that you study the target data while writing and rewriting its pattern.

1. Navigate to a page with the wikitext instances you are interested in mining. Or create one yourself, and save it to the database so the query will find it.
2. Open the wikitext, and enter a {{regex}} or {{tlusage}}.
3. Show preview, and activate the search link. On the search results page, note the bold text in each match.
4. Go back in your browser. Modify the regexp, and cycle until done. (Or don't go back, you may want to modify the query at the search box.)
5. Expand the search domain, and test the accuracy of those results. You can trim or expand the number of the results using prefix:.

Caveat emptor: if you change the target for an immediate retesting, you'll have to save and purge, but not if you just change the regexp.

Examples

As an ad hoc sandbox, you can show the wikitext of a section like this, (already saved in the database), modify some of the patterns in the regex-search-link template calls on this page, do a Show Preview, and see what matches when you click on the newly formed regex search-link, all quite safely, and without changing a thing in the database.

The template calls that produce "1 ft/s, 2 sq ft, 3 m/s, 4 m*s-2, 5 ft.s-2, 6 °C/J, and 7 J/C" appear in the wikitext of this section like this:

1. {{val|1|ul=ft/s|fmt = commas}}
2. {{val|2|u=ft2}}
3. {{val|3|u=m/s| fmt =commas }}
4. {{val|4|u=m*s-2}}
5. {{val|5|u=ft.s-2}}
6. {{val|6|u=C/J}}
7. {{val|7|ul=J/C}}

Note how the above targets are |numbered|, then click on the links below.

Query	Search link	Answer
Q1 Using {{search link}}, does this page employ template Val ?	{{sl|hastemplate: Val}} → hastemplate: Val	A. No, because this pagename is in Help not Article space.(Search link default). 1300 search results.
Q2 Using {{search link}} responsibly, does this page use Val's fmt parameter?	{{sl|insource:/\{[Vv]al\{{!}}[^}]*fmt/ prefix:{{FULLPAGENAME}}}} → insource:/\{[Vv]al\|[^}]*fmt/ prefix:Template:Regex/doc	A2.1. Look for 1 and 3 in the search results in bold text. (Adds an appropriate filter.)
Using {{regex}} instead...	{{slre|\{[Vv]al\{{!}}[^}]*fmt}} → insource:/\{[Vv]al\|[^}]*fmt/ prefix:Template:Regex/doc	A2.2 Less typing than {{search link}}.
Using {{template usage}} instead...	{{tlre|Val|pattern=fmt}} → Testing fmt on this page	A2.3 Easiest for templates.
Q3. Who uses u=ft OR ul=ft? (one-letter differs)	{{regex|ul?=ft}} → insource:/ul?=ft/ prefix:Template:Regex/doc	A. Look for 1, 2, and 5 in bold text.
Using {{template usage}}...	{{tlre|val|pattern = ul?=ft}} → Testing ul?=ft on this page	Finds same pattern, but only inside a Val template.
Q4. AND of these, who also uses fmt=commas after that?	{{slre|ul?=ft.*commas}} → insource:/regexp/ prefix:Template:Regex/doc	A. No context shown, but article title is shown. A half a Bug?
Who has one space before the word "commas"?	{{slre|. commas}} → insource:/. commas/ prefix:Template:Regex/doc	A. 1 but not 2.
Q5. Who uses either u or ul with "ft" OR uses "fmt=commas".	{{slre|(ul? *= *ft{{!}}fmt *= *commas)}} → insource:/regexp/ prefix:Template:Regex/doc	A. 1, 2, 3, and 5. (The pattern matches all possible spacing.)
Q6. Who uses ft or m, in |u= or |ul=?	{{slre|ul? *{{=}} *(ft{{!}}m)}} → insource:/ul? *= *(ft|m)/ prefix:Template:Regex/doc	A. 1, 2, 3, 4, and 5. Used {{!}} for the alternation metacharacter. Used {{=}}. (Could have used named 1 = or nicely named pattern = .)
Q7. Who uses . or * in the unit code?	{{tlre|val|pattern = u *= *(\.{{!}}\*)/}} → Testing u *= *(\.|\*)/ on this page	A. 4 and 5.
Who uses a pipe?	{{regex|\|}} → insource:/\/ prefix:Template:Regex/doc	All of them
Q8. Who uses / or - within the |u= or |ul= paramter?	{{tlre|val|ul? *= *[^{{!}}}]+(\/{{!}}-)}} → Testing ul? *= *[^|}]+(\/|-) on this page	A. 1,3,4,5,6 and 7.
Q9. Where is Val used in the template namespace for numbers only, (no u, ul, up, or upl parameters).	{{tlre|val|pattern = ~(u[lp].)|prefix = 10}} → hastemplate:"val" insource:/\{\{ *[Vv]al *\|[^}]*~(u[lp].)/ prefix:Template:	A. In the 30 or so templates listed.
Q10. Which articles use {{Convert}}'s and(-) option?	{{tlre|convert|pattern=and\(-\)| prefix=0}} → hastemplate:"convert" insource:/\{\{ *[Cc]onvert *\|[^}]*and\(-\)/ prefix::	A Coast Range Arc and Skipjack shad

In Q2, notice how the MediaWiki software ignores the spaces around parameters, but how in Q4 the same MediaWiki software processes the spaces inside parameters. Q2 might have been solved with a plain insource:val fmt search because "fmt" and "val" are whole words, and fmt is rarely seen apart from inside Val. How about hastemplate:val insource:fmt?

Indexed search

First, all pages are scanned by the search engine. The entire wiki is treated as one "full text" kept in a separate database built just for search indexes. It's like the index in a book, but practically every word and every number is indexed to every page.

Since each word in the prebuilt search index already points to the pages that contain it, most any word you search for, is actually a single record lookup in that index. (This is also true for phrases to a certain extent.)

There are separate indexes kept updated for the

• titles
• visual content
• wikitext
• templates

All the words each template outputs are indexed to the all pages onto which they are transcluded. In other words, any text transcluded by a template is indexed to its target page. "Index searches" take basically no time to execute. They are cheap and plentiful.

Preparing and maintaining the search indexes is done in the background in near real time. As soon as you save the page, a few seconds later you can search for the changes you just made. For templates that are transcluded onto many many pages, the propagation of those changes to all the pages index entries might take a minute.

Indexes are based on alphanumeric characters; they store no information on non-alphanumeric characters. Although an indexed search can ask for punctuation, brackets, math and other symbolic characters of the keyboard, these are ignored without warning. Now you understand why search indexes are so fast.

Each regex search needs an indexed search filter to provide them a search domain under 10,000 pages.

Indexed search

A basic indexed search

• searches only article space. That is the default.
• matches only letters and numbers. This is usually not a problem.
• works basically the same way for all public search engines. You can usually find the information you are looking for near the top of the search results by relying on page ranking software.
• lands a lot of search results. You rely heavily on page ranking rules. You then refine search results based on the topmost pages. This is done with the not filter, signified by a minus sign attached to the front of the unwanted word to filter out page-hit noise you could not have predicted. This is the first thing you learn.
• is an "aggressive matcher" including as many pages as it can by matching all forms of each word you enter.

Basically, why would anyone ever want to learn "how to search", since it is just key words, and these are obviously known?

An advanced index search

• targets specific pages, instead of seeking general information.
• doesn't need page ranking at all and cannot accept myriad results.
• Cares about the quantity of page-hits shown on the right hand side of the search results page.

Regular expressions

Regular expressions is a special but small language for specifying sequences of characters that define a search pattern. A regular expression is referred to as a "regex" for short.

A regex search actually scours each page in the search domain character-by-character. By contrast, an indexed search actually queries a few records from a database separately maintained from the wiki database, and provides nearly instant results. So when using an insource:// (a regexp of any kind), you should consider creating the other search terms that will limit the regex search domain as much as possible. There are many search terms that use an index and so instantly provide a more refined search domain for the /regexp/. In order of general effectiveness:

• insource:"" with quotation marks, duplicating the regexp except without the slashes or escape characters, is ideal.
• intitle, incategory, and linksto are excellent filters.
• hastemplate: is a very good filter.
• "word1 word2 word3", with or without the quotation marks, are good.
• namespace: is an advanced filter, but practically useless for regex, except that it may enable a slow regexp search to complete a long life.

The prefix operator is especially useful with a {{FULLPAGENAME}} in a search template, a search link, or an input box, because it automatically searches any subdirectories. To develop a new regexp, or refine a complex regexp, use prefix:{{FULLPAGENAME}} on a page with a sample of the target data.

Search terms that do not increase the efficiency of a regexp search are the page-weighting operators: morelike, boost-template, and prefer-recent. The regex search main concern is to first limit the search domain with an indexed search employing filters, not such "search engine queries". Then it actually searches every page character by character. It examines each page of a narrowly defined search domain.

A basic regex search

• can pattern any character string exactly using a regexp.
• excludes as many pages as it can.
• quotes each regexp in double quotes to turn off metacharacters.

An advanced regex search

• uses metacharacters.
• benefits by being developed in a sandbox.

The regexp can be a thousand words matching every character literally, or a few symbols of a regex metacharacter language, or any combination of the two. It can match any character from any keyboard.

Regex thus have the power to produce exactitude, but are slow (expensive), and come with the responsibility to add filters to increase speed (reduce costs).

Developing a regex search virtually always requires trial and error, an iterative development process supported by {{regex}} and {{template usage}}. The easiest filters to add are a namespace, or a prefix, or a copy of the regex without the slashes removed. Such filters all use an index to search, and by doing so are much faster. This is one of the first things we learn about regex searches: accompany them by filters.

Search domain size

• are voluntary, not automatic
• protect accessibility to regex
• sustain the open use of the most advanced search feature possible
• help avoid an HTML timeout, which will kill a search
• create {{regex}} developmental sandboxes
• are considerate, taking only the processing power they need

Running a bare regex can't hurt Wikipedia:performance, but without applying basic search techniques to them, regex searches can limit other regex searchers, and become an issue of contention.

all: insource "question sublinks" How many pages on the wiki is that? OK. sublinks]]?"/

This covers enough of the regular expressions to get started answering questions about wikitext contents on the wiki. Regex are about using meta characters to create patterns that match any literal characters. The pattern you give will match a target, character by character. To make some positions match with multiple possibilities, metacharacters are needed, and they are from the same keyboard characters that are also in the wikitext.

Metacharacters

The left curly bracket is a metacharacter, and so the regexp pattern given must "escape" any opening curly bracket \{ in the target "{" intending to match a template in the wikitext. All target text (all wikitext) is literal text, but we can backslash "escape" the regex metacharacters \. \? \+ \* \| \{ \[ \] \( \) \" \\ \# \@ \< \~ when we refer to them as literal characters in the wikitext we are interested in mining. Search will ignore the backslash wherever it is meaningless or unnecessary: \n matches n, and so on. So although you don't need to backslash escape & or > or }, it is safe to do so. An unnecessary backslash will not cause your pattern to fail, but what will is using certain characters literally— [ ] . * + ? | { ( ) " \ # @ < ~ .

• [0-9] will match any digit, [a-y] any lowercase letter except z, [zZ] any z, (and so on). So square brackets mean "character class".
• Dot . will match a newline, or any character in the targeted position.

The number of sequential digits or characters these symbols match is expressed by following it with a quantifying metacharacter:

• * means zero or more
• + means one or more
• ? means zero or one

of the character it follows after. The number of times it matches can also be given in a range, a{2} a{2,} a{2,5} matches exactly 2, 2 or more, or 2–5 a's. So curly brackets mean "quantifier".

• The parenthesis are a grouping mechanism, so we can quantify more than just the previous character, and so we can make boundaries for a set of alternative matches. (See alternation below.)
• The quotation marks are an escape mechanism, like square brackets or the backslash.
• The angle brackets stand for numerals, not digits. Say <5-799>, to match 5–799, in one to three positions. Compare this with the alternative: [0-9]{1,3} could match ones, tens, or thousands as, 0-999 or 00-999 or 000-999.
• Tilde ~ looks ahead and negates the next character. In other words, if the pattern matches in this position, then un-match it if the next character is ~character.

It is not safe to search for a lone @ because that single metacharacter matches literally everything; you can use \@ to find all pages that use an "at" symbol.

Similarly find all pages that use the number zero, Search returns an error to search for a lone 0; use one of the three escape mechanisms for 0 or @.

• "0"
• \0
• [0]

or find a larger pattern around the zero you seek. Although zero is not a metacharacter, these escape mechanisms work.

The rest of wiki regex is pretty straightforward. Characters stand for themselves unless they are metacharacters. If they are metacharacters they are escaped if outside of a character class. For this template, it is necessary to enter the pipe character using \{{!}} to find a literal pipe character in the wikitext.

Character classes

A character class means "literal characters", plural. It means "literal", and so normally you don't have to escape a metacharacter character in a character class; they're already square-brackets escaped. The /slash delimiters/ mean we must of course escape any slash character, even inside a character class. No other character in a character class except slash always needs escaping; but because ] and - have special meaning (metacharacter) to a character class, they must be escaped sometimes: those two are also literal (escaped) metacharacters if they are the first character, but otherwise they must be also, like dash, be escaped: only backslash-escape works as the escape mechanism in a character class.

A character class can serve to escape metacharacters, so [-|*\/.{\]] or []|*\/.{\-] means "either a dash OR pipe OR star OR slash OR dot OR left curly bracket or a right square bracket". So [][.?+*|\/{}()\-]" or [-[.?+*|\/{}()\]]" works to find all the metacharacters in the wikitext, all of them except the backslash. Neither [\] nor [\\] allows us to OR a literal backslash. To OR a backslash character, there's alternation with the pattern \\ to handle that case. (See below.)

A character class understands the "inverse" of itself, [^abc] is "not a or b or c". A character class stands for a single character in a targeted position, so it's not really an inverse of a set, but rather a NOT of a character.

Note that constructs such as \d (digit) or \a (alphabetic), used in some other regex implementations, are not accepted.

Workarounds for some character classes

While character classes \n, \s, \S are not supported, in case of an acute need to use them in a regular expression, you may use these workarounds:

PCRE	MediaWiki	Description
\n	[^ -􏿽]	A newline (also can find a tabulation character)
[^\n\t]	[ -􏿽]	Any character except for a newline and tabulation
\s	[^!-􏿽]	A white space character: space, newline, or tabulation
\S	[!-􏿽]	Any character except for white space characters (not a space, not a newline, and not a tabulation)

In these ranges, " " (space) is used as the character immediately following control characters, "!" – the character immediately following it, and "􏿽" as the U+10FFFD character, which is the last allowed character in Unicode. Thus, the range from " " to "􏿽" includes all characters except for control ones, of which articles may contain newlines and tabs, while the range from "!" to "􏿽" includes all characters except for control ones and the space.

Alternation

Finally, alternation is a class of regex that contains alternative possibilities for a match, say an AA or a BB, or a CC:

• "AA" OR "BB" OR "CC" to Word search an entire page
• AA|BB|CC to regexp search a two-character position
• (AA|BB|CC) where used within a larger regexp because an alternation finds the longest pattern, and so the parentheses define that boundary, but it's a boundary you don't have to make if an alternation is the entire regexp pattern.

Notes

Alternation

Finally, alternation is a class of regex that contains alternative possibilities for a match, say an AA or a BB, or a CC:

• "AA" OR "BB" OR "CC" to Word search an entire page
• AA|BB|CC to regexp search a two-character position
• (AA|BB|CC) where used within a larger regexp because an alternation finds the longest pattern, and so the parentheses define that boundary, but it's a boundary you don't have to make if an alternation is the entire regexp pattern.

For this template, we need to replace the pipe character with {{!}} so that the "pipe" for the regexp won't confuse this template (or any other template). We need the parentheses at times because an alternation finds the longest pattern, and so the parentheses define that boundary, but it's a boundary you don't have to make if an alternation is the entire regexp patter.

Regexp searches are restricted on the server, so this template reduces the regex search footprint by using the prefix: filter every time, restricting the search domain to a namespace at most. The prefix: parameter can further filter a namespace by specifying pagenames that start with a given letter(s).

Templates for searching Wikipedia

Search links

A search link stores a query in a link that takes you to live search results for that stored search. They're found on user pages and talk pages. Use one to bring the full feature set of MediaWiki Search, or features of external search engines, to bear on users unfamiliar with their search parameters.

One type of search link is a wikilink with all the capabilities of Search (search box), and with standard wikilink syntax: [[Special:Search/query| label]]. So this search link will (1) navigate: [[Special:search/Wales]] → Special:search/Wales or (2) search: [[Special:search/~Wales | search/~Wales]] → search/~Wales if you prefix a ~ tilde character.

All other search links are made from a template that will build a URL instead of wikilink. A URL can for example can call off-site search engines to search Wikipedia.

• {{Search link}} offers all the capabilities of Searching (search box), plus extra (URL) parameters for combinations of namespaces, and where you can escape the 20-results-per page-limitation, shareable: {{search link | et al | ''label'' | ns4 | ns5 | limit = 123}} → label.
• {{Regex}} – develop an advanced regex search. {{regex | \<--.*--> | label = Articles with comments missing the ! bang character | prefix=0}} → Articles with comments missing the ! bang character
• {{Template usage}} – develop a template regex search, and pinpoint specific template-call details. {{Template usage | Convert | \{{!}}C\{{!}}F | 0 | Articles that convert Celsius to Fahrenheit}} → Articles that convert Celsius to Fahrenheit
• {{ShortSearch}} – create three search links: {{ShortSearch | system operations research}} → WP GWP G (search Wikipedia, "Google" Wikipedia, and Google search)
• {{wpsearch}} – create five search links: {{wpsearch|collaborative search}} → collaborative search – Wikipedia search | Google search | Bing search | DuckDuckGo search | Yahoo search
• {{Wikidata search link}} – creates a Wikidata search link for descriptions, entities, items, properties, etc. → https://www.wikidata.org/w/index.php?search=Universe&title=Special:Search&fulltext=1

Search boxes

• {{Search box}} – Simple search box with choice of button below or to the right
• {{Search prefixes}} – Multiple pages' subpages are searched at once.
• {{Archive banner}} – For searching archives. It is of banner-style, like many other archive templates.

Search boxes are made by <inputbox> tags. See mw:Extension:InputBox.

Page title searches

• {{Canned search}} – Link to automated search results for a given term
• {{In title}} – Search for pages whose name contains given words
• {{Look from}} – Search for pages whose name begins with a given word

For searches with exact matches, exact in upper and lower cases, or in punctuation marks, see Help:Searching § grep.

Other Wikipedia editor help

• {{Linksearch}} – Searches for external links matching a URL
• {{dabsearch|term}} – External tool to find page titles containing a (term) in parentheses; useful for Wikipedia:disambiguation study
• {{Help desk searches}} – Navbox with list of links to Google pages, specialized to search for example user pages, village pump, etc.; useful for Wikipedia:Help desk tasks
• {{Spamsearch}} – Searches user pages for common spams, e.g. "we service", "leading manufacturer", etc.

See also

• Help:Searching
• Category:Search templates
• MediaWiki:Extension:InputBox § General syntax - how to create your own search box using <inputbox>...</inputbox>

• {{template usage}}
• {{search link}}
• {{for loop}}