xMarkup User's Guide

Version 1.6.9, July of 2002

Fixed found bugs.

Version 1.7.0, September of 2004

1. Added function tabto(i) to move to i-th position of current substring @subject. Call tabto(0) performs move to the end of line and therefore defines move to the next line.

2. Added function like(s1,s2) to check substring s1 by the search mask s2. Function returns 1 (true) if s1 satisfies to s2, else fails.

3. Added debug mode, which is defined by parameter debug=true in the section [options].

Version 1.7.1, February of 2005

1. Fixed bugs related to processing of special characters (\c) in the strings.

2. Added logical operation to check non-equality of string expressions ( a !== b ).

3. Enchanced debug output.

4. Adeed keys "-h" è "-d" in the command line of utility.

5. Added macro-definition @nfiles, which defines a total number of processed files.

6. Re-developed examples in this guide to make they maximum simple and clear.

Version 1.7.2, March of 2005

1. Fixed bugs related to using of macro-definitions for start/stop marks.

Version 1.7.3, November of 2005

1. Fixed an error while processing of strings, which ends with sequence of back slash characters (\).

Version 1.8.0, September of 2006

1. Added two predefined macro-procedures "initialize" and "finalize", which are automatically executed at the beginning and finishing of processing. These procedures can be defined in section [macros] if needed.

2. Fixed an error while processing of sequence of start marks @bof, @bol. Previously the first line of text may be skipped for such sequence of start marks.

3. Optimized the search of start/stop marks while processing.

Version 2.0, April of 2007

1. xMarkup GUI released.

2. Fixed an error while processing of compound multi-line structure in figure brackets {...}.

3. Added command line option -c to generate output to directory structure the same as for source files.

4. Added command line option -q1 for GUI integration.

5. Added macro-definition @regexp() to define start/stop marks as regular expressions.

6. Enchanced built-in function substr(), which now returns empty value instead of &fail exception when source string doesn't contain defined substring.

7. Fixed an error in description of macro-definitions @subject and @pos.

8. Added previously missed Cyrillic letters "¸" and "¨" (yo) to macro-definitions @cp866, @cp1251 and @cletters.

Version 2.0.1, July of 2007

1. Fixed an error in value of @subject (previously the last character of line was missed).

2. Fixed an error when file of parameters is being rewritten each time as processing is started (autosave mode) even it was not changed.

Version 2.1.0, October of 2007

1. Added command line option -x to simulate execution to check correctness of processing script.

2. Added directive @include to include content of external text file to body of script.

3. Added macro-definitions @version, @features, @host, @e and @pi.

4. Added built-in function sql_quotes(s) to change single apostroph character (') to double one ('') within string s.

5. Enchanced GUI - added possibility to change font and color settings for work windows.

6. Console version of xMarkup for POSIX/UNIX-like systems released.

Version 2.1.1, April of 2008

1. Added new features:

• When option -oCON: is specified in command line the whole output is redirected to standard output device. This output mode is added to GUI too.
• Debug mode became interactive now. After each processing step the utility stops and user then can continue or quit.
• Included text files may be opened for edit by double mouse click on the line with @include directive in macros window.

2. Fixed some minor bugs and oddities. Fixed the situation when output to direcory structure the same as for source files may fail.

3. Excluded situation when output files may overwrite the source files (input and output paths defined the same by user).

Version 2.1.2, May of 2008

1. Added macro-definition @call, with help of which you can "call" a macro-procedure from body of other macro-procedure.

2. Fixed a stupid error of version 2.1.1, when path to read/write file could not be properly defined if it's not implicitly specified.

Version 2.1.3, September of 2008

1. Added function eof() to skip processing of the source file and go to its end.

2. Added function file_exists(s) to check existance of the file s.

3. Added function get_separator(), which returns character-separator used in the file paths: (/) for Unix and (\) for Windows.

4. Added function sortfiles(L), which sorts the list of file names L by path and name.

5. Added comamnd line's switch -s to sort the list of source files before processing.

6. Revoked default output mode; now to create output files you shall explicitly specify output path in option -o.

7. At Sourceforge.net released source package of xmarkup, which may be used to port utility to required UNIX-like system (see chapter 9 for details).

Version 2.1.4, January of 2009

1. Fixed macros @time, which shall return elapsed time of processing.

2. Fixed GUI - in the previous version doesn't work an output mode in which output files shall be named by adding "xm$" prefix to the name of source files.

3. Changed GUI - link to xmarkup download page added on the tab "Help".

4. Script of processing rules

Script of processing rules (parameters file) includes a few sections, in which the search criteria, text transformation templates and other optional parameters are defined. Each section starts by predefined name in square brackets []. Section names are case insensitive. The list and order of sections in the script are arbitrary.

Parameters file can include remark lines, which begin with sharp (#) or semicolon (;) character. Comment lines in the befinning of script form its header. Any number of empty lines can be inserted to improve the readibility.

4.1 Start Marks

Start marks describe the search templates for beginning of text elements, which should be processed. Start mark can be defined by string, character set or position in the current source file or line. Predefined macro-definitions can be used to define the start marks too.

The list of alternative start marks is defined in the section [startEntity], for example:

[startEntity]
; to find elements beginning with any digit
@digits
; to find HTML-tag "title"
<title>
; to find elements beginning with double space
@space@space

4.2 Stop Marks

Stop marks describe the search templates for ending of text elements, which should be processed. Stop mark can be defined by string, character set or position in the current source file or line. Predefined macro-definitions can be used to define the stop marks too.

The list of alternative stop marks is defined in the section [stopEntity], for example:

[stopEntity]
@sp
</title>
@null

4.3 Transformation Templates

Transformation templates describe the conversion procedures of found text elements. Each transformation template is a string, which value is substituted instead of found text element (that is the text between start and stop marks). Transformation template can include both static and dynamic values, which are defined with help of various macro-definitions.

The list of transformation templates is defined in the section [startMarkup], for example:

[startMarkup]
@start<font color="red">@body</font>@stop
@null
@space

Optional section [stopMarkup] can be used to define a single post-transformation, which should be performed after each transformation defined in [startmarkup] section.

4.4 Optional parameters

With help of options various working modes and parameters of the utility are defined. Options are described in the optional section [Options]:

The names of parameters are case insensitive.

4.5 HTML-markup to be skipped while processing

To skip from processing text content, which is located within open and close paired HTML-tags, the list of names of such tags can be described in the optional section [tagExceptions]:

[tagExceptions]
head
title
script
table

In this case a whole text between open <name> and close </name> tags is skipped while processing. However the body of tags may be processed. Any elements with arbitraty set of atributes may be used as skipped tags.

If you need to skip while processing body of any standard HTML-tags please define option skipTags=true.

4.6 Procedural macros

Macro-procedures are defined in the optional section [Macros]. The using of macros can be justificated when the special or not trivial text processing is needed.

Each macro-procedure begins with the title

or

and ends with the line

The body of macro includes a set of statements on macro-language, which is described in the section 5. The macro is executed as a procedure and can return value, for example:

[Macros] macro increment # if current line begins with number n then return n+1 if i := many(@digits, @line)-1 then return numeric(substr(@line,1,i)) + 1 end

Macro-procedures are called from transformation templates with help of macro-definition @run(), for example

[startmarkup]
@start@run(name)@stop

The names of functions and variables within body of macro are case sensitive! The types of variables are not explicit specified as in Icon but defined by current values. Together with user variables the macro-definitions can be used (as read-only variables) and counters' variables (counter, counterIncr, counterInit). The last ones are defined as arrays and can be recomputed, for example:

  counter[1] := counterInit[1]
  counter[1] := counter[1] + counterIncr[1]

Since version 1.8.0 two predefined macro-procedures "initialize" and "finalize" were introduced. Procedure "initialize" is executed automatically at the very beginning of processing (before any source file is open) and procedure "finalize" is executed at the end of processing (after closing the last source file). With help of "initialize" procedure you can initialize required variables, which shall be used during processing. With help of "finalize" procedure you can perform required actions after finishing the processing. These procedures are optional.

[macros] procedure initialize write("Beginning of processing...") total := 0 end procedure finalize write("The end.") write("total: ",total) end

5. Description of macro language

5.1 Introduction

The language, which is used in macro-procedures is the simplicated dialect of the Icon Programming Language. The following features are supported:

  - integer and real values arithmetic;
  - string processing;
  - variables, macro-definitions, lists and arrays;
  - calls of all intrinsic Icon-functions;
  - unary operators:
     + (absolute value)
     - (negation);
  - assignment:
     :=
  - binary arithmetic operators:
     + (addition)
     - (substruction)
     * (multyplication)
     / (division)
     % (division by module)
     ^ (raise to power)
  - string concatenation:
     ||
  - relational operators for numeric values (returns 1 if true, else 0):
     =  (equality)
     != (non-equality)
     <  (less then)
     <= (less or equal)
     >  (greater then)
     >= (greater or equal)
  - relational operators for strings (returns 1 if true, else 0):
     ==  (equality)
     !== (non-equality)
  - compound statements in curly braces :
     {}
  - logical statements:
     if-then
     if-then-else
  - loop statements:
     while-do
     every-do
  - commentary lines beginning with character "#" or ";".

Described language features were implemented with help of program icalc.icn by Stephen B. Wampler. This program is included in Icon Program Library and intended to simulate infix desk calculator. The brilliant ideas implemented in icalc.icn allowed to use it as a basement for next enchancments and additions. Finally the idea to use it for macro-procedures came. The results are very fruitful.

Those who wants to learn more about features of Icon language can read the nice book of Ralph Griswold The Icon Programming Language. Ralph Griswold was author of Icon, he has gone in 2006.

5.2 Language Grammar in Backus-Naur Form (BNS)

  Statement ::= Expression | If | Loop | Return | Block-of-Statements

  Block-of-Statements ::= { List-of-Statements }

  List-of-Statements ::= Statement | Statement ; List-of-Statements

  If ::= if Expression then Statement Else

  Else ::= else Statement | ""

  Loop ::= While_loop | Every_loop

  While_loop ::= while Expression do Statement

  Every_loop ::= every Expression to Expression do Statement

  Return ::= return { Expression | "" }

  Expression ::= Condition | Variable := Expression

  Condition ::= Term {= | != | < | > | >= | <= | == | !== } Term | Term

  Term ::= T { + | - } Term | T

  T ::= F { * | / | % } T | F

  F ::= E ^ F | E

  E ::= L | { + | - | || } L

  L ::= Function | Variable | Constant | ( Expression ) | String | Character-Set | Macro-definition | List

  Fucntion ::= Identificator ( List-of-arguments )

  Variable ::= Identificator | Identificator[ Expression ]

  Constant ::= integer or real number
   
  String ::= "string"
	
  Character-Set ::= 'string'

  Macro-definition ::= &Identificator | @Identificator

  List ::= [ List-of-arguments ]

  List-of-arguments ::= "" | Expression | Expression , List-of-arguments

Each statement of macro-language is written on the separate line and can not be continued on the next lines. Exclusion is a block of statements in curve brackets, which can be written on the many sequental lines. Each statement in the block should be ended by character semicolon (;).

5.3 Variables

All variables used in macro procedures are global. It means that their values are stored after completion of the macro. And moreover these values are common for all macros. The variable types are not explicit specified as in Icon but defined by current values. The names of variables and functions are case sensitive.

5.4 Counters

The counters are used when the dynamic markup should be created, which content is defined by sequental numbers of the processed text elements. For example, with help of counters the navigation hyperlinks can be generated for a set of html-documents, the names of which includes sequental numbers.

The number of counters is not limited and their parameters (initial value, type and increment) are defined in the section [Options]. The counters can be of two types - relational and absolute. Relational counters vice verse of absolute ones are automatically initialized when the next source file is opened for processing.

The values of i-th counter can be substituted within transformation template with help of macro-definition @counter(i) or assigned in macro procedure by variable counter[i]. The macro-definition @counter(i) substitutes the current value of i-th counter and, if the auto increment mode is enabled (autoincr=true) then automatically increments it. When auto increment mode is disabled (autoincr=false) the incrementing of i-th counter can be defined in transformation template with help of macro-definition @next(i). This macro-definition doesn't substitute any value but only performs incrementing of counter. In macro procedure the incrementing of counter can be executed with help of statement:

counter[i] := counter[i] + counterIncr[i]

To perform manual initializing of i-th counter within transformation template the macro-definition @reset(i) can be used. This macro-definition doesn't substitute any value but resets the counter to initial value. In macro procedure the initializing of counter can be executed with help of statement:

counter[i] := counterInit[i]

5.5 Built-in functions

The short description of the most used functions of Icon is presented below. For each function the types of input parameters and results are given. The next signs are used:

N - natural number; i - integer; r - real; s - string; c - character set; L - list of values (array); x - any value; f - file descriptor.

The detailed description of all functions can be found on Icon home page in Arizona University http://www.cs.arizona.edu/icon/.

5.5.1 Numeric functions

5.5.2 String functions

Note: fail-interruption leads to rollback of execution of current operator. This behaivour is defined by operating logic of Icon language, which supports the backtracking (as in Prolog). If current statement fails it means that the statement is just not executed. For example, the following code will fail because string "qwerty" is not began with "123", so variable i will not get any value:

...
i := match("123", "qwerty")
write(i)
...

5.5.3 List functions

5.5.4 I/O functions

5.5.5 System functions

5.5.6 Internal functions

5.6 Macro-definitions

Macro-definitions can be used within start and stop markers, transformation tempaltes or procedural macros. Macro-definition is used to define string value, set of characters or position in source file/line.

Examples how to use macro-definitions:

Note. Macro-definitions, which return a set of values or position can not be used in the transformation templates (except of @read(s)). Names of macro-definitions are case sensitive.

5.7 Usage of macro-definition @null

Macro-definition of the null string @null can be used in three cases.

1. To define markers, which begin with sharp (#), semicolon (;) or opening square bracket ([) characters, which are used to define the comment lines or sections. For example,

[startEntity]
; marker is started with sharp character
@null#01
; marker is started with semicolon
@null;01
; marker is started with opening square bracket
@null[01]

2. To define search by one marker only (start or stop). For example, searching of "abc" can be defined as:

[options]
syncStop = true

[startEntity]
abc

[stopEntity]
@null

or

[options]
syncStop = true

[startEntity]
@null

[stopEntity]
abc

3. To define start/stop markers. That is if markup template is defined as @null then substring <start-marker><body><stop-marker> will be removed from the output text.

5.8 Special characters

Special characters are used within string values (in procedural macros) and defined as 2-character sequence beginning with character of backward slash (\).

Example.

  s := "\"this line will be outputted in quotes\""
  write_output(s,"\n")

5.9 Usage of regular expressions

Since version 2.0 of xMarkup start or stop marks can be defined with help of regular expressions. This feature was implemented with help intergartion of procedure regexp.icn, which author is Robert J. Alexander. This procedure is from Icon Public Library.

Regular expression is defined with help of macro-definition @regexp(s), for example: @regexp("[0-9]*\.[0-9]+"). Please see chapter Examples, in which usage of regular expressions is illustared in a few examples.

The regular expression format is very close to format supported by the UNIX "egrep" program, with modifications as described in the Perl programming language definition. Following is a brief description of the special characters used in regular expressions. In the description, the abbreviation RE means regular expression.

The following extensions to UNIX REs, as specified in the Perl programming language, are supported.

Symbols \w, \W, \s, \S, \d, \D can be used within [string] REs.

5.10 Usage of macro-definitions @subject and @pos

Macro-definition @subject defines a rest of current processed line, that is a substring between stop marker and end of line. Macro-definition @pos defines offset position in @subject (1 by default). Function tabto(i) produces move to offset position i. The move means that all characters between start of @subject till position i shall be skipped from processing (as there is no such substring in a source line). If a whole line @subject shall be skipped then offset position defined as 0. Negative value of offset position means move to i-th position starting with end of line. For example, tab(3) moves to third character of @subject; tab(-1) moves to last character of line; tab(-3) moves to third character from the end of line. See example 8.5, which demonstrates usage of tabto() function.

5.11 Usage of loop operators while and every

To get filling how to use these loops it will be usefull to show simple examples. Let's we have array or list of values, which we need to process in some way. For that we will use loop operator while or every - which we like.

In the case of while any expression, which return numeric value, may be used as logical condition. Check of condition gives true if value of expression is not equal 0 else false. For example,

  # print list of values with help of while-loop
  array := ["a","b","c"]
  i := 0
  while i < len(array) do write( array[i := i + 1] )

It could be strange but expression i < len(array) returns 1 if value of i less then length of array else 0. Such agreement is defined for any logical conditions. By the way, expression while 1 do ... defines endless loop.

Another example of while to process a list of values. In this case function get() is used. This function returns next element of a list and simultaneously pushes it from a list. After end of loop the source list will be empty.

  # print list of values with help of while-loop
  array := ["a","b","c"]
  while len(array)>0 do write( get(array) )
  # array now is empty

When you choose to use every-loop you need to specify interval of integer numbers, which define length of a loop. For example, every 3 to 5 defines length 3 (for values 3,4,5). Any expressions, which return interger numbers, may be used as boundaries of a loop interval. Below is example of every-loop:

  # print list of values with help of every-loop
  array := ["a","b","c"]
  i := 0
  every 1 to len(array) do write(array[i := i + 1])

6. Algorithm of text processing

The utility works as a kind of finite states machine. The instructions to the machine are defined by search conditions and transformation rules described in the script of processing rules. On the input is given a source text, on the output the results of its transformation are put. Let us describe the general algorithm of text processing implemented in the utility:

During searching of the start or stop marks their priority is taken into account. It means that the mark with higher priority will be choosen in the case of other equal conditions.
Let us enumerate marks in the descent order of their priority:

1. @bof
2. @bol
3. @null
4. any substring within source line
5. @eol
6. @eof

7. Limitations

Below is a list of limitations:

1. Start mark should be located a whole in a source line, else it will not be found.

2. Stop mark should be located a whole in a source line too.

3. Definition of start or stop mark may include only one macro-definition in the case of character set. And moreover nothing else can be set in such definition. For example, definition @digits@letters is wrong. This limitation is very easy to eleminate by using of procedural macros.

4. Statement if ... then ... else ... should be written in a single line. This limitation may be work arounded with help of block structure:

  if i = 1 then s := "one" else {
    if i = 2 then s := "two" else {
       if i = 3 then s := "three" else s := ""
       }
    }  
  }

5. It is impossible to use logical operators "or", "and", "not" to construct logical expressions.

6. Value of expression used in operator "if" or "while" as a checked condition shall be integer (0 corresponds to false, 1 - true). For example, expression
while s := read() do write(s)
will produce Run-time error when value of string s is not a number. This example code shall be rewritten as
while (s := read()) !== "" do write(s),
that defines reading loop from standard input while it contains not empty lines.

7. Macro-definition @bof may be used only for start mark.

8. As utility is a "double" interpreter (Icon binary is a byte-code itself) its performance by definition is much slow than C or C++ programs.

9. Source text files shall be in single-byte ANSII encoding; UTF-8 or Unicode data may be processed in a wrong way.

8. Examples

The print-outs of scripts are provided below to show how text may be processed by xMarkup.

8.1 HTML to plain text transformation

See example 8.9, which shows as a list of HTML-tags may be generated.

# html2txt.par
# html-markup cut-off.
# The list of html-tags is defined in file "tags.list".
# Script usage:
# xm -f*.html -phtml2txt.par

[StartEntity]
<!--
<script
<noscript
<@read(tags.list)
</@read(tags.list)

[StopEntity]
-->
</script>
</noscript>
>

[StartMarkup]
@null

[Options]
ignoreCase = true
syncStop   = true

The modification of the script to cut-off any XML/SGML/HTML markup is provided below. For that regular expressions are used.

# conv2txt.par
# Cut-off any XML/SGML/HTML markup
[startEntity]
@regexp("<(no)?script")
@regexp("<(/)?(\w)+")
<!doctype
<!--

[stopEntity]
@regexp("</(no)?script>")
>
>
-->

[startMarkup]
@null

[Options]
syncStop = true
ignoreCase = true

8.2 Purging of empty lines

# cut_blank_lines1.par
# Purging of empty lines, version #1
# Script usage:
# xm -f*.txt -pcut_blank_lines1.par
[StartEntity]
@bol

[StopEntity]
@eol

[StartMarkup]
@eval( if len(trim(@line)) > 0 then @line||@nl )

[Options]
addNewLine = false

This script produces not completly valid output: a single empty line is added always to the end of source text. Some minor modification of the script gives a true result:

# cut_blank_lines2.par
# Purging of empty lines, version #2
# Script usage:
# xm -f*.txt -pcut_blank_lines2.par
[StartEntity]
@bol

[StopEntity]
@eol

[StartMarkup]
@run(line)

[Options]
addNewLine = false

[Macros]
procedure initialize
  s := ""
end

procedure line
  if len(trim(@line)) > 0 then { return s||@line; s := @nl }
end

8.3 Insertion of header and footer

# headers.par
# Insertion of header and footer to text.
# Script usage:
# xm -f*.txt -pheaders.par
[StartEntity]
@bof
@eof

[StartMarkup]
<!-- This is a header of @file -->@nl
<!-- This a footer of @file -->

[Options]
syncMarkup = true

8.4 Markup of words in a sentence

It is supposed that all words in a sentence are in Latin alphabet. The sentence starts by word with first capital letter and ends by point, question or exlamation mark.

# words.par
# Markup of Latin words in a sentence
[startentity]
@ucase
@lcase

[stopentity]
@cset(" \t,-:;\"'.?!")
@eol

[startmarkup]
@run(sentence)@run(word)
@run(word)

[macros]
macro sentence
  # beginning of the sentence
  counter[1] := counter[1] + 1
  counter[2] := 0
  write_output("<sentence id=\"",counter[1],"\">")
end

macro word
  counter[2] := counter[2] + 1
  write_output("<word id=\"",counter[1],".",counter[2],"\">",@start,@body,"</word>",@stop)
  # if the end of sentence is reached
  if any(".?!", @stop) then write_output("</sentence>")
end

[options]
# counter[1] - counter of sentences,
# counter[2] - counter of words
counterInit = 0,0
syncMarkup  = true

Mommy watches TV. 
Daddy drinks a wine.
Children are playing.

<sentence id="1"><word id="1.1">Mommy</word> <word id="1.2">watches</word> <word id="1.3">TV</word>.</sentence>
<sentence id="2"><word id="2.1">Daddy</word> <word id="2.2">drinks</word> <word id="2.3">a</word> <word id="2.4">wine</word>.</sentence>
<sentence id="3"><word id="3.1">Children</word> <word id="3.2">are</word> <word id="3.3">playing</word>.</sentence>

The modification of the script in which regular expressions are used:

# words2.par
# Markup of words in a sentence with help of regular expressions
[startentity]
@regexp("[A-Za-z]+[\s,-:;\"'\.?!]")

[startmarkup]
@run(markup)

[macros]
macro initialize
  sentence_counter := 0
  word_counter := 0
  is_end_sentence := 1
end

macro markup
# beginning of the next sentence
  if is_end_sentence then {
    sentence_counter := sentence_counter + 1
    word_counter := 0
    write("<sentence id=\"",sentence_counter,"\">")
  }	
  word := substr(@start, 1, len(@start)-1)
  word_end := substr(@start, len(@start), 1)
  word_counter := word_counter + 1
  write("<word id=\"", sentence_counter, ".", word_counter, "\">", word, "</word>")
  # check, is it the end of sentence?
  if any(".?!", word_end) then { write("</sentence>"); is_end_sentence := 1 } else is_end_sentence := 0
end

8.5 Purging of long spaces

This example illustrates usage of shift-pos function tabto() and macro-definition @subject.

# trim.par
# Purging of long spaces.
# The sequncees of long spaces are replaced by single white space. End of line is trimmed.
[startEntity]
@cset(' \t')

[startMarkup]
@run(trim)

[Macros]
procedure trim
# shift to the first not empty character at the current line
  if i := many(' \t', @subject) then tabto(i)
# if we are at the end of line then trim trailing space
  if @pos >= len(@subject) then return @null
  return @sp  
end

The modification of script, which uses regular expressions.

# trim2.par
# Purging of long spaces
[StartEntity]
@regexp([\s]+$)
@regexp([\s]+)

[StartMarkup]
@null
@sp

[Options]
syncmarkup=true

8.6 Merging of text files

In this example a set of source files are merged to a single output file, name of which is prompted to enter (by default "unite.dat"). After completion of processing the number of lines written to output is displayed. With help of this example you can understood how useful may be procedures initialize è finalize.

# unite.par
# Merging of text files to a single one
[StartEntity]
@eol

[StartMarkup]
@run(line)

[macros]
macro initialize
  writes("Output file [unite.dat]: ")
  if (s:= read()) == "" then s := "unite.dat"
  f := open(s,"w")
  rows := 0
end

macro line
  write(f,@line)
  rows := rows + 1
end

macro finalize
  write(rows, " lines written to ",s)
end

8.7 Reformatting of long lines

This example shows as to wrap extremely long source lines of XML/SGML/HTML document to a set of short and readable lines.

# reformat.par
# Reformat lines of XML/SGML/HTML document, which length exceed specified size.
# Long lines are wrapped to a set of short and readable lines.
# As psition of line break the beginning of nearest tag is used.
[startEntity]
@bol

[stopEntity]
@eol

[startMarkup]
@run(reformat)

[Options]
addNewLine = false

[Macros]
procedure initialize
# define required max length of output line
# (ineed it may be slightly longer)
  max_len := 80
end

procedure reformat
  line_len := len(@line)
  i := 1
  while (i <= line_len) do { 
     s := substr(@line, i, max_len)
     i := i + max_len
     # split source line by segments with lenght at least max_len characters;
     # use the beginning of nearest tag as position of line break  
     if (j := upto("<", substr(@line,i))) then { s := s || substr(@line, i, j-1); i := i + j -1 }
     write_output(s, "\n") 
 }
end

8.8 Text sorting

The very simple example how to order the source lines by alphabet.

# sort.par
# Sort text and output it on the screen.
# Script usage:
# xm -f*.txt -psort.par -oNUL:

[startentity]
@bol

[startmarkup]
@run(line)

[macros]
macro initialize
# define empty list
  p := list()
end

macro line
# put to the list next line
  put(p, @line)
end

macro finalize
# sort the list and output it
  p := sort(p)
  while len(p) > 0 do write(get(p))
end

8.9 Analysis of markup

This script outputs a list of tag names, which are used in source XML/SGML/HTML files.

The unique list of tag names is generated by Icon-function set(), which is used in macro-procedure finalize.

# list_tags.par
# Output list of tag names, which are used in source xml/sgml/html files.
# Usage example:
# xm -f*.sgm;*.html;*.xml -plist_tags.par -oNUL:
[startentity]
<

[stopentity]
>

[startmarkup]
@run(tag)

[macros]
procedure initialize
  p := list()
end

procedure tag
  t := lower(@body)
  if match("/", t) then return
  if (i := upto(" \t\n",t)) > 0 then t := substr(t,1,i-1)
  put(p, t)
end

procedure finalize
  p := sort(set(p))
  while len(p) > 0 do write(get(p)) 
end

8.10 Counting of lines in a file

# Name: lines-count.par
# Counting of number of lines. Outputs current time and number of lines for each source file.
# Usage example:
# xm -ftext.txt -plines-count.par -onul:
[startentity]
@eof

[startmarkup]
@eval(write(@date||" "||@clock,"\t", @lineno))

8.11 Converting XML-data to CSV format

Source XML-data (personnel.xml) are represented by following structure:

<?xml version="1.0" encoding="windows-1251" ?> <personnel> <employee id="7234" mgr="7777"><dept>Sales</dept><name>Tom Scott</name><salary>5700</salary></employee> <employee id="7777"><name>Alan Cruzo</name><dept>Administration</dept><salary>15000</salary></employee> <employee id="7001" mgr="1234"><dept>Delivery</dept><name>Jane Fisher</name><salary>6100</salary></employee> <employee id="1234" mgr="7777"><dept>Delivery</dept><name>John Asher</name><salary>15100</salary></employee> </personnel>

We need to generate string values of following attributes and elements:
employee.id, employee.mgr, dept, name, salary

# Name: xml2csv.par
# Convert XML-data to CSV (comma separated values). The results are outputted to console.
# Usage example:
# xm -fpersonnel.xml -pxml2csv.par -onul:
[startentity]
<employee
id="
mgr="
<dept>
<name>
<salary>
</employee>

[stopentity]
@null
"
"
</dept>
</name>
</salary>
@null

[startmarkup]
@run(employee)
@run(employee_id)
@run(employee_mgr)
@run(dept)
@run(name)
@run(salary)
@run(write_employee_data)

[options]
syncmarkup=true
syncstop=true

[macros]
macro employee
# Set default value of mgr element (if it is missed)
  mgr := ""
end

macro employee_id
# get value of attribute employee.id 
 id := @body
end

macro employee_mgr
# get value of attribute employee.mgr 
 mgr := @body
end

macro dept
# get value of element dept 
 dept := @body
end

macro name
# get value of element name 
 name := @body
end

macro salary
# get value of element salary 
 salary := @body
end

macro write_employee_data
# output the result using specified delimeter
 d := ","
 write(id, d, mgr, d, dept, d, name, d, salary)
end 

The result of processing of source XML-data could be:

7234,7777,Sales,Tom Scott,5700 7777,,Administration,Alan Cruzo,15000 7001,1234,Delivery,Jane Fisher,6100 1234,7777,Delivery,John Asher,15100

8.12 Analysis of words frequency in a text

This example demonstrates a simple method how to calculate words frequency in a text. Extraction of words from a source text is done with help of regular expressions.
The processing speed is very slow but is a quite enough for small texts.

# words-count.par
# Word is defined as following:
# it begins with letter and may include else letters characters of apostroph (') or hyphen (-).
[startEntity]
@regexp("[A-Za-z][A-Za-z\-']*")

[startMarkup]
@run(count)

[Macros]
procedure initialize
# to store pairs of values (word,frequency) we use table structure;
# word is a key in table, frequency is its value (by default 0)
  t := table(0)
end

procedure count
  word := @start
# check current word in a table:
# if no such word the value of key t[word] will be 0
  i := t[word]
  if i = 0 then t[word] := 1 else t[word] := i + 1
end

procedure finalize
# sort table of words by alphabet (1) or frequence (2)
  orderby := 1
  words := sort(t, orderby)
# each element of output structure words contains list of two values [word, frequency].
  i := 0
  while len(words) > 0 do { 
    w := get(words)
    i := i + 1 
    write(left(i,5), left(w[1],30), w[2])
  }
end

8.13 Generation of SQL-script from CSV data

This example demonstrates how to generate SQL-script to add data to database from source CSV data. Description of configuration (template of insert SQL-statement, number of columns and delimeter) is defined in external file csv2sql.config, which is loaded with help of directive @include.
To process characters of apostroph (') within strings the built-in finction sql_quotes(s) is used.
Note: for this example xMarkup later version 2.1.0 is required.

# csv2sql.par
# Convert ÑSV data (comma separated values)
# to a set of SQL-statements insert into ... values().
[startEntity]
@bol

[stopEntity]

[startMarkup]
@run(markup)

[Options]

[Macros]
procedure initialize
  @include csv2sql.config
  rows := 0
end

procedure markup
  rows := rows + 1
  if rows >= commit_cycle then { write("commit;"); rows := 0 }
  vstr := ""
  s := @line
  j := 0
  while (i := find(delim, s)) do if j >= cols then s:="" else {
    val := "'" || sql_quotes(trim(substr(s,1,i-1))) ||"'"
    s := trim(substr(s, i+1))
    j := j + 1
    if j = 1 then vstr := val else vstr := vstr || ", " || val
  }
  vstr := vstr || ", '" || sql_quotes(s) || "'"
  j := j + 1
  while j < cols do {
    vstr := vstr || ", NULL"
	j := j + 1
  }	
  vstr := vstr || ");"
  write(sql, vstr)
end

# finalize processing
procedure finalize
  write("commit;")
end

Configuration file for source data, which were generated in example 8.11 could be:

# csv2sql.config # Configuration for csv2sql.par # template of SQL-statement: sql := "insert into personnel(id, mgr, dept, name, salary) values(" # character delimeter: delim := ',' # Number of columns in source CSV-file: cols := 5 # Interval between commits: commit_cycle := 1000

Then after processing of source data we will get the following SQL-script:

insert into personnel(id, mgr, dept, name, salary) values('7234', '7777', 'Sales', 'Tom Scott', '5700'); insert into personnel(id, mgr, dept, name, salary) values('7777', '', 'Administration', 'Alan Cruzo', '15000'); insert into personnel(id, mgr, dept, name, salary) values('7001', '1234', 'Delivery', 'Jane Fisher', '6100'); insert into personnel(id, mgr, dept, name, salary) values('1234', '7777', 'Delivery', 'John Asher', '15100'); commit;

8.14 Exchange of text values

Let the names of persons in source text are messed and we shall synchronically change one names to others. Ordinal search and replace is not good for that task because we shall perform many intermediate steps when text includes many messed names. With help of xMarkup this operation is simple and trivial. Let's we have "corrupted" part of Genesis (11:14):

When Eber had lived 30 years, he became the father of Shelah. And after he became the father of Shelah, Eber lived 403 years and had other daughters and sons.

To have canonical text we need to change "Eber" to "Shelah", "Shelah" to "Eber", "daughters" to "sons" and "sons" to "daughters".

# exchange.par
# Exchange names in a text
[startEntity]
Eber
Shelah
daughter
son

[startMarkup]
Shelah
Eber
son
daughter

[Options]
syncMarkup = true

All what we had to do - define two synchronized lists of source and target values.

8.15 Generation of script to process a list of files

This is a simple example how to generate script, which shall consequentally process a list of source files by some command. Take into account using of eof() call as a parameter of write(). Function eof() returns nothing but move us to the end of file. For huge files that can dramatically speed up the process.
Note: use mode without creation of output files and supressing work messages (options "-oNUL:" and "-q" of console command).

# generate-script.par
# Generation of script to process source files
[startEntity]
@bof

[startMarkup]
@eval( write("some-command ",@input, eof()) )

Ready text of script may be copied from console window and written to BAT-ôàéë.

8.16 Generation of content table for html-document

A pair of interesting processing techniques are demonstrated in this example:

• redirecting of current output
• "on-fly" reformating of this output.

• Remember the starting part of document (from the beginning till the tag <body> including);
• Redirect current output with help of functions close_output and open_output;
• Subsequentally process all headings for levels <h2> and <h3>:
• for each heading generate unique id value and put ancor <a name="id">;
• generate the next row of content table and remeber it;
• After reaching the end of source document redirect output againg and create final output document.

# content_table.par
# Generation of content table in html-document.
# Items of content table will be headers <h2>...</h2> and <h3>...</h3>.
# SL, 2008/10/15
[startEntity]
@bof
<h2>
<h3>
@eof

[stopEntity]
<body>
</h2>
</h3>
@null

[startMarkup]
@run(header)
@run(level1)
@run(level2)
@run(finish)

[Options]
syncStop = true
syncMarkup = true
counterinit = 0

[Macros]
# Initialize the variables
procedure initialize
  content_table := []
  header := ""
  level := 0
end

# Process the beginning part of document(from the beginning till the tag <body>)
procedure header
  # remeber the beginning part of the source html-document
  header:=@body||@stop
  # get path and name of current output file
  file := getpath(@input)||getname(@input)
  # close and remove this file (it shall be empty at the moment)
  close_output()
  remove(@output)
  # redirect output to new output file (temporary)
  open_output(file||".tmp")
end

# Generate the next item of content table
procedure element
  # generate unique id of the item (left padded by zeroes to length 3)
  id := right(counter[1]:=counter[1]+1,3,0)
  # store value of the item in the list
  put(content_table, "<li><a href=\"#"||id||"\">"||@body||"</a></li>")
  # add required markup (the ancor tag) to output
  write_output("<a name=\"",id,"\"></a>",@start,@body,@stop)
end

# process the first level header <h2>...</h2>
procedure level1
  if level = 2 then put(content_table,"</ul>")
  level := 1
  @call element
end

# process the second level header  <h3>...</h3>
procedure level2
  if level = 1 then put(content_table,"<ul>")
  level := 2
  @call element
end

# finalize processing
procedure finish
  # close current output file and store its's content
  close_output()
  text := get_content(@output)
  # then remove this file - we needn't it more
  remove(@output)
  # open final output file and generate it's content
  file := file||"_output.html"
  f := open(file,"w")
  # write beginning part...
  write(f,header)
  # write content table...
  write(f,"<h2>CONTENT TABLE</h2>")
  write(f,"<ul>")
  while len(content_table)>0 do write(f,get(content_table))
  write(f,"</ul>")
  # write body of source document...
  while len(text)>0 do write(f,get(text))
  close(f)
  write(">", file, " created.")
end

Note that output file in this example will be created regardless the choosen output mode. Because this output is intentionally redirected while processing.

After running this script on the source document it will be reformatted to following output document.

9. Using of xMarkup on POSIX/UNIX-like OSes

As Icon is a cross-platform language xMarkup utility may be used on any system, which supports Icon. Now Icon is available for the following UNIX systems:

• Linux x86/x64
• FreeBSD x86/x64
• SUN Solaris on SPARC
• Darwin and X for Macintosh
• CygWin for Windows

To build binary file of xMarkup on required system you shall install Icon-compiler and then make utility from the source code. The Icon distributives for different UNIX systems are availaible for download from Icon Project Home http://www.cs.arizona.edu/icon/v943/. The sources of xMarkup are available at SourceForge http://sourceforge.net/projects/xmarkup/.

9.1 Installation of Icon from binaries

• 1. Download Icon binaries for your platform from http://www.cs.arizona.edu/icon/v943/
• 2. Unpack archive file to local directory, for example

       cd /usr/bin
       gunzip <freebsd.v943.tgz | tar xf – 

• 3. Check Icon compiler (icont) in directory icon.v943/bin

9.2 Build Icon compiler from sources

This step is requried when prebuilded binaries of Icon are not compatible with your specific system.

• 1. Download Icon sources from http://www.cs.arizona.edu/icon/v943/
• 2. Unpack archive file to local directory, for example

       cd /usr/bin
       gunzip <icon.v943src.tgz | tar xf – 

• 3. Change directory to icon.v943src
• 4. Do "make Configure" to list available configurations.
• 5. Do "make Configure name=<config_name>" to choose required configuration.
• 6. Do "make" to build Icon binaries.
• 7. Check Icon compiler (icont) in directory icon.v943src/bin

9.3 Build xmarkup utility

• 1. Unpack archive file with xmarkup sources to local directory, for example

       cd /usr/bin
       unzip xmsrc_213.zip 

• 2. Change directory to xmarkup/src and edit file env.icn:

       #check that file contains line 
       $define _UNIX 1
       #set name of your Operating System (use "uname -a" to check this name)
       $define ENV "FreeBSD 4.0"

• 3. Edit script "make" to set IPATH to home of Icon compiler, for example

       IPATH=/usr/bin/icon.v943/bin

• 3. Execute script "make" to build xmarkup

       sh make

• 4. Check directory xmarkup/bin for output.