Class: Tokenzr::Charset

Inherits:
Object
  • Object
show all
Defined in:
lib/tokenzr/charset.rb

Overview

Describes the character sets a Tokenizer uses to classify characters.

Each attribute is a string of characters belonging to that class:

  • text — characters that start and continue an identifier
  • digits — characters that start and continue a number
  • lone — single-character symbols (parens, operators, punctuation)
  • space — whitespace characters that are skipped
  • quotes — characters that delimit a string

The five sets must be pairwise disjoint; Tokenizer.new validates this and raises ConfigurationError on any conflict.

Grab a mutable copy of the defaults with Charset.default, tweak any field, and pass it to Tokenizer.new:

charset = Tokenzr::Charset.default
charset.lone = charset.lone.delete('`')
charset.quotes = charset.quotes + '`'
Tokenizer.new(charset:)

Defined Under Namespace

Classes: Conflict

Instance Attribute Summary collapse

Class Method Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(text: nil, digits: nil, lone: nil, space: nil, quotes: nil, operators: []) ⇒ Charset

Returns a new instance of Charset.



34
35
36
37
38
39
40
41
 
# File 'lib/tokenzr/charset.rb', line 34

def initialize(text: nil, digits: nil, lone: nil, space: nil, quotes: nil, operators: [])
  @text = text
  @digits = digits
  @lone = lone
  @space = space
  @quotes = quotes
  @operators = operators
end

Instance Attribute Details

#digits ⇒ Object

Returns the value of attribute digits.



32
33
34
 
# File 'lib/tokenzr/charset.rb', line 32

def digits
  @digits
end

#lone ⇒ Object

Returns the value of attribute lone.



32
33
34
 
# File 'lib/tokenzr/charset.rb', line 32

def lone
  @lone
end

#operators ⇒ Object

Returns the value of attribute operators.



32
33
34
 
# File 'lib/tokenzr/charset.rb', line 32

def operators
  @operators
end

#quotes ⇒ Object

Returns the value of attribute quotes.



32
33
34
 
# File 'lib/tokenzr/charset.rb', line 32

def quotes
  @quotes
end

#space ⇒ Object

Returns the value of attribute space.



32
33
34
 
# File 'lib/tokenzr/charset.rb', line 32

def space
  @space
end

#text ⇒ Object

Returns the value of attribute text.



32
33
34
 
# File 'lib/tokenzr/charset.rb', line 32

def text
  @text
end

Class Method Details

.default ⇒ Object

Returns a fresh, mutable copy of the default charset.



44
45
46
47
48
49
50
51
52
53
 
# File 'lib/tokenzr/charset.rb', line 44

def self.default
  new(
    text: 'abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ_',
    digits: '0123456789',
    lone: '()[]<>{}!#$%&*+,-./:;=?@\\^`|~',
    space: " \t\n\r\v\f",
    quotes: "\"'",
    operators: []
  )
end

Instance Method Details

#conflicts ⇒ Object

Returns an array of conflict structs, one per character that appears in more than one charset. Empty array when the sets are disjoint.



69
70
71
72
73
74
75
76
77
78
79
80
 
# File 'lib/tokenzr/charset.rb', line 69

def conflicts
  sets = to_sets
  tally = Hash.new { |h, k| h[k] = [] }
  sets.each do |type, set|
    set.each { |chr| tally[chr] << type }
  end
  tally.filter_map do |chr, types|
    next unless types.length > 1

    Conflict.new(chr, types)
  end
end

#to_sets ⇒ Object

Returns a hash of Set objects keyed by symbol, built from the current charset strings. Nil charsets become empty sets.



57
58
59
60
61
62
63
64
65
 
# File 'lib/tokenzr/charset.rb', line 57

def to_sets
  {
    text: Set.new((text || '').chars),
    digits: Set.new((digits || '').chars),
    lone: Set.new((lone || '').chars),
    space: Set.new((space || '').chars),
    quotes: Set.new((quotes || '').chars)
  }
end