Token.shape_ not keeping a consistent length

[rafelafrance]

I've been using shapes in certain matcher rules and I came across this: The shape_ field is not always a consistent length. If I look for a title case like Xxxxx, the shape_ is always at most 5 characters long. However if I look for a shape for a string like "McClure", the shape_ winds up being 7 characters long XxXxxxx. Is it a bug? That's for your call, but it was certainly a surprise.

You (or me via pull request) may want to slice the shape_ (and its interned shape) to a consistent length, and 5 seems like a perfectly fine length.

Now off to look at what happens to "O'Connor" and friends.

[adrianeboyd]

The only limit is on how long a single substring of the same repeated type like x can be.

It's hard to describe this succinctly, but here's the description from Lexeme.shape in the docs:

Transform of the word’s string, to show orthographic features. Alphabetic characters are replaced by x or X, and numeric characters are replaced by d, and sequences of the same character are truncated after length 4. For example,"Xxxx"or"dd".

The implementation is here:

spaCy/spacy/lang/lex_attrs.py

Lines 118 to 142 in 7369832

	def word_shape(text: str) -> str:
	if len(text) >= 100:
	return "LONG"
	shape = []
	last = ""
	shape_char = ""
	seq = 0
	for char in text:
	if char.isalpha():
	if char.isupper():
	shape_char = "X"
	else:
	shape_char = "x"
	elif char.isdigit():
	shape_char = "d"
	else:
	shape_char = char
	if shape_char == last:
	seq += 1
	else:
	seq = 0
	last = shape_char
	if seq < 4:
	shape.append(shape_char)
	return "".join(shape)

If you want, you can replace this with your own shape method by using a custom language.

[rafelafrance]

Lexeme.shape... I did not find that, my bad, I assumed it would be in Usage or under Token.shape because that's what the shape was hanging off of. Lesson learned: Look at ALL of the references in the search before posting.

Actually, the text explanation is quite clear, except for the examples. Maybe an example of something along the lines of: the shape of "a123456789z" is truncated to "x9999x" but that's obviously your call.

If you want, you can replace this with your own shape method by using a custom language.

I'm just guessing here, but I suppose you can base this custom language off of one of the stock ones. I'll keep that in mind but the reason for using spaCy is that I wanted to move away from doing it all myself via my own home-cooked (aka half-baked) rule generator. This mosaic of rules and statistical methods I'm writing is a steppingstone to using a true model. I am not a fan of writing rules and the first thing I do on my base repo is warn people away from relying too much on rules. But here I am. Anyhoo...

Thank you again for another patience, informative and, given that it was an RTFM, polite reply.

[adrianeboyd]

No worries, we know that this kind of thing is a bit hard to find in the docs. If you just want to modify SHAPE, it does make sense to base a custom language on an existing one. https://spacy.io/usage/linguistic-features#language-subclass, but you want to set custom lex_attr_getters instead of stop_words. Most of the current language defaults show an example for this for LIKE_NUM.