Welcome to XLM_Rongorongo Discussions! #1
Replies: 3 comments 8 replies
-
|
To get started what convention for the Glyphs should we use in training? Also, how might we then begin to prepare data in a consistent (direction as read) format of the corpus based on this convention for the Glyphs? |
Beta Was this translation helpful? Give feedback.
-
|
Thanks for inviting me. I believe everyone agrees that Barthel's encoding should be replaced by a simplified encoding that correctly splits the glyphs into their components and accounts for allographs. So far, the most solid proposal is the one by Paul Horley in his recent publication of the corpus (https://rapanuipress.com/producto/rongorongo-2). I have code to transform the original text into Horley's encoding from my other repositories that we can reuse. |
Beta Was this translation helpful? Give feedback.
-
|
Thank you and you are more than welcome. My problem has been 1. my lack of coding and data preparation experience and 2. getting access to good sources. It seems Paul Horley's book is only available via the publisher and the ISBN doesn't seem to be available on most book selling platforms. I did think it the best source but I'm unsure of how to get a copy at my location yet. Though if you have access and the ability to transform the original text into Horley's encoding that would be amazing and make this a viable project. I find the topic of translating the Rongorongo language fascinating and this method the current most viable option of finding some patterns for further research. |
Beta Was this translation helpful? Give feedback.
-
|
What exact methods would you plan on using to convert the corpus of text and prepare it for training? Would you agree that the data should be prepared according to the reading direction so it is all read left to right? Also, would it be useful to add some of that code to this project? |
Beta Was this translation helpful? Give feedback.
-
|
As for the book, indeed I had to get it shipped from Chile and it took over a month to get here, but it was totally worth it! We can use the transcription as provided by CEIPP and the function(s) I have for replacing each glyph (including ligatures) with the correspoding equivalent in Horley's conversion tables. Typically, I end up with a dictionary where each key is a tablet and each value is another key-value pair with the line's name (e.g. Aa01) and the string or array of re-encoded glyphs. I have to familiarize myself with the format required by XLM. We can also join all the lines to avoid artificially breaking the text along line boundaries. It would all be left to right. One thing that we can discuss is whether to use the entire corpus or a subset (for example, what Horley calls the independent text - i.e. removing redundant parallel passages). Another issue is that some correspondences in Horley's encoding could be improved (I particularly like this paper by Pozdniakov which suggests the possibility that all glyphs in the 256 series are actually just allographs of the 200 series). But we can discuss those things later. Tonight I'm planning to make a pull request with a folder for data processing. |
Beta Was this translation helpful? Give feedback.
-
|
All of that sounds good, as you are much more adept in this area! My current contribution is just realising Facebook Research already created such a useful tool that seems to fit this purpose perfectly. I'm not really a "coder", ML expert or linguist but have long had an interest in this area and specifically Rongorongo as I believe it is one of the very few independently invented written langauges, contrary to some skeptics. We can try with different subsets of data maybe i.e. the examples you have suggest and also the entire corpus? I am aware for example there are some sources that seem to have punctuation. My view would be to remove punctutation if it is rare since it is such a small body of text to attempt this with, and then there are clearly elements that are decoration and not part of the text that also should not be included. |
Beta Was this translation helpful? Give feedback.
-
|
Explain the horley_encoding.csv to me a bit. I had a look and noticed a lot of question marks. Are these queried, placeholders or problems with uploading? What you've added thus far looks good for the data side. |
Beta Was this translation helpful? Give feedback.
-
|
The question marks are from the original mapping as it appears in the book. They are there for cases where glyph identification is not possible. This is something that needs to be improved - for example, the way I have the code now, illegible glyphs end up as "?" as well. We should definitely make a distinction (encoding the latter as "#", for example, which is Horley's proposal in the book). |
Beta Was this translation helpful? Give feedback.
-
|
Is there a way to do this that is best suited for XLM training as you don't want it finding a pattern in what are simply unrelated illegible or hard to identify glyphs? Maybe if we give each a unique non-latin alphabet designation? That way they won't "corrupt" the data. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
The XLM-R model is also hosted at huggingface and can be further finetuned https://huggingface.co/FacebookAI/xlm-roberta-large |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
👋 Welcome!
We’re using Discussions as a place to connect with other members of our community. We hope that you:
build together 💪.
To get started, comment below with an introduction of yourself and tell us about what you do with this community. Tell us why you are interested in the Rongorongo langauge and what, if any, expertise you hope to bring to this ML translation project!
Beta Was this translation helpful? Give feedback.
All reactions