improve xml_dump processing. #43
Conversation
groceryheist
commented
Feb 17, 2017
groceryheist
commented
- add lang field to Iterator to store wiki lanaguge.
- extract lang from root tag of xml dumps
- load siteinfo when it comes in a 'siteinfo' element.
1. add lang field to Iterator to store wiki lanaguge. 2. extract lang from root tag of xml dumps 3. load siteinfo when it comes in a 'siteinfo' element.
|
Where are you seeing SHA1s in page elements? Are these in the Wikia dumps? |
|
Yes.
…On Fri, May 12, 2017 at 2:09 PM, Benjamin Mako Hill < ***@***.***> wrote:
Where are you seeing SHA1s in page elements? Are these in the Wikia dumps?
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
<#43 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAG6n41EespKzXw1MgyeZbnwNJGRqTc_ks5r5MpygaJpZM4MEr_2>
.
--
Nate
|
|
This is soon to be deprecated in favor of https://github.com/mediawiki-utilities/python-mwxml. Transition is slow because I'm the sole maintainer. I wonder if that other library works as expected here. |
| if tag == "page": | ||
| yield Page.from_element(sub_element) | ||
| else: | ||
| assert MalformedXML("Expected to see 'page'. " + | ||
| "Instead saw '{0}'".format(tag)) | ||
|
|
||
| @classmethod | ||
| def from_element(cls, element): | ||
|
|
||
| def from_element(cls, element, lang=None): |
There was a problem hiding this comment.
Why not extract the lang inside of this method?
mw/xml_dump/iteration/iterator.py
Outdated
| @@ -140,6 +145,9 @@ def load_site_info(cls, element): | |||
| namespaces = {} | |||
|
|
|||
| for sub_element in element: | |||
|
|
|||
| if sub_element.tag == 'siteinfo': | |||
| return(cls.load_site_info(sub_element)) | |||
There was a problem hiding this comment.
Also, what's going on here? Is there a inside of a tag?
There was a problem hiding this comment.
Do you mean is language data inside a tag?
Unfortunately not. The language is in the xml header, not a tag.
| if tag == "page": | ||
| yield Page.from_element(sub_element) | ||
| else: | ||
| assert MalformedXML("Expected to see 'page'. " + | ||
| "Instead saw '{0}'".format(tag)) | ||
|
|
||
| @classmethod | ||
| def from_element(cls, element): | ||
|
|
||
| def from_element(cls, element, lang=None): |
mw/xml_dump/iteration/iterator.py
Outdated
| @@ -140,6 +145,9 @@ def load_site_info(cls, element): | |||
| namespaces = {} | |||
|
|
|||
| for sub_element in element: | |||
|
|
|||
| if sub_element.tag == 'siteinfo': | |||
| return(cls.load_site_info(sub_element)) | |||
There was a problem hiding this comment.
Do you mean is language data inside a tag?
Unfortunately not. The language is in the xml header, not a tag.
| @@ -140,6 +143,9 @@ def load_site_info(cls, element): | |||
| namespaces = {} | |||
|
|
|||
| for sub_element in element: | |||
|
|
|||
| if sub_element.tag == 'siteinfo': | |||
| return cls.load_site_info(sub_element) | |||
There was a problem hiding this comment.
I'm confused about this line because it looks like you're expecting to find something like this:
<siteinfo>
<siteinfo> ... </siteinfo>
</siteinfo>
There was a problem hiding this comment.
OK i see now why this looks weird. I'll take another look. It's been a while since I did this so I'll see if it's a mistake or just something strange going on with Wikia dumps.
There was a problem hiding this comment.
Thanks a lot for taking a look at this.
|
I left one more note because I'm still confused about this one. Sorry if it seems nitpicky or if I'm missing something obvious :S |
