Welcome to HomeHarvest Discussions!

[cullenwatson]

Feel free to discuss real estate, issues, ideas, etc

[awb715]

Cool website. Love the idea of what bunsly is. I've had similar ideas.

[RSpencerFink]

@cullenwatson, I'm curious if you have any ideas what to do when a search returns more than 10k listings. Is there a way to paginate the results?

[cullenwatson]

try search by zips not city. or is that still more than 10k

[RSpencerFink]

Yeah, that was my initial approach but to do a big state like CA it was taking a long time. I'll give it another go

[Spacewalker215]

I really appreciate the work you all are doing—this repo has been incredibly useful.

I had an idea that might help improve the scraping process. By integrating Botasaurus, scraping could become much harder to detect, reducing the need for constant proxy rotation since Botasaurus handles that automatically.

Would love to hear your thoughts on this.

[cullenwatson]

If the proxies work on realtor, yea would be good. Have you tested

[Spacewalker215]

Yes they do, I tested it out, and it isn't that difficult to implement.

[azchandev]

This is awesome! Been wanting to implement something like this for a while but through scrapping.
Question:
How did you find those search GQL endpoints?

    SEARCH_GQL_URL = "https://www.realtor.com/api/v1/rdc_search_srp?client_id=rdc-search-new-communities&schema=vesta"
    PROPERTY_URL = "https://www.realtor.com/realestateandhomes-detail/"
    PROPERTY_GQL = "https://graph.realtor.com/graphql"
    ADDRESS_AUTOCOMPLETE_URL = "https://parser-external.geo.moveaws.com/suggest"

Some sort of reverse engineering?

[cullenwatson]

just go to chrome dev tools, pretty simple