Best way to loop through for a long list of location with geo-coordinates

[FayzulSaimun]

Hello! I have a list of locations with their respective geo-coordinates. Currently, I am running the following code:

for city in tqdm(batch):
        query = f"{city['city']}, {city['state_name']} karaoke"
        queries = [query]
        coordinate_string = f"{city['lat']},{city['lng']}"
        data = Gmaps.places(
            queries, fields=Gmaps.ALL_FIELDS, geo_coordinates=coordinate_string, zoom=16
        )

I tried this way also but it didn't work

queries = [{'keywords': 'erie pennsylvania karaoke', 'geo_coordinates': '42.10958,-80.104055', 'zoom': 16}]
Gmaps.places(queries, fields=Gmaps.ALL_FIELDS)

Can the full list be given to the scraper without the loop to minimize the run time? Currently, it takes a lot of time to get 100 location results.

[Chetan11-dev]

Yes, you need to use first code block, Also, in a few months, I plan to integrate kubernetes feature into the Botasaurus Framework. Once integrated, it will be made available as an API. and you can get 100's of keywords scraped in just 10 minutes. I am working on it. So just wait few months, and then you can scrape really really fast.

[FayzulSaimun]

Thanks for this tool. It helps a lot. Also please update the docker instruction. The scraper is not properly working on Linux server.

[Chetan11-dev]

use docker-compose build and docker-compose up it runs correctly

[FayzulSaimun]

I got this error using docker-compose up

[Chetan11-dev]

Fixed, repull the latest docker image and run

[Chetan11-dev]

Also Fayzul,

I came across your freelancing profile and noticed your expertise in web scraping, which is impressive. I'm curious to hear your thoughts on the Botasaurus Framework. Do you consider it a strong project?

[FayzulSaimun]

Hey Chetan,

I came across your Bose framework around 4-5 months ago, but I didn't get a chance to try it out. However, I recently used your Google Maps Scraper and I must say I am very impressed with your work. The tool is top-notch. Currently, I am trying to familiarize myself with Botasaurus and I am using it in a web scraper. While it is a strong project, I have noticed some issues with it. I will make note of these issues so that you can consider fixing them to make the project even stronger.

[Chetan11-dev]

Thank You!

[Chetan11-dev]

@FayzulSaimun, we have released v4 of botasaurus, I am really interested in knowing your review of botasaurus v4 as a expert web scraper. Could you visit https://github.com/omkarcloud/botasaurus and let me know?

[FayzulSaimun]

@Chetan11-dev Hello, I am currently using version 4 of Botasaurus. I have developed a few scrapers based on Botasaurus for my clients, and one of them is being used by several users. The best features are the parallel operations and proxy support. I also like the UI creation features and I am planning to give it a try soon.

[Chetan11-dev]

Yes Sure, the humaness of the Scrapers have been really upped as well, and is the best in js and python universe. Also let me know about the UI Feautres review.

[DanciVasile]

Hello, I am really interested in doing this too. I have a list of 50.000 locations, and 300 cities ready to be scrapped. However I need some help in making the right modifications in the code. If you want, I think it would be great to hop on a call on Discord and see what we can do together.

[FayzulSaimun]

Hey, if you're interested we can connect on Discord and can discuss more.

[DanciVasile]

Great, I am #boxi8220. I just had a striking good idea to implement, should not be hard but I believe it would increase drastically the chances of finding ideal leads.

[Kten987]

I don't know if this is what you're looking for, but I think you're trying to crawl data for many different coordinates. If so, you can refer to the following:

1. Add a for loop at line 326 of gmaps.py
   

2. Add your input variable geo_coordinates in main.py
   For example
   geo_coordinates = ["10.060336116445198, 105.7793725804415",
   "10.041776517690092, 105.79192278333913",
   "10.04857477707304, 105.78772157139154",
   "10.046438333359838, 105.78541529587908"]

[FayzulSaimun]

@Kten987 Thanks. I will try this method.

[Kten987]

@FayzulSaimun forgot you will also have to change the geo coordinates variable to list[str] instead of just [str]. At line 292 of gmaps.py

[Chetan11-dev]

@FayzulSaimun Do you know code/api that when given a location like Dhaka returns a list of multiple geo-coordinates for it?

[FayzulSaimun]

I am using Worldcitydb.com to get the location and their geo-coordinates.

[Kten987]

I use platform.here.com to get geo-coordinates for the location I need. However, using it for free will only get 1000 requests a day. If you use more, you have to pay or you can register for many different accounts.

[Chetan11-dev]

Thanks for letting me know.