docs: blog How to scrape Bluesky with Python by Mantisus · Pull Request #2784 · apify/crawlee

Mantisus · 2024-12-28T03:59:07Z

souravjain540

Pretty nice one. Add a few comments.

Also please follow this: https://www.notion.so/apify/Apify-tone-and-style-cheat-sheet-0fe6873372e44d88a1bd029d5fd76cea

Basic rules for writing, A is big in Actor always, never use Title case in titles when writing for Apify i.e, Making An BlueSky Actor Using Crawlee -> Making an BlueSky Actor using Crawlee.

and few more attached. please fix all of them.

And also please try always to link to the relevant docs/blog/resources by Apify or Crawlee wherever possible.

Missing section for GitHub Star CTA too.

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

souravjain540 · 2025-02-17T09:38:47Z

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

+
+### Project setup
+
+1. If you don't have UV installed yet, follow the [guide](https://docs.astral.sh/uv/getting-started/installation/) or use this command:


a little about UV?

i mean i dont know as a new reader what it is and why we need it

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

souravjain540 · 2025-02-17T09:41:43Z

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

+When first exploring Bluesky, it might be disconcerting to find that the [main page](https://bsky.app/) lacks a search function without authentication. The same applies when trying to access individual [posts](https://bsky.app/profile/github-trending-js.bsky.social/post/3ldbe7b3ict2v).
+
+Even if you navigate directly to the [search page](https://bsky.app/search?q=apify), while you'll see data, you'll encounter a limitation - the site doesn't allow viewing results beyond the first page.
+
+Fortunately, Bluesky provides a well-documented [API](https://docs.bsky.app/docs/get-started) that's accessible to any registered user without additional permissions. This is what we'll use for data collection.


maybe adding screenshots will explain more

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

souravjain540 · 2025-02-17T18:05:41Z

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

+
+### 5. Saving data to files
+
+For saving results, we'll use the `write_to_json` method in Dataset.


link to the method doc

souravjain540 · 2025-02-17T18:09:53Z

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

+---
+
+[Bluesky](https://bsky.app/) is an emerging social network developed by former members of the [Twitter](https://x.com/) development team. The platform has been showing significant growth recently, reaching 132.9 million visits according to [SimilarWeb](https://www.similarweb.com/website/bsky.app/#traffic). Like Twitter, Bluesky generates a vast amount of data that can be used for analysis. In this article, we'll explore how to collect this data using [Crawlee for Python](https://github.com/apify/crawlee-python).
+


missing part where you list all the sections of the blog from intro to making an Actor.

souravjain540 · 2025-02-17T18:10:56Z

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

+
+![Users Example](./img/users.webp)
+
+## Create Apify actor for Bluesky crawler


link to what is an Actor, and a little explanation why are we making Actor, because its the easiest way to deploy a software on cloud, etc, etc.

also loved it :)

souravjain540 · 2025-02-17T18:11:26Z

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

+View results in the Dataset:
+
+![Dataset Results](img/actor_results.webp)
+


maybe also show how to publish it on Apify Store

Co-authored-by: Saurav Jain <[email protected]>

souravjain540 · 2025-03-12T06:48:30Z

@janbuchar whenever you have time, can you please see if its good to go or not :)

janbuchar

Overall, the article is fine, maybe a bit more code-heavy than what we usually do? Other than that, I have just a couple of comments. Nice job!

janbuchar · 2025-03-17T13:42:13Z

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

+
+        # Variables for storing session data
+        self._domain: str | None = None
+        self._did: str | None = None


This deserves a better name.

Maybe it would look better according to the fields in the API response

janbuchar · 2025-03-17T13:43:27Z

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

+    self._users = await Dataset.open(name='users')
+    self._posts = await Dataset.open(name='posts')


If you run this on Apify, the datasets will be shared with previous runs of the crawler. Perhaps you could erase them beforehand?

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

janbuchar · 2025-03-17T13:46:04Z

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

+        # Add user request if not already added in current context
+        if post['author']['did'] not in user_requests:
+            user_requests[post['author']['did']] = Request.from_url(
+                url=f'{self._domain}/xrpc/app.bsky.actor.getProfile?actor={post["author"]["did"]}',


Perhaps you could use yarl here as well?

janbuchar · 2025-03-17T14:04:12Z

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

+    Creates a crawler instance, manages the session, and handles the complete
+    crawling lifecycle including proper cleanup on completion or error.
+    """
+    crawler = BlueskyCrawler()


I'm not a huge fan of the name BlueskyCrawler - the fact that it wraps HttpCrawler makes things a bit confusing. Also, needing to call init_crawler , save_data, etc., from the outside is not too convenient - maybe there could be just a single method on the wrapper class, should you decide to keep it?

Changed the naming so it wouldn't be so similar to the Crawlee naming. :)

I think calling methods from the outside will make it a bit easier to understand for users unfamiliar with Crawlee.

Co-authored-by: Jan Buchar <[email protected]>

janbuchar · 2025-03-18T10:41:49Z

Looks like there are some broken links - can you fix them?

janbuchar · 2025-03-18T10:45:29Z

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

+
+        data = response.json()
+
+        self._service_edpoint = data['didDoc']['service'][0]['serviceEndpoint']


typo in attribute name

janbuchar · 2025-03-18T10:45:50Z

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

+    """A crawler class for extracting data from Bluesky social network using their official API.
+
+    This crawler manages authentication, concurrent requests, and data collection for both
+    posts and user profiles. It uses separate datasets for storing post and user information.
+    """


The docblock should reflect the new name

janbuchar

I noticed a typo, otherwise it looks good to me.

janbuchar · 2025-03-18T12:14:06Z

website/blog/2025/02-12-scrape-bluesky-using-python/index.md

    posts = []

-    prfile_url = URL(f'{self._service_edpoint}/xrpc/app.bsky.actor.getProfile')
+    prfile_url = URL(f'{self._service_endpoint}/xrpc/app.bsky.actor.getProfile')


Mantisus added 5 commits December 28, 2024 03:58

docs: blog How to scrape Bluesky with Python

7f92da5

Merge branch 'master' into blog-bsky-crawler

9290ba4

update folder

ef96744

fix image text

00e0a47

add section for apify actor

8d00ab3

souravjain540 reviewed Feb 17, 2025

View reviewed changes

Mantisus and others added 6 commits February 17, 2025 20:18

Update website/blog/2025/02-12-scrape-bluesky-using-python/index.md

ea3fb7b

Co-authored-by: Saurav Jain <[email protected]>

Update website/blog/2025/02-12-scrape-bluesky-using-python/index.md

6739767

Co-authored-by: Saurav Jain <[email protected]>

Update website/blog/2025/02-12-scrape-bluesky-using-python/index.md

cb73315

Co-authored-by: Saurav Jain <[email protected]>

Update website/blog/2025/02-12-scrape-bluesky-using-python/index.md

e8a2fc9

Co-authored-by: Saurav Jain <[email protected]>

update bluesky article

f7a4fac

add image

15cf432

souravjain540 marked this pull request as ready for review March 12, 2025 06:47

souravjain540 requested a review from janbuchar March 12, 2025 06:48

janbuchar reviewed Mar 17, 2025

View reviewed changes

Mantisus and others added 2 commits March 17, 2025 16:26

Update website/blog/2025/02-12-scrape-bluesky-using-python/index.md

dbb40dc

Co-authored-by: Jan Buchar <[email protected]>

update code

d2ab768

janbuchar reviewed Mar 18, 2025

View reviewed changes

Mantisus added 2 commits March 18, 2025 11:57

fix typo and naming

f33011c

check links

b1a3481

Mantisus requested a review from janbuchar March 18, 2025 12:16

janbuchar approved these changes Mar 18, 2025

View reviewed changes

Mantisus added 5 commits March 18, 2025 12:35

fix typo

7bd822a

update date in folder

cbe3559

update date

03223e7

fix english

a0cb7e6

add CTA for github star

75564d2

souravjain540 merged commit f7f9728 into apify:master Mar 21, 2025
9 checks passed


		### Project setup

		1. If you don't have UV installed yet, follow the [guide](https://docs.astral.sh/uv/getting-started/installation/) or use this command:


		### 5. Saving data to files

		For saving results, we'll use the `write_to_json` method in Dataset.

		---

		[Bluesky](https://bsky.app/) is an emerging social network developed by former members of the [Twitter](https://x.com/) development team. The platform has been showing significant growth recently, reaching 132.9 million visits according to [SimilarWeb](https://www.similarweb.com/website/bsky.app/#traffic). Like Twitter, Bluesky generates a vast amount of data that can be used for analysis. In this article, we'll explore how to collect this data using [Crawlee for Python](https://github.com/apify/crawlee-python).


		![Users Example](./img/users.webp)

		## Create Apify actor for Bluesky crawler

		View results in the Dataset:

		![Dataset Results](img/actor_results.webp)

		self._users = await Dataset.open(name='users')
		self._posts = await Dataset.open(name='posts')


		data = response.json()

		self._service_edpoint = data['didDoc']['service'][0]['serviceEndpoint']

Conversation

Mantisus commented Dec 28, 2024

Uh oh!

souravjain540 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

souravjain540 commented Mar 12, 2025

Uh oh!

janbuchar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janbuchar commented Mar 18, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janbuchar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants