fix: 🐛 Hotfix. Change selectors due to new html structure #905

chakaponi · 2024-11-21T18:56:17Z

LinkedIn has changed their HTML structure therefore app stopped to apply to jobs

udpate v3.1.0 with Release/v11.15.2024

repalced union with base class.

Integration with AI/ML API

Add GroqAIModel support

Some providers (as OpenAI) support reducing cost of calls by caching repetitive content, but the feature works only with the initial portion (prefix) of prompts

included links to AI/ML API docs and the endpoint to be used in the code.

Add AI/ML API info

…-caching perf: ⚡ Optimize prompt for better caching

(cherry picked from commit 6b23715)

(cherry picked from commit c75f78f)

chakaponi · 2024-11-21T18:59:13Z

Sorry, I supposed to begin branch name with "hotfix" instead of "bug"

chakaponi · 2024-11-21T19:03:45Z

Currently, only the selectors that I have changed have been tested for changes on LinkedIn side. There might be more

surapuramakhil · 2024-11-21T19:46:59Z

@chakaponi released hotfix, I just applied - automation didn't break. can you explain on impact on description

chakaponi · 2024-11-24T16:19:26Z

@surapuramakhil Discord's having auth problems so I can't use it. Only telegram

surapuramakhil · 2024-11-24T16:34:38Z

@chakaponi not able to find you. can you ping me on telegram

surapuramakhil · 2024-11-24T17:23:26Z

thread link for sharing your jobs view structure, for having a fix which works for all.

https://discord.com/channels/1300208460788400159/1310293395226230896

run this command on dev tools console

copy($('#main'))

you can run copy($('#main')) in your chrome dev tools on, for example, this page: https://www.linkedin.com/jobs/search/?currentJobId=4042558197&distance=10&f_E=4&f_LF=f_AL&f_WT=2&keywords=JavaScript%20Developer&location=Mexico It will copy element which contain currently open job and block with list of jobs (left sidebar)

Current solution is fine for hotfix, but not as constant

surapuramakhil · 2024-11-24T21:30:53Z

@chakaponi can you test hotfix branch? and see if this required

feder-cr · 2024-11-24T22:00:52Z

@chakaponi hello, if it is no longer useful remove it with no problem!

squash

chakaponi · 2024-11-25T00:06:21Z

@surapuramakhil Hotfix branch has been merged into mine with some changes and rewritten tests. All tests pass. Works in normal environment for me.

src/ai_hawk/job_manager.py

tests/test_aihawk_job_manager.py

surapuramakhil · 2024-11-25T01:40:30Z

@chakaponi can you comment on #927 #926 so that I can assign those to you.

Shuo-Wang-UCBerkeley · 2024-11-25T05:17:21Z

Hi guys, I git clone the hotfix/v2024.11.23, but still does not work.

surapuramakhil · 2024-11-25T13:30:54Z

@chakaponi Next time don't force push. It corrupts subsequent processes.

Shuo-Wang-UCBerkeley · 2024-11-25T14:06:09Z

Thank you for your effort @surapuramakhil , may I ask if the issue has fixed?
Do I need to get clone again for the AIHawk-FOSS:hotfix/v2024.11.23?

@chakaponi Next time don't force push. It corrupts subsequent processes.

surapuramakhil · 2024-11-25T14:09:26Z

Thank you for your effort @surapuramakhil , may I ask if the issue has fixed?
Do I need to get clone again for the AIHawk-FOSS:hotfix/v2024.11.23?

just update your main.

git fetch 
git pull

if you face errors then clone again.

chakaponi · 2024-11-25T14:36:11Z

src/ai_hawk/job_manager.py

-            jobs_container = self.driver.find_element(By.CLASS_NAME, 'scaffold-layout__list-container')
+            # XPath query to find the ul tag with class scaffold-layout__list-container
+            jobs_xpath_query = "//ul[contains(@class, 'scaffold-layout__list-container')]"
+            jobs_container = self.driver.find_element(By.XPATH, jobs_xpath_query)


They unlikely will keep this class with not ul tag so both approaches are equal

chakaponi

Some bad approaches is choosen. Needs partial revert

Please notice me after revert so I will be able to be sure that we are getting working approach

chakaponi · 2024-11-25T14:37:45Z

src/ai_hawk/job_manager.py

@@ -262,7 +264,7 @@ def get_jobs_from_page(self, scroll=False):
                browser_utils.scroll_slow(self.driver, jobs_container_scrolableElement)
                browser_utils.scroll_slow(self.driver, jobs_container_scrolableElement, step=300, reverse=True)

-            job_element_list = jobs_container.find_elements(By.CSS_SELECTOR, 'div[data-job-id]')
+            job_element_list = jobs_container.find_elements(By.XPATH, ".//li[contains(@class, 'jobs-search-results__list-item') and contains(@class, 'ember-view')]")


I would say that rely on "ember-view" instead of "data-job-id" more likely to cause problems again so I would revert this change

chakaponi · 2024-11-25T14:40:15Z

src/ai_hawk/job_manager.py

@@ -483,7 +473,7 @@ def job_tile_to_job(self, job_tile) -> Job:
            logger.warning("Job link is missing.")

        try:
-            job.company = job_tile.find_element(By.XPATH, './/span[contains(normalize-space(), " · ")]').text.split(' · ')[0].strip()
+            job.company = job_tile.find_element(By.XPATH, ".//div[contains(@class, 'artdeco-entity-lockup__subtitle')]//span").text


Not robust. Also advise to revert

chakaponi · 2024-11-25T14:42:20Z

src/ai_hawk/job_manager.py

@@ -500,15 +490,21 @@ def job_tile_to_job(self, job_tile) -> Job:
            logger.warning(f"Failed to extract job ID: {e}", exc_info=True)

        try:
-            job.location = job_tile.find_element(By.XPATH, './/span[contains(normalize-space(), " · ")]').text.split(' · ')[-1].strip()
+            job.location = job_tile.find_element(By.CLASS_NAME, 'job-card-container__metadata-item').text


Simply doesn't work. If we check, 'job-card-container__metadata-item' never follow as to location, but to salary fork. Has to be reverted with no doubt

chakaponi · 2024-11-25T14:43:36Z

src/ai_hawk/job_manager.py

        try:
-            job.apply_method = job_tile.find_element(By.XPATH, ".//div[contains(@class, 'job-card-container__job-insight-text') and normalize-space() = 'Easy Apply']").text
+            job_state = job_tile.find_element(By.XPATH, ".//ul[contains(@class, 'job-card-list__footer-wrapper')]//li[contains(@class, 'job-card-container__apply-method')]").text


Doesn't work. Return empty array all of the time. Revert

chakaponi · 2024-11-25T14:59:21Z

@surapuramakhil If you want to keep your approach, you can rewrite code a little and keep both so if one class returns an empty array we could try other one right after

surapuramakhil · 2024-11-25T15:27:53Z

My Apologies @chakaponi as this was breaking change, I took decision for "quicker release, for pushing fix which at least work, than no fix at all".

we can update this in release branch (more robust)
only if main breaks - we can push this two second hotfix

Shuo-Wang-UCBerkeley · 2024-11-25T15:29:56Z

Thank you for your great work on this. @surapuramakhil and @chakaponi
May I ask which branch works now?
so I can git clone and start the job application.

chakaponi · 2024-11-25T15:33:23Z

My Apologies @chakaponi as this was breaking change, I took decision for "quicker release, for pushing fix which at least work, than no fix at all".

we can update this in release branch (more robust)

only if main breaks - we can push this two second hotfix

Don't get it. My fix was ready and tested so why wouldn't have used it
Anyways, we need second hotfix as long as current fix doesn't work

chakaponi · 2024-11-25T15:35:53Z

Thank you for your great work on this. @surapuramakhil and @chakaponi May I ask which branch works now? so I can git clone and start the job application.

Main branch may work if you are lucky enough. If not, try this branch, but with reverted commit with hash 22f2c3b
We really would appreciate if you try both and tell us which ones works for you. Thank you

surapuramakhil · 2024-11-25T15:38:29Z

Anyways, we need second hotfix as long as current fix doesn't work

created a poll to see if the main branch works or not. #919 (comment)

surapuramakhil · 2024-11-26T03:02:39Z

@chakaponi I have created the hotfix branch. Please raise a PR to the hotfix branch.

…nt-selectors fix: 🐛 Hotfix. Change selectors due to new html structure Co-authored-by: chakaponi <[email protected]>

waterstark and others added 21 commits November 5, 2024 16:03

Added integration with aiml

91c7b17

Update README.md

e3b861c

Add GroqAIModel support

b64eb03

Merge pull request AIHawk-FOSS#824 from AIHawk-co/release/v11.15.2024

69e45b6

udpate v3.1.0 with Release/v11.15.2024

README.md updated for GROQ API

b8d7ffa

Update llm_manager.py

3a9e5b7

repalced union with base class.

Resolve conflicts

8c10dcd

Merge branch 'release/v11.15.2024' into integration-with-aiml

ed337c7

Merge branch 'release/v3.1.0' into integration-with-aiml

6fc96ed

Merge pull request AIHawk-FOSS#757 from waterstark/integration-with-aiml

be54105

Integration with AI/ML API

Merge remote-tracking branch 'upstream/release/v3.1.0'

72027a8

groq added in constants

6280296

just groq version added in requirements

306fca6

Merge pull request AIHawk-FOSS#822 from innovatorved/main

1c1b7f3

Add GroqAIModel support

perf: ⚡ Optimize prompt for better caching

fd69fc0

Some providers (as OpenAI) support reducing cost of calls by caching repetitive content, but the feature works only with the initial portion (prefix) of prompts

Add AI/ML API info

c5955bd

included links to AI/ML API docs and the endpoint to be used in the code.

Merge pull request AIHawk-FOSS#882 from OctavianTheI/patch-1

0330af9

Add AI/ML API info

Merge branch 'release/v4.1.0' into fix/improve-prompts-caching

e18e2e6

Merge pull request AIHawk-FOSS#872 from chakaponi/fix/improve-prompts…

e038183

…-caching perf: ⚡ Optimize prompt for better caching

"Log time in Job application result"

1f68c27

(cherry picked from commit 6b23715)

"Refactor logging current time"

a349a4a

(cherry picked from commit c75f78f)

chakaponi force-pushed the bug/incorrect-element-selectors branch from bc83944 to a603455 Compare November 21, 2024 18:56

chakaponi mentioned this pull request Nov 21, 2024

[FEATURE]: Separate Browser Automation code #906

Open

chakaponi changed the title ~~fix: 🐛 Change selectors due to new html structure~~ fix: 🐛 Hotfix. Change selectors due to new html structure Nov 21, 2024

surapuramakhil changed the base branch from main to release/v4.1.0 November 21, 2024 19:47

Tgenz1213 added 2 commits November 21, 2024 14:09

Add files via upload

48481b5

Update pull_request_template.md

f21530d

fix: Add classes for temporary solution

2dd187a

Current solution is fine for hotfix, but not as constant

chakaponi added 3 commits November 25, 2024 02:16

Merge branch 'bug/incorrect-element-selectors' into hotfix/v2024.11.23

76b06ae

Rewrite test, merge solution with changes, deduplicate code

dba7b10

squash

Merge branch 'hotfix/v2024.11.23' into bug/incorrect-element-selectors

170c413

style: 🎨 Fix naming back

8411cf8

surapuramakhil suggested changes Nov 25, 2024

View reviewed changes

reveiw changes

22f2c3b

surapuramakhil approved these changes Nov 25, 2024

View reviewed changes

surapuramakhil merged commit 65f8ac9 into AIHawk-FOSS:hotfix/v2024.11.23 Nov 25, 2024
1 check passed

chakaponi commented Nov 25, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: 🐛 Hotfix. Change selectors due to new html structure #905

fix: 🐛 Hotfix. Change selectors due to new html structure #905

chakaponi commented Nov 21, 2024

chakaponi commented Nov 21, 2024

chakaponi commented Nov 21, 2024

surapuramakhil commented Nov 21, 2024 •

edited

Loading

chakaponi commented Nov 24, 2024

surapuramakhil commented Nov 24, 2024

surapuramakhil commented Nov 24, 2024

surapuramakhil commented Nov 24, 2024

feder-cr commented Nov 24, 2024

chakaponi commented Nov 25, 2024

surapuramakhil commented Nov 25, 2024

Shuo-Wang-UCBerkeley commented Nov 25, 2024

surapuramakhil commented Nov 25, 2024

Shuo-Wang-UCBerkeley commented Nov 25, 2024

surapuramakhil commented Nov 25, 2024 •

edited

Loading

chakaponi Nov 25, 2024

chakaponi left a comment •

edited

Loading

chakaponi Nov 25, 2024

chakaponi Nov 25, 2024

chakaponi Nov 25, 2024

chakaponi Nov 25, 2024

chakaponi commented Nov 25, 2024

surapuramakhil commented Nov 25, 2024

Shuo-Wang-UCBerkeley commented Nov 25, 2024

chakaponi commented Nov 25, 2024

chakaponi commented Nov 25, 2024 •

edited

Loading

surapuramakhil commented Nov 25, 2024

surapuramakhil commented Nov 26, 2024 •

edited

Loading

fix: 🐛 Hotfix. Change selectors due to new html structure #905

fix: 🐛 Hotfix. Change selectors due to new html structure #905

Conversation

chakaponi commented Nov 21, 2024

chakaponi commented Nov 21, 2024

chakaponi commented Nov 21, 2024

surapuramakhil commented Nov 21, 2024 • edited Loading

chakaponi commented Nov 24, 2024

surapuramakhil commented Nov 24, 2024

surapuramakhil commented Nov 24, 2024

surapuramakhil commented Nov 24, 2024

feder-cr commented Nov 24, 2024

chakaponi commented Nov 25, 2024

surapuramakhil commented Nov 25, 2024

Shuo-Wang-UCBerkeley commented Nov 25, 2024

surapuramakhil commented Nov 25, 2024

Shuo-Wang-UCBerkeley commented Nov 25, 2024

surapuramakhil commented Nov 25, 2024 • edited Loading

chakaponi Nov 25, 2024

Choose a reason for hiding this comment

chakaponi left a comment • edited Loading

Choose a reason for hiding this comment

chakaponi Nov 25, 2024

Choose a reason for hiding this comment

chakaponi Nov 25, 2024

Choose a reason for hiding this comment

chakaponi Nov 25, 2024

Choose a reason for hiding this comment

chakaponi Nov 25, 2024

Choose a reason for hiding this comment

chakaponi commented Nov 25, 2024

surapuramakhil commented Nov 25, 2024

Shuo-Wang-UCBerkeley commented Nov 25, 2024

chakaponi commented Nov 25, 2024

chakaponi commented Nov 25, 2024 • edited Loading

surapuramakhil commented Nov 25, 2024

surapuramakhil commented Nov 26, 2024 • edited Loading

surapuramakhil commented Nov 21, 2024 •

edited

Loading

surapuramakhil commented Nov 25, 2024 •

edited

Loading

chakaponi left a comment •

edited

Loading

chakaponi commented Nov 25, 2024 •

edited

Loading

surapuramakhil commented Nov 26, 2024 •

edited

Loading