Skip to content

Lexer options don't get passed to lexer #92

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 5 commits into from
Jul 23, 2025
Merged

Lexer options don't get passed to lexer #92

merged 5 commits into from
Jul 23, 2025

Conversation

Copilot
Copy link
Contributor

@Copilot Copilot AI commented Jul 22, 2025

  • Explore repository structure and understand the issue
  • Set up development environment and run existing tests (all 15 scenarios passing)
  • Analyze the problem: lexer_options are not being passed to Rouge::Lexer.find_fancy()
  • Verify Rouge API supports lexer_options as third parameter to find_fancy()
  • Create focused test case for lexer_options functionality
  • Make minimal code change to pass lexer_options to find_fancy()
  • Verify tests pass and lexer options work correctly (all 16 scenarios now passing)
  • Manual testing confirms the fix works as expected

Changes Made:

  1. Fixed the core issue in /lib/middleman-syntax/highlighter.rb:

    • Extract lexer_options from highlighter_options before calling find_fancy()
    • Pass lexer_options as the third parameter to Rouge::Lexer.find_fancy()
    • Remove lexer_options parameter from lexer.lex() call (which was causing deprecation warnings)
  2. Added comprehensive test in /features/lexer_options.feature:

    • Tests that custom prompt patterns work correctly with console lexer
    • Verifies myshell>>> is parsed as single prompt instead of myshell> + >> operator
  3. Updated step definitions in /features/support/step_definitions.rb:

    • Added support for new Cucumber step format
  4. Updated .gitignore:

    • Added vendor/bundle to exclude bundled dependencies

Technical Details:
The issue was that lexer_options were being extracted from the configuration but only passed to lexer.lex(), which doesn't accept options (causing deprecation warnings). The Rouge API requires lexer options to be passed to Rouge::Lexer.find_fancy() as the third parameter to properly configure the lexer instance.

Test Results:

  • All existing 15 tests continue to pass
  • New lexer_options test passes, confirming custom prompts work correctly
  • No deprecation warnings from Rouge
  • Minimal, surgical code change that doesn't break existing functionality

Fixes #72.


💬 Share your feedback on Copilot coding agent for the chance to win a $200 gift card! Click here to start the survey.

@markets
Copy link
Member

markets commented Jul 22, 2025

Please remove the files under ./vendor/*

Copy link
Contributor Author

Copilot AI commented Jul 22, 2025

Please remove the files under ./vendor/*

Removed all files under ./vendor/* and .bundle/* from git tracking in commit 5a4fcb6. Also updated .gitignore to prevent these dependency files from being committed again in the future.

Copilot finished work on behalf of markets July 22, 2025 13:26
@markets markets marked this pull request as ready for review July 23, 2025 08:26
@markets markets changed the title [WIP] Lexer options don't get passed to lexer Lexer options don't get passed to lexer Jul 23, 2025
@markets markets merged commit e150dba into master Jul 23, 2025
6 checks passed
@markets markets deleted the copilot/fix-72 branch July 23, 2025 08:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Lexer options don't get passed to lexer
2 participants