Please add support for parsing messages from "Live chat" (not "Top chat")

[rekmixa]

This package parses messages from "Top chat" only. How to parse all messages from "Live chat"?

[neverclear86]

I'll think about it!
Thank you.☺

[faab007nl]

Did you find a solution @rekmixa ?

[rekmixa]

Did you find a solution @rekmixa ?

No ideas. I dug into api method for getting chat for a long time and did not understand by what parameter messages are filtered

[rekmixa]

@neverclear86 any thoughts?

[rekmixa]

@neverclear86 I think it somehow depends on parameter "continuation". but I'm not sure about it

[neverclear86]

Good morning. I just woke up🌅
I have some time today, so i'm gonna look it.

[neverclear86]

sorry.
I looked it up
but i really can't find it😭

[ENBYSS]

An option could be to use a headless browser like puppeteer or PhantomJS. I'm not experienced with them, but theoretically if a headless browser launches the live page they can then simulate a click on the "Top chat" toggle and switch it to "Live chat" - not sure how realistic this is though.

[eslym]

i found something, inside a script tag, the ytInitialData is defined, and a part of the data defined the continuations for 2 different mode

Screenshot

【Update】
i compared 3 sets of continuation and found something more interesting

Screenshot

i made a POC code snippet for this

let continuation: string

let cursor = data.indexOf('viewSelector');

try {
    if (cursor === -1) {
        throw 'break';
    }
    cursor += 12; // length of "viewSelector"
    while (data[cursor] !== '{') cursor++; // find the starting of json
    let json = "{";
    let open = 1;
    do {
        json += data[++cursor];
        if (data[cursor] === '{') {
            open++;
        } else if  (data[cursor] === '}')  {
            open--;
        }
    } while (open > 0 && cursor < data.length);
    let vsData = parse(json); // use json5 parse to parse this portion of data
    continuation = vsData.sortFilterSubMenuRenderer.subMenuItems[1].continuation.reloadContinuationData.continuation;
} catch (_) {
    // use the fallback method to get continuation
    const continuationResult = data.match(/['"]continuation['"]:\s*['"](.+?)['"]/)
    if (continuationResult) {
        continuation = continuationResult[1]
    } else {
        throw new Error("Continuation was not found")
    }
}

[rekmixa]

An option could be to use a headless browser like puppeteer or PhantomJS. I'm not experienced with them, but theoretically if a headless browser launches the live page they can then simulate a click on the "Top chat" toggle and switch it to "Live chat" - not sure how realistic this is though.

I've already tried this, it didn't help

[neverclear86]

Even if possible, we currently have no plans to use crawler libraries such as PhantomJS. 😩

@eslym
This content may be useful 👍
I'll check again when I have time

[eslym]

i wrote a crawler function to extract useful data from youtube link which may help in this project

index.ts

const axios = require('axios');
const getMetaData = require('metadata-scraper');
const JSON5 = require('json5');

interface LiveStreamabilityRenderer {
    offlineSlate: {
        liveStreamOfflineSlateRenderer: {
            scheduledStartTime: string;
        }
    }
}

interface SubMenuItem {
    continuation: {
        reloadContinuationData: {
            continuation: string;
        }
    }
}

interface ViewSelector {
    sortFilterSubMenuRenderer: {
        subMenuItems: SubMenuItem[];
    }
}

export interface Thumbnail {
    url: string
    width?: number
    height?: number
}

export interface VideoDetails {
    videoId: string;
    title: string;
    lengthSeconds: string;
    isLive?: boolean;
    keywords?: string[];
    channelId: string;
    isOwnerViewing: boolean;
    shortDescription: string;
    isCrawlable: boolean;
    isLiveDvrEnabled?: boolean;
    thumbnail: {
        thumbnails: Thumbnail[];
    };
    liveChunkReadahead?: number;
    allowRatings: boolean;
    viewCount: string;
    author: string;
    isLowLatencyLiveStream: boolean;
    isPrivate: boolean;
    isUnpluggedCorpus: boolean;
    latencyClass: string;
    isLiveContent: boolean;
}

export interface LiveAbility {
    scheduled: Date | false;
    continuations: string[];
}

export interface CrawlResult {
    details: VideoDetails,
    apiKey: string | false,
    liveAbility: LiveAbility | false,
    clientVersion: string | false,
}

const quotes: Readonly<Record<string, boolean>> = Object.freeze({"'": true, '"': true});

const allowedHosts: Readonly<Record<string, boolean>> = Object.freeze({
    'youtube.com': true,
    'www.youtube.com': true,
    'youtu.be': true,
});

function findString(source: string, index: number = 0): [number, number] | null{
    while(!quotes[source[index]] && index < source.length) index++;
    if(index >= source.length){
        return null;
    }
    let start = index;
    let quote = source[index];
    while((++index) < source.length && source[index] !== quote){
        if(source[index] === '\\'){
            index++; // escape
        }
    }
    if(source[index] === quote){
        return [start, index + 1];
    }
    return null;
}

function findObject(source: string, index: number = 0): [number, number] | null{
    let start = index = source.indexOf('{', index);
    if(start === -1){
        return null;
    }
    let open = 1;
    while((++index) < source.length && open > 0){
        if(source[index] === '{'){
            open++;
        } else if (source[index] === '}'){
            open--;
        } else if (quotes[source[index]]){ // when string
            let loc = findString(source, index);
            if(!loc){
                return null;
            }
            index = loc[1] - 1;
        }
    }
    if(open === 0){
        return [start, index];
    }
    return null;
}

function extractObject<T extends object>(keyword: string, source: string, index: number = 0): {result: T, location: [number, number]} | null{
    index = source.indexOf(keyword, index);
    if(index < 0){
        return null;
    }
    let loc = findObject(source, index);
    if(!loc){
        return null;
    }
    try{
        let json = source.slice(...loc);
        return {
            location: loc,
            result: JSON5.parse(json),
        }
    }catch(e){}
    return null;
}

function extractString(keyword: string, source: string, index: number = 0): {result: string, location: [number, number]} | null {
    index = source.indexOf(keyword, index);
    if(index < 0){
        return null;
    }
    let loc = findString(source, index);
    if(!loc){
        return null;
    }
    try{
        let json = source.slice(...loc);
        return {
            location: loc,
            result: JSON5.parse(json),
        }
    }catch(e){}
    return null;
}

async function crawlDetails(url: string): Promise<CrawlResult> {
    let uri = new URL(url);
    if(!allowedHosts[uri.hostname.toLowerCase()]){
        throw new Error('Not youtube link!');
    }
    try{
        let res = await axios.get(url);
        let meta = await getMetaData({html: res.data});
        if(meta.type === 'profile'){
            return await crawlDetails(`${meta.url}/live`);
        }
        if(meta.type === 'video.other' || meta.url.startsWith('https://www.youtube.com/watch?v=')) {
            let result = extractObject<VideoDetails>("videoDetails", res.data);
            if(!result){
                throw new Error('Failed to extract data');
            }
            let details = result.result;
            let findString = extractString(":", res.data, res.data.indexOf("INNERTUBE_API_KEY"));
            let apiKey = findString ? findString.result : false;
            findString = extractString(":", res.data, res.data.indexOf("clientVersion"));
            let clientVersion = findString ? findString.result : false;
            let liveAbility: LiveAbility | false = false;
            let findStream = extractObject<LiveStreamabilityRenderer>("liveStreamabilityRenderer", res.data);
            if(findStream) {
                liveAbility = {
                    scheduled: false,
                    continuations: [],
                };
                let streamAbility = findStream.result;
                if(streamAbility.offlineSlate){
                    liveAbility.scheduled = new Date(Number.parseInt(streamAbility.offlineSlate.liveStreamOfflineSlateRenderer.scheduledStartTime) * 1000);
                }
                let findViewSelector = extractObject<ViewSelector>("viewSelector", res.data);
                if(findViewSelector){
                    liveAbility.continuations = findViewSelector.result.sortFilterSubMenuRenderer.subMenuItems.map(i => i.continuation.reloadContinuationData.continuation);
                }
            }
            return {
                details, liveAbility, apiKey, clientVersion
            };
        }
    } catch (e){
        throw new Error("Error while fetching info.");
    }
    throw new Error('Cannot find a video or live from the url.');
}

(async _ => {
    let details = await crawlDetails('https://www.youtube.com/watch?v=hxWxUcVNgr8');
    console.log(details);
})();