codenib.languages

Central language metadata registry.

This module is intentionally declarative. The existing language support surfaces do not all use the same canonical key yet: for example agent compile keeps c distinct from cpp, while graph indexing routes c through the C/C++ backend. LanguageSpec records those per-surface choices in one place so new language work can add metadata without editing every router by hand.

Classes:

Name	Description
ScipColdStartOption	SCIP cold-start backend status for one language.
LanguageSpec	Declarative metadata for one CodeNib language family.
LanguageCapability	Derived support matrix row for one registered language.

Functions:

Name	Description
get_language_spec	Return the spec for a language key or general alias.
normalize_language	Normalize a raw language string to a registry key.
normalize_chunker_language	Normalize a raw language string for create_chunker.
get_chunker_spec	Return the LanguageSpec used by the chunker factory.
normalize_graph_language	Normalize a raw language string to the graph/LS backend key.
normalize_agent_language	Normalize a raw language string for agent compile scenarios.
supported_agent_languages	Return language scenario keys accepted by agent compile.
chunker_language_aliases	Return raw chunker aliases mapped to chunker language keys.
chunker_languages	Return canonical language keys accepted by repository chunking.
chunker_class_paths	Return chunker language keys mapped to chunker class paths.
chunker_class_path	Return the chunker class path for a raw chunker language.
graph_language_aliases	Return raw LS/graph aliases mapped to backend language keys.
agent_language_aliases	Return raw agent language aliases mapped to scenario keys.
extension_to_language_map	Return an extension-to-language map for the requested surface.
extensions_for_language	Return extensions for a language on the requested surface.
graph_extensions_by_language	Return graph backend language keys mapped to accepted extensions.
graph_indexer_paths	Return graph backend language keys mapped to cold-start indexer paths.
graph_decoder_paths	Return graph backend language keys mapped to graph decoder paths.
graph_cold_start_backends	Return graph backend language keys mapped to cold-start backend names.
graph_indexer_path	Return the cold-start indexer class path for a raw graph language.
graph_decoder_path	Return the graph decoder class path for a raw graph language.
graph_cold_start_backend	Return the cold-start backend name for a raw graph language.
scip_cold_start_options	Return languages mapped to configured SCIP cold-start options.
scip_cold_start_option	Return SCIP cold-start metadata for a language key or alias.
scip_candidate_indexer_paths	Return languages mapped to explicit candidate SCIP indexer class paths.
scip_candidate_indexer_path	Return an opt-in candidate SCIP indexer path for a language or alias.
scip_cold_start_command_for_language	Return the configured SCIP cold-start command for a language or alias.
lsp_language_id_for_language	Return the LSP language id for a raw graph language.
lsp_command_for_language	Return the configured LSP command for a raw graph language.
incremental_patcher_paths	Return graph backend language keys mapped to incremental patcher paths.
incremental_patcher_path	Return the incremental patcher class path for a raw graph language.
core_decoder_languages	Return languages accepted by the optional C++ core decoder.
language_capability_rows	Return a registry-derived language support matrix.

ScipColdStartOption dataclass

ScipColdStartOption(
    tool: str,
    status: ScipColdStartStatus,
    command: tuple[str, ...] = (),
    command_env: str | None = None,
    note: str = "",
)

SCIP cold-start backend status for one language.

active means the language's current cold-start graph backend is SCIP. candidate means a known SCIP indexer exists, but CodeNib has not yet promoted it to the primary backend because smoke, backend-alignment, decoder, and parity gates are still open.

LanguageSpec dataclass

LanguageSpec(
    key: str,
    display_name: str,
    aliases: tuple[str, ...] = (),
    chunker_language: str | None = None,
    chunker_aliases: tuple[str, ...] = (),
    chunker_class: str | None = None,
    chunker_pass_language: bool = False,
    chunk_extensions: tuple[str, ...] = (),
    gt_language: str | None = None,
    gt_extensions: tuple[str, ...] = (),
    graph_language: str | None = None,
    graph_aliases: tuple[str, ...] = (),
    graph_extensions: tuple[str, ...] = (),
    agent_languages: tuple[str, ...] = (),
    agent_aliases: tuple[tuple[str, str], ...] = (),
    cold_start_backend: str | None = None,
    graph_indexer: str | None = None,
    graph_decoder: str | None = None,
    scip_cold_start: ScipColdStartOption | None = None,
    scip_candidate_indexer: str | None = None,
    incremental_backend: str | None = None,
    incremental_patcher: str | None = None,
    lsp_language_id: str | None = None,
    lsp_command: tuple[str, ...] = (),
    lsp_command_env: str | None = None,
    lsp_command_factory: str | None = None,
    core_decoder: bool = False,
    core_decoder_aliases: tuple[str, ...] = (),
)

Declarative metadata for one CodeNib language family.

LanguageCapability dataclass

LanguageCapability(
    key: str,
    display_name: str,
    chunker: bool,
    ground_truth: bool,
    agent: bool,
    graph_backend: str | None,
    scip_cold_start: str,
    incremental_backend: str | None,
    lsp: bool,
    core_decoder: bool,
    core_parity: str,
)

Derived support matrix row for one registered language.

get_language_spec

get_language_spec(language: str) -> LanguageSpec | None

Return the spec for a language key or general alias.

Source code in codenib/languages.py

def get_language_spec(language: str) -> Optional[LanguageSpec]:
    """Return the spec for a language key or general alias."""

    key = _GENERAL_ALIASES.get(_norm(language))
    if key is None:
        return None
    return SPECS_BY_KEY[key]

normalize_language

normalize_language(language: str) -> str | None

Normalize a raw language string to a registry key.

Source code in codenib/languages.py

def normalize_language(language: str) -> Optional[str]:
    """Normalize a raw language string to a registry key."""

    spec = get_language_spec(language)
    return spec.key if spec else None

normalize_chunker_language

normalize_chunker_language(language: str) -> str | None

Normalize a raw language string for create_chunker.

Source code in codenib/languages.py

def normalize_chunker_language(language: str) -> Optional[str]:
    """Normalize a raw language string for ``create_chunker``."""

    return _CHUNKER_ALIASES.get(_norm(language))

get_chunker_spec

get_chunker_spec(language: str) -> LanguageSpec | None

Return the LanguageSpec used by the chunker factory.

Source code in codenib/languages.py

def get_chunker_spec(language: str) -> Optional[LanguageSpec]:
    """Return the LanguageSpec used by the chunker factory."""

    chunker_language = normalize_chunker_language(language)
    if chunker_language is None:
        return None
    for spec in LANGUAGE_SPECS:
        if spec.chunker_language == chunker_language:
            return spec
    return None

normalize_graph_language

normalize_graph_language(language: str) -> str | None

Normalize a raw language string to the graph/LS backend key.

Source code in codenib/languages.py

def normalize_graph_language(language: str) -> Optional[str]:
    """Normalize a raw language string to the graph/LS backend key."""

    key = graph_language_aliases().get(_norm(language))
    return key

normalize_agent_language

normalize_agent_language(language: str) -> str | None

Normalize a raw language string for agent compile scenarios.

Source code in codenib/languages.py

def normalize_agent_language(language: str) -> Optional[str]:
    """Normalize a raw language string for agent compile scenarios."""

    return _AGENT_ALIASES.get(_norm(language))

supported_agent_languages

supported_agent_languages() -> FrozenSet[str]

Return language scenario keys accepted by agent compile.

Source code in codenib/languages.py

def supported_agent_languages() -> FrozenSet[str]:
    """Return language scenario keys accepted by agent compile."""

    return frozenset(lang for spec in LANGUAGE_SPECS for lang in spec.agent_languages)

chunker_language_aliases

chunker_language_aliases() -> dict[str, str]

Return raw chunker aliases mapped to chunker language keys.

Source code in codenib/languages.py

def chunker_language_aliases() -> Dict[str, str]:
    """Return raw chunker aliases mapped to chunker language keys."""

    return dict(_CHUNKER_ALIASES)

chunker_languages

chunker_languages() -> tuple[str, ...]

Return canonical language keys accepted by repository chunking.

Source code in codenib/languages.py

def chunker_languages() -> Tuple[str, ...]:
    """Return canonical language keys accepted by repository chunking."""

    return _unique(
        spec.chunker_language for spec in LANGUAGE_SPECS if spec.chunker_language
    )

chunker_class_paths

chunker_class_paths(include_aliases: bool = True) -> dict[str, str]

Return chunker language keys mapped to chunker class paths.

Source code in codenib/languages.py

def chunker_class_paths(include_aliases: bool = True) -> Dict[str, str]:
    """Return chunker language keys mapped to chunker class paths."""

    by_language: Dict[str, str] = {}
    for spec in LANGUAGE_SPECS:
        if spec.chunker_language and spec.chunker_class:
            by_language[spec.chunker_language] = spec.chunker_class

    if not include_aliases:
        return dict(by_language)

    result = dict(by_language)
    for alias, language in chunker_language_aliases().items():
        if language in by_language:
            result[alias] = by_language[language]
    return result

chunker_class_path

chunker_class_path(language: str) -> str | None

Return the chunker class path for a raw chunker language.

Source code in codenib/languages.py

def chunker_class_path(language: str) -> Optional[str]:
    """Return the chunker class path for a raw chunker language."""

    spec = get_chunker_spec(language)
    if spec is None:
        return None
    return spec.chunker_class

graph_language_aliases

graph_language_aliases() -> dict[str, str]

Return raw LS/graph aliases mapped to backend language keys.

Source code in codenib/languages.py

def graph_language_aliases() -> Dict[str, str]:
    """Return raw LS/graph aliases mapped to backend language keys."""

    aliases: Dict[str, str] = {}
    for spec in LANGUAGE_SPECS:
        if not spec.graph_language:
            continue
        aliases[spec.graph_language] = spec.graph_language
        for alias in spec.graph_aliases:
            aliases[alias] = spec.graph_language
    return aliases

agent_language_aliases

agent_language_aliases() -> dict[str, str]

Return raw agent language aliases mapped to scenario keys.

Source code in codenib/languages.py

def agent_language_aliases() -> Dict[str, str]:
    """Return raw agent language aliases mapped to scenario keys."""

    return dict(_AGENT_ALIASES)

extension_to_language_map

extension_to_language_map(kind: ExtensionKind) -> dict[str, str]

Return an extension-to-language map for the requested surface.

Source code in codenib/languages.py

def extension_to_language_map(kind: ExtensionKind) -> Dict[str, str]:
    """Return an extension-to-language map for the requested surface."""

    result: Dict[str, str] = {}
    for spec in LANGUAGE_SPECS:
        language = getattr(spec, _language_attr(kind))
        extensions = getattr(spec, _extensions_attr(kind))
        if not language:
            continue
        for ext in extensions:
            result[ext] = language
    return result

extensions_for_language

extensions_for_language(language: str, kind: ExtensionKind) -> set[str]

Return extensions for a language on the requested surface.

Source code in codenib/languages.py

def extensions_for_language(language: str, kind: ExtensionKind) -> set[str]:
    """Return extensions for a language on the requested surface."""

    if kind == "graph":
        graph_key = normalize_graph_language(language)
        if graph_key is None:
            return set()
        return graph_extensions_by_language().get(graph_key, set())

    spec = get_language_spec(language)
    if spec is None:
        return set()
    return set(getattr(spec, _extensions_attr(kind)))

graph_extensions_by_language

graph_extensions_by_language(include_aliases: bool = True) -> dict[str, set[str]]

Return graph backend language keys mapped to accepted extensions.

Source code in codenib/languages.py

def graph_extensions_by_language(include_aliases: bool = True) -> Dict[str, set[str]]:
    """Return graph backend language keys mapped to accepted extensions."""

    by_backend: Dict[str, set[str]] = {}
    for spec in LANGUAGE_SPECS:
        if not spec.graph_language:
            continue
        by_backend.setdefault(spec.graph_language, set()).update(spec.graph_extensions)

    if not include_aliases:
        return {
            language: set(extensions) for language, extensions in by_backend.items()
        }

    result = {language: set(extensions) for language, extensions in by_backend.items()}
    aliases = graph_language_aliases()
    for alias, graph_language in aliases.items():
        if graph_language in by_backend:
            result[alias] = set(by_backend[graph_language])
    return result

graph_indexer_paths

graph_indexer_paths(include_aliases: bool = True) -> dict[str, str]

Return graph backend language keys mapped to cold-start indexer paths.

Source code in codenib/languages.py

def graph_indexer_paths(include_aliases: bool = True) -> Dict[str, str]:
    """Return graph backend language keys mapped to cold-start indexer paths."""

    return _graph_surface_values("graph_indexer", include_aliases=include_aliases)

graph_decoder_paths

graph_decoder_paths(include_aliases: bool = True) -> dict[str, str]

Return graph backend language keys mapped to graph decoder paths.

Source code in codenib/languages.py

def graph_decoder_paths(include_aliases: bool = True) -> Dict[str, str]:
    """Return graph backend language keys mapped to graph decoder paths."""

    return _graph_surface_values("graph_decoder", include_aliases=include_aliases)

graph_cold_start_backends

graph_cold_start_backends(include_aliases: bool = True) -> dict[str, str]

Return graph backend language keys mapped to cold-start backend names.

Source code in codenib/languages.py

def graph_cold_start_backends(include_aliases: bool = True) -> Dict[str, str]:
    """Return graph backend language keys mapped to cold-start backend names."""

    return _graph_surface_values("cold_start_backend", include_aliases=include_aliases)

graph_indexer_path

graph_indexer_path(language: str) -> str | None

Return the cold-start indexer class path for a raw graph language.

Source code in codenib/languages.py

def graph_indexer_path(language: str) -> Optional[str]:
    """Return the cold-start indexer class path for a raw graph language."""

    graph_language = normalize_graph_language(language)
    if graph_language is None:
        return None
    return graph_indexer_paths(include_aliases=False).get(graph_language)

graph_decoder_path

graph_decoder_path(language: str) -> str | None

Return the graph decoder class path for a raw graph language.

Source code in codenib/languages.py

def graph_decoder_path(language: str) -> Optional[str]:
    """Return the graph decoder class path for a raw graph language."""

    graph_language = normalize_graph_language(language)
    if graph_language is None:
        return None
    return graph_decoder_paths(include_aliases=False).get(graph_language)

graph_cold_start_backend

graph_cold_start_backend(language: str) -> str | None

Return the cold-start backend name for a raw graph language.

Source code in codenib/languages.py

def graph_cold_start_backend(language: str) -> Optional[str]:
    """Return the cold-start backend name for a raw graph language."""

    graph_language = normalize_graph_language(language)
    if graph_language is None:
        return None
    return graph_cold_start_backends(include_aliases=False).get(graph_language)

scip_cold_start_options

scip_cold_start_options(include_aliases: bool = True) -> dict[str, ScipColdStartOption]

Return languages mapped to configured SCIP cold-start options.

Candidate entries are intentionally returned even when a language's active graph backend is still LSP or tree-sitter-only. They are planning metadata, not an enabled router path.

Source code in codenib/languages.py

def scip_cold_start_options(
    include_aliases: bool = True,
) -> Dict[str, ScipColdStartOption]:
    """Return languages mapped to configured SCIP cold-start options.

    Candidate entries are intentionally returned even when a language's active
    graph backend is still LSP or tree-sitter-only. They are planning metadata,
    not an enabled router path.
    """

    result: Dict[str, ScipColdStartOption] = {}
    for spec in LANGUAGE_SPECS:
        option = spec.scip_cold_start
        if option is None:
            continue
        result[spec.key] = option
        if include_aliases:
            for alias in spec.aliases:
                result[alias] = option
            if spec.graph_language:
                result[spec.graph_language] = option
                for alias in spec.graph_aliases:
                    result[alias] = option
    return result

scip_cold_start_option

scip_cold_start_option(language: str) -> ScipColdStartOption | None

Return SCIP cold-start metadata for a language key or alias.

Source code in codenib/languages.py

def scip_cold_start_option(language: str) -> Optional[ScipColdStartOption]:
    """Return SCIP cold-start metadata for a language key or alias."""

    return scip_cold_start_options(include_aliases=True).get(_norm(language))

scip_candidate_indexer_paths

scip_candidate_indexer_paths(include_aliases: bool = True) -> dict[str, str]

Return languages mapped to explicit candidate SCIP indexer class paths.

These paths are opt-in only. They do not change a language's active graph backend and are used by smoke/profiling flows that intentionally evaluate a candidate SCIP cold-start route.

Source code in codenib/languages.py

def scip_candidate_indexer_paths(include_aliases: bool = True) -> Dict[str, str]:
    """Return languages mapped to explicit candidate SCIP indexer class paths.

    These paths are opt-in only. They do not change a language's active graph
    backend and are used by smoke/profiling flows that intentionally evaluate a
    candidate SCIP cold-start route.
    """

    result: Dict[str, str] = {}
    for spec in LANGUAGE_SPECS:
        path = spec.scip_candidate_indexer
        if not path:
            continue
        result[spec.key] = path
        if include_aliases:
            for alias in spec.aliases:
                result[alias] = path
            if spec.graph_language:
                result[spec.graph_language] = path
                for alias in spec.graph_aliases:
                    result[alias] = path
    return result

scip_candidate_indexer_path

scip_candidate_indexer_path(language: str) -> str | None

Return an opt-in candidate SCIP indexer path for a language or alias.

Source code in codenib/languages.py

def scip_candidate_indexer_path(language: str) -> Optional[str]:
    """Return an opt-in candidate SCIP indexer path for a language or alias."""

    return scip_candidate_indexer_paths(include_aliases=True).get(_norm(language))

scip_cold_start_command_for_language

scip_cold_start_command_for_language(language: str) -> list[str] | None

Return the configured SCIP cold-start command for a language or alias.

Source code in codenib/languages.py

def scip_cold_start_command_for_language(language: str) -> Optional[list[str]]:
    """Return the configured SCIP cold-start command for a language or alias."""

    option = scip_cold_start_option(language)
    if option is None:
        return None

    if option.command_env:
        override = os.environ.get(option.command_env)
        if override:
            return shlex.split(override)
    if option.command:
        return list(option.command)
    return [option.tool] if option.tool else None

lsp_language_id_for_language

lsp_language_id_for_language(language: str) -> str | None

Return the LSP language id for a raw graph language.

Source code in codenib/languages.py

def lsp_language_id_for_language(language: str) -> Optional[str]:
    """Return the LSP language id for a raw graph language."""

    graph_language = normalize_graph_language(language)
    if graph_language is None:
        return None
    return _graph_surface_values("lsp_language_id", include_aliases=False).get(
        graph_language
    )

lsp_command_for_language

lsp_command_for_language(language: str) -> list[str] | None

Return the configured LSP command for a raw graph language.

Source code in codenib/languages.py

def lsp_command_for_language(language: str) -> Optional[list[str]]:
    """Return the configured LSP command for a raw graph language."""

    graph_language = normalize_graph_language(language)
    if graph_language is None:
        return None

    spec = next(
        (spec for spec in LANGUAGE_SPECS if spec.graph_language == graph_language),
        None,
    )
    if spec is None:
        return None

    if spec.lsp_command_env:
        override = os.environ.get(spec.lsp_command_env)
        if override:
            return shlex.split(override)
    if spec.lsp_command_factory:
        return _call_no_arg_factory(spec.lsp_command_factory)
    if spec.lsp_command:
        return list(spec.lsp_command)
    return None

incremental_patcher_paths

incremental_patcher_paths(include_aliases: bool = True) -> dict[str, str]

Return graph backend language keys mapped to incremental patcher paths.

Source code in codenib/languages.py

def incremental_patcher_paths(include_aliases: bool = True) -> Dict[str, str]:
    """Return graph backend language keys mapped to incremental patcher paths."""

    return _graph_surface_values("incremental_patcher", include_aliases=include_aliases)

incremental_patcher_path

incremental_patcher_path(language: str) -> str | None

Return the incremental patcher class path for a raw graph language.

Source code in codenib/languages.py

def incremental_patcher_path(language: str) -> Optional[str]:
    """Return the incremental patcher class path for a raw graph language."""

    graph_language = normalize_graph_language(language)
    if graph_language is None:
        return None
    return incremental_patcher_paths(include_aliases=False).get(graph_language)

core_decoder_languages

core_decoder_languages(include_aliases: bool = True) -> tuple[str, ...]

Return languages accepted by the optional C++ core decoder.

Source code in codenib/languages.py

def core_decoder_languages(include_aliases: bool = True) -> Tuple[str, ...]:
    """Return languages accepted by the optional C++ core decoder."""

    languages: list[str] = []
    for spec in LANGUAGE_SPECS:
        if spec.core_decoder:
            languages.append(spec.key)
            if include_aliases:
                languages.extend(spec.core_decoder_aliases)
    return _unique(languages)

language_capability_rows

language_capability_rows() -> tuple[LanguageCapability, ...]

Return a registry-derived language support matrix.

The matrix intentionally distinguishes tree-sitter-only languages from graph-capable languages with no core decoder. This makes serial-only/core parity coverage explicit instead of relying on skipped tests as a signal.

Source code in codenib/languages.py

def language_capability_rows() -> Tuple[LanguageCapability, ...]:
    """Return a registry-derived language support matrix.

    The matrix intentionally distinguishes tree-sitter-only languages from
    graph-capable languages with no core decoder.  This makes serial-only/core
    parity coverage explicit instead of relying on skipped tests as a signal.
    """

    core_graph_languages = {
        spec.graph_language
        for spec in LANGUAGE_SPECS
        if spec.core_decoder and spec.graph_language
    }
    rows: list[LanguageCapability] = []
    for spec in LANGUAGE_SPECS:
        has_graph = bool(
            spec.graph_language and spec.graph_indexer and spec.graph_decoder
        )
        has_core = bool(
            spec.core_decoder
            or (spec.graph_language and spec.graph_language in core_graph_languages)
        )
        if has_core:
            core_parity = "covered"
        elif has_graph:
            core_parity = "n/a-no-core-decoder"
        else:
            core_parity = "n/a-tree-sitter-only"

        rows.append(
            LanguageCapability(
                key=spec.key,
                display_name=spec.display_name,
                chunker=bool(spec.chunker_language and spec.chunker_class),
                ground_truth=bool(spec.gt_language and spec.gt_extensions),
                agent=bool(spec.agent_languages),
                graph_backend=spec.cold_start_backend if has_graph else None,
                scip_cold_start=(
                    spec.scip_cold_start.status if spec.scip_cold_start else "none"
                ),
                incremental_backend=(
                    spec.incremental_backend if spec.incremental_patcher else None
                ),
                lsp=bool(
                    spec.lsp_language_id
                    and (
                        spec.lsp_command
                        or spec.lsp_command_env
                        or spec.lsp_command_factory
                    )
                ),
                core_decoder=has_core,
                core_parity=core_parity,
            )
        )
    return tuple(rows)