codenib.code_chunking
¶
Code chunking module for splitting source code files into semantic chunks.
Modules:
| Name | Description |
|---|---|
base |
Base code chunker class with common functionality. |
cpp_chunker |
C++ specific code chunker implementation. |
csharp_chunker |
C#-specific code chunker implementation. |
go_chunker |
Go-specific code chunker implementation. |
java_chunker |
Java-specific code chunker implementation. |
js_chunker |
JavaScript/TypeScript code chunker implementation. |
kotlin_chunker |
Kotlin-specific code chunker implementation. |
lua_chunker |
Lua-specific code chunker implementation. |
php_chunker |
PHP-specific code chunker implementation. |
python_chunker |
Python-specific code chunker implementation. |
ruby_chunker |
Ruby-specific code chunker implementation. |
rust_chunker |
Rust-specific code chunker implementation. |
scala_chunker |
Scala-specific code chunker implementation. |
swift_chunker |
Swift-specific code chunker implementation. |
Classes:
| Name | Description |
|---|---|
BaseCodeChunker |
Base class for language-specific code chunkers. |
CppCodeChunker |
Code chunker specifically for C++ files. |
CSharpCodeChunker |
Code chunker for C# source files. |
GoCodeChunker |
Code chunker for Go source files. |
JavaCodeChunker |
Code chunker for Java source files. |
JsTsCodeChunker |
Chunker for JavaScript and TypeScript source files. |
KotlinCodeChunker |
Code chunker for Kotlin source files. |
LuaCodeChunker |
Code chunker for Lua source files. |
PhpCodeChunker |
Code chunker for PHP source files. |
PythonCodeChunker |
Code chunker specifically for Python files. |
RubyCodeChunker |
Code chunker for Ruby source files. |
RustCodeChunker |
Code chunker for Rust source files. |
ScalaCodeChunker |
Code chunker for Scala source files. |
SwiftCodeChunker |
Code chunker for Swift source files. |
Functions:
| Name | Description |
|---|---|
create_chunker |
Create a code chunker for the specified language. |
BaseCodeChunker
¶
BaseCodeChunker(
language: str,
max_lines_per_chunk: int | None = None,
chunk_depth: int = 2,
include_header_epilogue: bool = False,
l2_level_exclusive: bool = True,
skeleton_mode: bool = False,
include_l2_in_file_skeleton: bool = True,
)
Bases: ABC
Base class for language-specific code chunkers.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
language
|
str
|
Programming language to parse ('python', 'cpp', 'java', etc.) |
required |
max_lines_per_chunk
|
int | None
|
Maximum number of lines per emitted chunk. When set, large logical chunks (function/class) will be split into multiple sequential chunks of at most this many lines. node_id and name remain the same across the split pieces. Default: None (no splitting). Set to a number to enable. |
None
|
chunk_depth
|
int
|
Depth of AST traversal for chunking: 0 = Treat entire file as a single chunk 1 = Top-level only (classes and top-level functions, no methods) 2 = Method-level (classes, functions, and methods) [default] |
2
|
include_header_epilogue
|
bool
|
Whether to include file header (imports, module docstrings) and epilogue (trailing code) in chunks. Default: False (skip them to reduce noise). |
False
|
l2_level_exclusive
|
bool
|
When chunk_depth is 2 (method/member level), whether to exclude the containing type/scope chunks (classes, structs, impls). Default: True to keep L2-only output; set to False to emit both L1 container chunks and L2 members. |
True
|
skeleton_mode
|
bool
|
When True, emit skeletonized content (signatures only) for file- and type-level chunks. File-level skeletons list top-level definitions and, when include_l2_in_file_skeleton is True, member signatures for a hierarchical view. Type-level skeletons list member signatures. Defaults to False (emit full source). |
False
|
include_l2_in_file_skeleton
|
bool
|
When chunk_depth is 0 and skeleton_mode is enabled, whether to include L2/member signatures in the file skeleton output. Defaults to True for richer, hierarchical skeletons. |
True
|
Methods:
| Name | Description |
|---|---|
chunk_file |
Chunk a code file into function/class level pieces. |
save_chunks_to_json |
Save chunks to a JSON file. |
print_chunk_summary |
Print a summary of the generated chunks. |
Source code in codenib/code_chunking/base.py
chunk_file
¶
chunk_file(
file_path: str, relative_path: str | None = None, skeleton_mode: bool | None = None
) -> list[CodeChunk]
Chunk a code file into function/class level pieces.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
file_path
|
str
|
Absolute path to the code file to chunk |
required |
relative_path
|
str | None
|
Relative path for node_id generation |
None
|
skeleton_mode
|
bool | None
|
Override instance-level skeleton setting. When True, chunks contain signature-only skeletons instead of full bodies. |
None
|
Returns:
| Type | Description |
|---|---|
list[CodeChunk]
|
List of CodeChunk objects representing the chunks |
Source code in codenib/code_chunking/base.py
119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 | |
save_chunks_to_json
¶
Save chunks to a JSON file.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
chunks
|
list[CodeChunk]
|
List of CodeChunk objects |
required |
output_path
|
str
|
Path to save the JSON file |
required |
Source code in codenib/code_chunking/base.py
print_chunk_summary
¶
Print a summary of the generated chunks.
Source code in codenib/code_chunking/base.py
CppCodeChunker
¶
CppCodeChunker(
max_lines_per_chunk: int | None = None,
chunk_depth: int = 2,
l2_level_exclusive: bool = True,
**kwargs
)
Bases: BaseCodeChunker
Code chunker specifically for C++ files.
L1 entities: free functions and class/struct definitions. L2 entities: methods declared/defined inside class bodies. Skeleton mode: file skeleton lists L1 declarations and member signatures; class skeleton lists method signatures.
Source code in codenib/code_chunking/cpp_chunker.py
CSharpCodeChunker
¶
CSharpCodeChunker(
max_lines_per_chunk: int | None = None,
chunk_depth: int = 2,
l2_level_exclusive: bool = True,
**kwargs
)
Bases: BaseCodeChunker
Code chunker for C# source files.
L1 entities: classes, interfaces, enums, records, structs, and top-level local functions. L2 entities: methods, constructors, and properties inside type declarations. Skeleton mode: file skeleton lists type declarations and member signatures.
Source code in codenib/code_chunking/csharp_chunker.py
GoCodeChunker
¶
GoCodeChunker(
max_lines_per_chunk: int | None = None,
chunk_depth: int = 2,
l2_level_exclusive: bool = True,
**kwargs
)
Bases: BaseCodeChunker
Code chunker for Go source files.
L1 entities: top-level functions and type declarations (struct/interface/type). L2 entities: methods with receivers. Skeleton mode: file skeleton lists L1 declarations and method signatures; type skeleton lists receiver-bound methods.
Source code in codenib/code_chunking/go_chunker.py
JavaCodeChunker
¶
JavaCodeChunker(
max_lines_per_chunk: int | None = None,
chunk_depth: int = 2,
l2_level_exclusive: bool = True,
**kwargs
)
Bases: BaseCodeChunker
Code chunker for Java source files.
L1 entities: top-level classes, interfaces, enums, and records. L2 entities: methods and constructors inside those containers. Skeleton mode: file skeleton lists L1 declarations and member signatures.
Source code in codenib/code_chunking/java_chunker.py
JsTsCodeChunker
¶
JsTsCodeChunker(
language: str = "javascript",
max_lines_per_chunk: int | None = None,
chunk_depth: int = 2,
l2_level_exclusive: bool = True,
**kwargs
)
Bases: BaseCodeChunker
Chunker for JavaScript and TypeScript source files.
L1 entities: top-level functions and classes. L2 entities: methods within classes. Skeleton mode: file skeleton lists L1 declarations and member signatures; class skeleton lists method signatures.
Source code in codenib/code_chunking/js_chunker.py
KotlinCodeChunker
¶
KotlinCodeChunker(
max_lines_per_chunk: int | None = None,
chunk_depth: int = 2,
l2_level_exclusive: bool = True,
**kwargs
)
Bases: BaseCodeChunker
Code chunker for Kotlin source files.
L1 entities: classes, interfaces, enums, objects, top-level functions, and top-level properties. L2 entities: functions, secondary constructors, and properties inside type/object declarations. Skeleton mode: file skeleton lists container declarations and member signatures.
Source code in codenib/code_chunking/kotlin_chunker.py
LuaCodeChunker
¶
LuaCodeChunker(
max_lines_per_chunk: int | None = None,
chunk_depth: int = 2,
l2_level_exclusive: bool = True,
**kwargs
)
Bases: BaseCodeChunker
Code chunker for Lua source files.
Source code in codenib/code_chunking/lua_chunker.py
PhpCodeChunker
¶
PhpCodeChunker(
max_lines_per_chunk: int | None = None,
chunk_depth: int = 2,
l2_level_exclusive: bool = True,
**kwargs
)
Bases: BaseCodeChunker
Code chunker for PHP source files.
L1 entities: classes, interfaces, traits, enums, and top-level functions. L2 entities: methods and properties inside type declarations. Skeleton mode: file skeleton lists type declarations and member signatures.
Source code in codenib/code_chunking/php_chunker.py
PythonCodeChunker
¶
PythonCodeChunker(
max_lines_per_chunk: int | None = None,
chunk_depth: int = 2,
l2_level_exclusive: bool = True,
**kwargs
)
Bases: BaseCodeChunker
Code chunker specifically for Python files.
L1 entities: module-level functions and classes. L2 entities: methods inside classes (including async and decorated definitions). Skeleton mode: module skeleton lists L1 definitions and class member signatures; class skeleton lists method signatures.
Source code in codenib/code_chunking/python_chunker.py
RubyCodeChunker
¶
RubyCodeChunker(
max_lines_per_chunk: int | None = None,
chunk_depth: int = 2,
l2_level_exclusive: bool = True,
**kwargs
)
Bases: BaseCodeChunker
Code chunker for Ruby source files.
L1 entities: top-level modules, classes, and methods. L2 entities: instance and singleton methods inside modules/classes. Skeleton mode: file skeleton lists namespaces and member signatures.
Source code in codenib/code_chunking/ruby_chunker.py
RustCodeChunker
¶
RustCodeChunker(
max_lines_per_chunk: int | None = 200,
chunk_depth: int = 2,
l2_level_exclusive: bool = True,
**kwargs
)
Bases: BaseCodeChunker
Code chunker for Rust source files.
L1 entities: free functions plus struct/enum/trait/impl blocks. L2 entities: functions inside impl blocks. Skeleton mode: file skeleton lists L1 declarations and impl member signatures; impl skeleton lists method signatures.
Source code in codenib/code_chunking/rust_chunker.py
ScalaCodeChunker
¶
ScalaCodeChunker(
max_lines_per_chunk: int | None = None,
chunk_depth: int = 2,
l2_level_exclusive: bool = True,
**kwargs
)
Bases: BaseCodeChunker
Code chunker for Scala source files.
Source code in codenib/code_chunking/scala_chunker.py
SwiftCodeChunker
¶
SwiftCodeChunker(
max_lines_per_chunk: int | None = None,
chunk_depth: int = 2,
l2_level_exclusive: bool = True,
**kwargs
)
Bases: BaseCodeChunker
Code chunker for Swift source files.
Source code in codenib/code_chunking/swift_chunker.py
create_chunker
¶
create_chunker(
language: str,
max_lines_per_chunk: int | None = None,
chunk_depth: int = 2,
include_header_epilogue: bool = False,
l2_level_exclusive: bool = True,
skeleton_mode: bool = False,
include_l2_in_file_skeleton: bool = True,
) -> BaseCodeChunker
Create a code chunker for the specified language.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
language
|
str
|
Programming language ('python', 'cpp', 'java', etc.) |
required |
max_lines_per_chunk
|
int | None
|
Maximum number of lines per emitted chunk. Default: None (no splitting) |
None
|
chunk_depth
|
int
|
Granularity level 0 = Entire file as a chunk 1 = Top-level declarations only 2 = Include methods/impl members |
2
|
include_header_epilogue
|
bool
|
Whether to include file headers and epilogues. Default: False |
False
|
l2_level_exclusive
|
bool
|
When chunk_depth is 2, whether to omit L1 container nodes (classes/structs/impls) and emit only L2 members. Default: True. |
True
|
skeleton_mode
|
bool
|
Emit signature-only skeletons instead of full bodies when True. |
False
|
include_l2_in_file_skeleton
|
bool
|
When chunk_depth is 0, include member signatures in file-level skeletons for a hierarchical view. Default: True. |
True
|
Returns:
| Type | Description |
|---|---|
BaseCodeChunker
|
Language-specific code chunker instance |
Raises:
| Type | Description |
|---|---|
ValueError
|
If the language is not supported |