As we saw in my last blog post, there is a shape for stories.
The next ask we necessitate to ask is, if the stories have a shape, can we include that adviseation to originate RAG better. Turns out we can.
The shape of the stories helps us in chunking, which is a presentant problem area in RAG.
Chunking is one problem with RAG. Too big a chunk, you neglect particularity. Too petite a chunk, you neglect context. What’s the right size. This is the problem we try to settle by empathetic the “shape of stories”. Use the alter in tardynt space to determine context alter.
Semantic chunking has been tried before and is useable in summarizelabors enjoy LlamaIndex. But I wanted to get a better sense for the chunking.
Lets have a see at some stories:
Here we can evidently see we can have two chunks.
This one is alittle difficulter, but if we zoom in, we can hold some threshgreaters and discover a jump. Whenever there is a jump in the tardynt space, we can chunk.
The presentant part is acunderstandledgeing “what”, the jump is.
It can be as modest as the distance meacertain. Or we can see at the slope etc.
We have experimented with this and are releasing an API that you can spendigate to see if this chunking strategy labors for you.
A comparison of the semantic chunking in Llamaindex to our chunking is given below
The chunking was done on Paul Grahams essay.
All the chunks for the essay can be establish in this gist.
As you can see, our chunking has much more immacutardyer topics whereas the default semantic chunking bunches a bunch of topics into one chunk. Read the essay and chunk it manuassociate and contrast 🙂
