Retaining Code snippet in markdown content using UnstructuredMarkdownLoader

[kavinkumarrajendran2000]

Checked other resources

• I added a very descriptive title to this question.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.

Commit to Help

• I commit to help with one of those options 👆

Example Code

loader = DirectoryLoader(path="testing", glob="*.md", loader_cls=UnstructuredMarkdownLoader , recursive=True)
documents = loader.load()

Description

Hi,

I am using UnstructuredMarkdownLoader to load the Markdown content, but the ``` used for code snippet in Markdown content are removed after loading, so it's considered as a text instead code snippet. Also, the content in code snippet is removed sometimes.

**Example: MD content before loading: **

Area

@using Charts
    public class ChartData
    {
        public string X { get; set; }
        public double Y { get; set; }
    }
    public List<ChartData> MedalDetails = new List<ChartData>
    {
        new ChartData { X= "South Korea", Y= 39.4 },
        new ChartData { X= "India", Y= 61.3 },
        new ChartData { X= "Pakistan", Y= 20.4 },
        new ChartData { X= "Germany", Y= 65.1 },
        new ChartData { X= "Australia", Y= 15.8 }
    };
}

After loaded using UnstructuredMarkdownLoader :(``` is removed and often code snippet content is removed)

Area

@using Charts
public class ChartData
{
public string X { get; set; }
public double Y { get; set; }
}
}

How to retain the actual code snippet content and ``` in the code snippet.
Thanks.

System Info

Python 3.9.7
langchain latest version

[dosubot[bot]]

Hey there @kavinkumarrajendran2000! 🎉 I'm Dosu, a friendly bot here to assist you with bugs, answer your questions, and guide you on your journey to becoming a contributor. While we wait for a human maintainer to join us, I'm here to help you out. Let's tackle this issue together!

To modify the UnstructuredMarkdownLoader in LangChain to ensure that backticks and the content within code snippets in Markdown files are retained after loading, you will need to override or modify the method responsible for parsing the Markdown content. This involves ensuring that the parsing process does not strip out or ignore the backticks and the content within them.

Given the context provided, there isn't a direct snippet of the UnstructuredMarkdownLoader implementation to modify. However, I can guide you through a general approach on how to achieve this with a hypothetical example, assuming the UnstructuredMarkdownLoader uses a common Markdown parsing library like markdown or mistune.

First, ensure you have the necessary library installed:

pip install markdown  # or mistune, depending on what LangChain uses

Then, you can extend the UnstructuredMarkdownLoader class to override its Markdown loading method. Here's a conceptual example using the markdown library, which retains code snippets:

from langchain_community.document_loaders.markdown import UnstructuredMarkdownLoader
import markdown

class CustomMarkdownLoader(UnstructuredMarkdownLoader):
    def load_content(self, content):
        """
        Override the load_content method to parse Markdown content
        in a way that retains code snippets.
        """
        # Assuming the original loader converts Markdown to plain text
        # Modify this to retain code snippets
        md = markdown.Markdown(extensions=['fenced_code'])
        html_content = md.convert(content)
        # Further processing to extract or retain code snippets from HTML
        # This part depends on how you want to handle the HTML content
        # For simplicity, this example does not implement HTML parsing
        return html_content

# Use your custom loader
loader = CustomMarkdownLoader()
content = loader.load_content(your_markdown_content)

This example assumes that the original UnstructuredMarkdownLoader has a method named load_content that you are overriding. The key here is to use the Markdown library in a way that retains the code snippets, which is typically done by enabling the appropriate extension (e.g., fenced_code for the markdown library).

Please note, this example might need adjustments based on the actual implementation details of UnstructuredMarkdownLoader and how it processes Markdown content. If UnstructuredMarkdownLoader uses a different method for loading or parsing the content, or if it uses a different Markdown library, you'll need to adapt the code accordingly.

Unfortunately, I wasn't able to find the exact implementation of the UnstructuredMarkdownLoader class in the LangChain codebase. If you could provide more details about its implementation or the specific methods it uses to load and parse Markdown content, I could provide a more accurate solution.

Sources

• libs/langchain/langchain/document_loaders/markdown.py

About Dosu

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[kavinkumarrajendran2000]

Hi,

I am using below code for "UnstructuredMarkdownLoader" and loading this class in my main file. But it is excluding HTML tags<> and , so the code snippets and are removed for some MD files alone based on code snippet format.

Also, I tried above code, but it does not work still same issue.

from typing import List
from langchain_community.document_loaders.unstructured import UnstructuredFileLoader
import markdown
class UnstructuredMarkdownLoader(UnstructuredFileLoader):
def _get_elements(self) -> List:
from unstructured.version import version as unstructured_version
from unstructured.partition.md import partition_md

    # NOTE(MthwRobinson) - enables the loader to work when you're using pre-release
    # versions of unstructured like 0.4.17-dev1
    _unstructured_version = __unstructured_version__.split("-")[0]
    unstructured_version = tuple([int(x) for x in _unstructured_version.split(".")])

    if unstructured_version < (0, 4, 16):
        raise ValueError(
            f"You are on unstructured version {__unstructured_version__}. "
            "Partitioning markdown files is only supported in unstructured>=0.4.16."
        )

    return partition_md(filename=self.file_path, **self.unstructured_kwargs)
    
    
Can you suggest any other way to retain the code block inside ``` as it is.

Thanks.

[paulinetheitgirl]

For those who encounter this issue:
While the bot's suggested answer of a subclass may work if you call the load_content() method, it doesn't override all the other methods of UnstructuredMarkdownLoader, as seen in the documentation about Custom Document Loaders. So there is a risk that if your teammate uses the subclass, but calls load(), lazy_load(), and so on, they will not get the same behavior that load_content() does.
This is because in the source of UnstructuredMarkdownLoader,

from unstructured.partition.md import partition_md

        return partition_md(filename=self.file_path, **self.unstructured_kwargs)

it is using the external Unstructured.io library, whose source has this line:

html = markdown.markdown(text, extensions=["tables", "fenced_code"])

Those extensions are out-of-the-box extensions of the markdown library.
Unless the maintainers of the UnstructuredMarkdownLoader find a way to make the extensions customizable, I don't see a way to actually make usages of UnstructuredMarkdownLoader, or any subclasses of it, behave in a customized way for specific Markdown use-cases.

(This comment was fully written by me, a human who did some research 🙂 )