Skip to content

Introduce WritableFS #1153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Introduce WritableFS #1153

wants to merge 2 commits into from

Conversation

olekukonko
Copy link

The current Extraction functions make an assumption that writers will always write to a local directory. This limits flexibility and prevents integration with various storage backends like:

  • Block storage systems
  • HTTP APIs
  • In-memory storage
  • Temporary file systems
  • etc

Proposed Solution:
Introduce a WritableFS interface that abstracts file system operations, allowing for multiple storage implementations. This provides:

// WritableFS defines an interface for a writable file system or directory-like storage.
type WritableFS interface {
    // Base returns the base path or identifier of the file system (e.g., directory path or "memory").
    Base() string

    // WriteFile writes data to the named file, creating it if necessary.
    // If the file exists, it may overwrite it (implementation-dependent).
    // The perm parameter specifies the file permissions (e.g., 0644).
    WriteFile(name string, data []byte, perm fs.FileMode) error

    // Mkdir creates a directory named path, along with any necessary parents.
    // The perm parameter specifies the directory permissions (e.g., 0755).
    Mkdir(path string, perm fs.FileMode) error
}
  1. Better separation of concerns
  2. Flexibility to use different storage backends
  3. Cleaner API surface
  4. Improved testability with in-memory implementations

Key Changes:

  1. Added WritableFS
  2. Implemented Two concrete types:
    • OsFS for local file system operations
    • MemFS for in-memory storage (useful for testing)

Example Usage:

// Using local filesystem
osFs := NewOsFS("/output/dir")

// Using in-memory storage (for testing)
memFs := NewMemFS()
// Normal way:
ExtractContent(inFile, "/output/dir", pages, conf)

// Using WritableFS:
ExtractContentFS(inFile, NewOsFS("/output/dir"), pages, conf)

Benefits:

  • Backward compatible (can still use local file system)
  • Enables new storage integrations without changing core logic
  • Makes code more testable
  • Follows Go's interface-oriented design principles
  • Opens possibilities for future storage implementations (S3, GCS, etc.)

This is a production code hence Baby steps are required, hence i wanted to start with ExtractContentFS

Introduce writeable
add basic test
@CLAassistant
Copy link

CLAassistant commented May 10, 2025

CLA assistant check
All committers have signed the CLA.

@hhrutter
Copy link
Collaborator

Please create a feature request issue.
PRs are the result of issue discussions.
Thank you!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants