Abstractions on top of LLMs using DSPy.ai

Laith Zumot • 2025

Abstractions on top of LLMs

  • Prompt/Context Engineering is not scalable.
    • You cannot just 0 shot models for problems that matter, you need to iterate.
    • Models are trained differently, they need to be prompted differently
    • Pipelines and usecases change.

Abstractions on top of LLMs

  • A prompt programing library (Stanford)

  • Optimize prompts using python.

  • Forces you to THINK about your problem output and input structure first.

  • An Abstraction over language that can be “managed”

  • Separates concerns and automates the lower-level ones until you need to consider them.

    • Shorter code
    • Portability and modularity
    • Optimize later rather than sooner

  • Programming. Defining your task, its constraints, exploring a few examples, and using that to inform your initial pipeline design.

  • Evaluation. Collect an initial development set, define your DSPy metrics, iterate on your system more systematically.

  • Optimization. Tune the prompts or weights in your program.

Q&A

Q&A

CoT

CoT

Thread safe primitives, param args, hist

  • change the default LM globally with dspy.configure 
  • locally inside a block of code with dspy.context

step inside

Signatures

  • Declarative specification of input/output behavior
  • Declare and initialize the behavior of modules, DSPy will optimize the prompt for you
    • Question : str -> Answer : str
    • sql_query, python_code
    • sentence -> sentiment: bool
  • QA: document -> summary
  • RAG: context: list[str], question: str -> answer: str
  • MQA+R: question, choices: list[str] -> reasoning: str, selection: int

Signatures

Signatures

Signatures

Signatures

  • dspy.ChainOfThought also adds a reasoning field

Class based Signatures

  • Create a function with a clear, verbose signature, including type annotations and a docstring that explains the task.
  • Attach a descriptive hint (desc=) to the dspy.InputField to guide users on what input is expected.
  • Attach A descriptive constraint (desc=) to the dspy.OutputField to specify any restrictions or requirements on the output.

Class based Signatures

  • Classify.inspect_history

Signature

  • faithfulness to citations (to be fixed)

Signature

  • faithfulness to citations (to be fixed)

Multimodal Signatures

Multimodal Signatures

Class Multimodal Signatures

Class Multimodal Signatures

Signatures - Custom Types

Signatures - support for Python’s type system

  • Entity extraction
  • code

Signatures - support for Python’s type system

  • Entity extraction

OpenAI API

  • Entity extraction with openai api

DSPy Modules

  • Types: change the internal behavior with which your signature is implemented
    • dspy.Predict Does not modify the signature.
    • dspy.ChainOfThought - Think step-by-step
    • dspy.ProgramOfThought
    • dspy.ReAct - (tools)
    • dspy.MultiChainComparison
    • dspy.majority - Majority voting

DSPy Modules

# COT
math = dspy.ChainOfThought("question -> answer: float")

# Entity extraction: 
dspy.Predict("text -> title, headings: list[str], entities_and_metadata: list[dict[str, str]]")

# ReACT
dspy.ReAct("question -> answer: float", tools=[tool1, tool2])
  #Code execution:
  Dspy.PythonInterpreter({}).execute(expression)
  # RAG:
  dspy.ColBERTv2(url=….')(query, k=3)

DSPy Modules

  • code, whose execution results will dictate the response.

DSPy Modules

DSPy Modules

  • RAG

DSPy Modules

  • Tool Use

DSPy Modules

  • Pythonic

Thank You for Reading!