POMDPs.jl Interface

ContinuousDPs.jl ships a package extension connecting the collocation solver to the POMDPs.jl ecosystem. It is activated by loading both trigger packages:

using POMDPs, POMDPTools

The extension makes CollocationSolver available as a solver for POMDPs.jl models: POMDPs.solve(CollocationSolver(basis), m) solves an MDP with continuous states covered by the basis, finitely many actions or a scalar action interval, and explicit transition distributions by the Bellman equation collocation method, and returns a standard POMDPs.Policy. This gives POMDPs.jl models access to smooth global approximation of the value function and the policy over a continuous state space, complementing the ecosystem's sampling-based planners.

Solving a POMDPs.jl model

A stochastic optimal growth model written as a QuickPOMDPs model and solved by collocation. The model's primitives are the familiar ones — reward f, transition g, and action bounds x_lb, x_ub — and the feasible actions at a state are declared as an interval, so the consumption choice stays continuous:

using ContinuousDPs
using BasisMatrices: Basis, ChebParams
using QuantEcon: qnwlogn
using POMDPs, POMDPTools, QuickPOMDPs

function build_growth_mdp(; alpha = 0.4, beta = 0.95,
                          s_min = 0.1, s_max = 2.0)
    f(s, x) = log(x)                                # consumption x
    g(s, x, e) = clamp((s - x)^alpha * e, s_min, s_max)
    x_lb(s) = s_min
    x_ub(s) = s
    shocks, weights = qnwlogn(7, 0.0, 0.05^2)
    return QuickMDP(
        statetype = Float64,
        actiontype = Float64,
        actions = s -> ActionInterval(x_lb(s), x_ub(s)),  # continuous
        discount = beta,
        transition = (s, x) -> SparseCat(g.(s, x, shocks), weights),
        reward = f,
    )
end

m = build_growth_mdp()

basis = Basis(ChebParams(30, 0.1, 2.0))
policy = POMDPs.solve(CollocationSolver(basis), m)

action(policy, 1.0)   # optimal consumption at s = 1.0
value(policy, 1.0)    # fitted value function at s = 1.0

# The ecosystem's tooling works as usual, e.g. rollout simulation
POMDPs.simulate(RolloutSimulator(max_steps=100), m, policy, 1.0)

A finite action set is declared instead by passing the collection as actions (with actions(m, s) optionally restricting it per state); the inner maximization is then an exact enumeration. A fuller example, the household problem of the Aiyagari model with continuous assets and savings, is ext/examples/cdpexaiyagari_pomdps.jl.

Requirements and scope

POMDPs.solve checks its requirements at solve time where feasible and throws an informative error when a check fails:

• Explicit actions: a finite set or a scalar interval. Either actions(m) returns a finite collection — a state-dependent restriction via actions(m, s) is supported and mapped to infeasibility; every collocation node needs at least one feasible action, and actions(m, s) must be a subset of actions(m) (actions outside the global set are not seen by the solver); the model's transition and reward are never evaluated on infeasible state–action pairs — or actions(m, s) returns an ActionInterval(lo, hi), declaring a continuous scalar action space with state-dependent bounds (the model's action type must accept Float64); the inner maximization then uses the derivative-free (Brent) solver. Multi-dimensional continuous actions are not supported through this interface.
• Explicit transition distributions. transition(m, s, x) must return a distribution with finite support and explicit probabilities (SparseCat, Deterministic, ... — anything supporting POMDPTools.weighted_iterator). Zero-probability entries are skipped without evaluating the model there (a SparseCat may list impossible successors). Generative-only models are out of scope.
• No terminal states (not supported in this version): isterminal(m, s) must be false on the entire basis domain. The collocation nodes are checked at solve time; keeping off-node states non-terminal is the model's responsibility.
• Rewards may be defined as reward(m, s, x) or reward(m, s, x, sp). The arity is chosen once at solve time by a probe call, preferring the direct form; with only the next-state-dependent form, the expectation over the transition's branches is taken, at the cost of one reward call per branch per evaluation.
• The state space is continuous, with the domain and dimension given by the solver's basis. States are passed to the model as statetype(m) when a conversion from the solver's coordinates applies: coordinates that already match the state type pass through (always the case for a Float64 state on a one-dimensional basis), Tuple state types such as NTuple{2,Float64} get the natural elementwise conversion, and any other state type is routed through POMDPs.convert_s when the model provides a method for the coordinate form (an AbstractVector of Float64). Otherwise states are passed as indexable coordinate points. The model's next states must be indexable (scalars, tuples, or static vectors).
• Next states must stay within the basis domain for every feasible action, as everywhere in ContinuousDPs: the fitted value function is extrapolated outside the domain, which can silently destroy a solve (astronomically so for Chebyshev bases). Clamp inside the transition, or choose the domain to be invariant.

The returned policy

The returned CollocationPolicy is a standard POMDPs.Policy:

• action(policy, s) evaluates the computed policy: for a continuous action space by interpolating the solved policy and clamping it into the feasible interval at s; for a finite action set by exact greedy recomputation at s (the solution's coefficients are interpolated; a discrete policy itself is never interpolated);
• value(policy, s) evaluates the fitted value function;
• the full CDPSolveResult remains available as policy.res (residuals, set_eval_nodes!, the native simulate); after set_eval_nodes!(policy.res, ...), action evaluates the policy on the new evaluation nodes.

CollocationPolicy is not thread-safe: for parallel rollouts, use one policy instance per thread (the underlying evaluation caches are single-threaded).

Example: a belief MDP

Finite-observation learning problems reduce to belief MDPs whose transition distributions have finite support, as the extension requires. The job search model with learning about the offer distribution (examples/cdpexodu.jl, following the QuantEcon lecture) solves natively as a ContinuousDP with state-dependent callable weights; the same belief MDP written as a POMDPs.jl model solves through the extension:

function build_belief_mdp(sp)   # sp = SearchProblem() from the example
    return QuickMDP(
        statetype = NTuple{2,Float64},
        actions = [:reject, :accept],
        discount = sp.beta,
        reward = (s, x) -> x === :accept ? s[1] : sp.c,
        transition = function (s, x)
            x === :accept && return Deterministic((s[1], s[2]))
            SparseCat([(wp, sp.update(wp, s[2])) for wp in sp.shocks],
                      sp.weights(s))
        end,
    )
end

bm = build_belief_mdp(sp)
basis = Basis(
    SplineParams(collect(range(0.0, sp.w_max, length=40)), 0, 3),
    SplineParams(collect(range(sp.pi_min, sp.pi_max, length=40)), 0, 3))
policy = POMDPs.solve(CollocationSolver(basis), bm)

POMDPs.action(policy, (1.5, 0.5))
hist = POMDPs.simulate(HistoryRecorder(max_steps=30), bm, policy, (1.0, 0.5))

The two formulations are the same transition kernel in two vocabularies: the SparseCat pairing each offer with its Bayes-updated belief and the belief-mixed probabilities sp.weights(s) is precisely the native model's g over shocks weighted by the callable weights. The native and POMDPs solves produce identical coefficients.

Two solve generic functions

QuantEcon.jl and POMDPs.jl both export solve and simulate, so under using QuantEcon, POMDPs (with or without ContinuousDPs) the bare names are ambiguous and qualified calls (POMDPs.solve, POMDPs.simulate) are needed, as in the examples above. This name clash between the two ecosystems predates this extension.

Note that CollocationSolver does not subtype POMDPs.Solver: POMDPs.solve is an empty generic function, so the methods above work by ordinary dispatch. Subtyping would require an unconditional POMDPs dependency in the core package; if you need a POMDPs.Solver-typed object (e.g. for a solver-annotated container), wrap the solver in a small struct of your own.