QuantEconPOMDPsExt

API documentation for the POMDPs.jl integration, a package extension activated by loading both trigger packages alongside QuantEcon:

using QuantEcon
using POMDPs, POMDPTools

Note that solve must then be qualified (QuantEcon.solve or POMDPs.solve), as both packages export it.

QuantEconPOMDPsExt

Example: Aiyagari model

Consider the household problem in the Aiyagari model. The state is (a, z), current assets and productivity, and the action is next-period assets. Define the model through the POMDPs.jl interface:

using QuantEcon, POMDPs, POMDPTools

struct Household{TZ<:MarkovChain,TA<:AbstractVector,TU} <:
        POMDPs.MDP{Tuple{Float64,Float64},Float64}
    r::Float64
    w::Float64
    sigma::Float64
    beta::Float64
    z_chain::TZ
    a_vals::TA
    u::TU
end

function Household(; r = 0.01,
                   w = 1.0,
                   sigma = 1.0,
                   beta = 0.96,
                   z_chain = MarkovChain([0.9 0.1; 0.1 0.9], [0.1; 1.0]),
                   a_min = 1e-10,
                   a_max = 18.0,
                   a_size = 200,
                   a_vals = range(a_min, a_max, length = a_size),
                   u = sigma == 1 ? x -> log(x) :
                       x -> (x^(1 - sigma) - 1) / (1 - sigma))
    return Household(r, w, sigma, beta, z_chain, a_vals, u)
end

POMDPs.states(am::Household) =
    Iterators.product(am.a_vals, am.z_chain.state_values)
POMDPs.actions(am::Household) = am.a_vals
POMDPs.actions(am::Household, (a, z)::Tuple) =
    (a_new for a_new in am.a_vals if am.w * z + (1 + am.r) * a - a_new > 0)
POMDPs.reward(am::Household, (a, z)::Tuple, a_new) =
    am.u(am.w * z + (1 + am.r) * a - a_new)
POMDPs.transition(am::Household, (a, z)::Tuple, a_new) =
    SparseCat([(a_new, z_new) for z_new in am.z_chain.state_values],
              am.z_chain.p[findfirst(==(z), am.z_chain.state_values), :])
POMDPs.discount(am::Household) = am.beta

Create an instance of the model:

am = Household(; a_max = 20.0, r = 0.03, w = 0.956)
a_vals = am.a_vals
z_vals = am.z_chain.state_values

A model defined this way can be used in two ways.

The POMDPs.jl interface can be used as a modeling interface for DiscreteDP, in place of building the reward and transition arrays by hand: pass the model to the DiscreteDP constructor to obtain a native DiscreteDP, with the model's states and actions attached as state_values and action_values:

am_ddp = DiscreteDP(am)            # sparse state-action pair form
results = QuantEcon.solve(am_ddp, PFI)

# Policy function decoded to state and action values
pf = DDPPolicyFunction(results)    # (a, z) -> next-period assets
a_star = [pf((a, z)) for a in a_vals, z in z_vals]

# Obtain the controlled Markov chain and stationary mean assets
mc = markov_chain(results)
K = sum(stationary_distributions(mc)[1] .* first.(mc.state_values))

DiscreteDP(am; sparse=Val(false)) builds the dense product form instead.

POMDPs.jl solver

DiscreteDPSolver is QuantEcon's solver for the POMDPs.jl interface: pass it to POMDPs.solve to obtain a POMDPs.Policy:

solver = DiscreteDPSolver(PFI)     # solver isa POMDPs.Solver
policy = POMDPs.solve(solver, am)  # policy isa POMDPs.Policy

# Query the policy and value function
s = (a_vals[117], z_vals[2])       # (a, z) ≈ (11.658, 1.0)
action(policy, s)                  # next-period assets
value(policy, s)

a_star = [action(policy, (a, z)) for a in a_vals, z in z_vals]  # same table

mc = markov_chain(policy)          # same chain as above

See Interacting with Policies and Implemented Simulators in the POMDPs.jl documentation for tools that work with policy. Like solve, simulate must be qualified as POMDPs.simulate. The native DPSolveResult remains available as policy.res.

Index

Interface

QuantEcon.DiscreteDP — Method

DiscreteDP(m::POMDPs.MDP; sparse=Val(true))

Tabulate an explicit-finite POMDPs.jl model into a DiscreteDP, with state_values = vec(collect(states(m))) and action_values = vec(collect(actions(m))) attached. With sparse=Val(true) (the default) the result is in state-action pair form with a sparse transition matrix; with sparse=Val(false) it is in dense product form, constructed directly (equivalent to to_product_form of the sparse import, without the intermediate). The keyword is value-typed so that the formulation is determined in the type domain.

The importer is index-free: it enumerates states(m) and actions(m) and builds its own value-to-index maps, so stateindex/actionindex and initialstate are not consumed. States and actions must enumerate unique, value-hashable objects; actions(m, s) defines the feasible set at s; rewards are the expected reward(m, s, a, sp) under transition(m, s, a) (3-argument models are covered by the POMDPs.jl fallback); states with isterminal(m, s) are encoded as zero-reward self-loops under every action in actions(m), so their value is exactly zero (actions(m, s) is not consulted at terminal states, so it may be empty there).

Rewards and transition probabilities are normalized to Float64, whatever the model's numerical element types. Transition weights are taken from transition(m, s, a) as-is, with only exactly-zero branches dropped: their validity (nonnegativity, summing to one) is the model's responsibility, as it is the caller's in the native constructors.

Arguments

m::POMDPs.MDP: Explicit-finite POMDPs.jl model to tabulate.
;sparse::Val(Val(true)): Value-typed tabulation-formulation flag: Val(true) for the state-action pair form with a sparse transition matrix, Val(false) for the dense product form.

Returns

ddp::DiscreteDP: The tabulated model, with state_values and action_values attached.

source

QuantEconPOMDPsExt.DiscreteDPPolicy — Type

DiscreteDPPolicy

Policy returned by POMDPs.solve(::DiscreteDPSolver, m). Supports action(policy, s) and value(policy, s), queried by state value; the native result is reachable as policy.res, and markov_chain(policy) returns the controlled Markov chain.

Fields

m::POMDPs.MDP: The source model.
pf::DDPPolicyFunction: Policy function, s -> optimal action value.
vf::DDPValueFunction: Value function, s -> optimal value.
res::DPSolveResult: The native solution.

source

QuantEconPOMDPsExt.DiscreteDPSolver — Type

DiscreteDPSolver{Algo,S}

POMDPs.jl solver based on the DiscreteDP solution methods: POMDPs.solve(solver, m) tabulates m via DiscreteDP(m), solves it, and returns a DiscreteDPPolicy. The type parameters carry the solution algorithm (Algo, one of VFI, PFI, or MPFI) and the tabulation-formulation flag (S, the Bool of the constructor's value-typed sparse keyword); the fields store the solve options. Construct with DiscreteDPSolver(algo; sparse, max_iter, epsilon, k).

Fields

max_iter::Int: Maximum number of iterations, passed to solve.
epsilon::Float64: Value for epsilon-optimality, passed to solve.
k::Int: Number of iterations for partial policy evaluation, passed to solve.

source

QuantEconPOMDPsExt.DiscreteDPSolver — Method

DiscreteDPSolver(algo=VFI; sparse=Val(true), max_iter=250,
                 epsilon=1e-3, k=20)

Construct a DiscreteDPSolver with solution algorithm algo.

Arguments

algo::Type{<:DDPAlgorithm}(VFI): Solution algorithm: VFI, PFI, or MPFI.
;sparse::Val(Val(true)): Value-typed tabulation-formulation flag, carried as a type parameter of the solver: Val(true) for the state-action pair form with a sparse transition matrix, Val(false) for the dense product form.
;max_iter::Integer(250): Maximum number of iterations, passed to solve.
;epsilon::Real(1e-3): Value for epsilon-optimality, passed to solve.
;k::Integer(20): Number of iterations for partial policy evaluation, passed to solve.

Returns

solver::DiscreteDPSolver: The solver instance.

source

QuantEconPOMDPsExt

Example: Aiyagari model

Modeling interface

POMDPs.jl solver

Index

Interface