Haskell Pipes -- Having a pipe consume what it yields (itself) -

i'm trying write webscraper using pipes , i've come part of following scraped links. have process function downloads url, finds links, , yields them.

process :: pipe item item (statet cstate io) ()  ....     (each links) yield  .... 

now want how recursively follow these links, threading statet through. realize there doing more idiomatic using single pipe bulk of scraper (especially start adding more features), i'm open suggestions. i'm going have rethink design when consider multithreading w/ shared state anyways.

you can connect pipe b m r side-effect through m parameter, swaps out monad pipe operating over. can use requeue links connecting downstream end of pipe pipe sticks links in queue , connecting upstream end of pipe pipe reads links queue.

our goal write

import pipes  loopleft :: monad m => pipe (either l a) (either l b) m r -> pipe b m r 

we'll take pipe downstream output, either l b, either left l send upstream or right b send downstream, , send ls in upstream input either l a, either queued left l or right a coming upstream. we'll connect left ls make pipe sees as coming upstream , yields bs headed downstream.

at downstream end we'll push ls left l onto stack. yield rs right r downstream.

import control.monad import control.monad.trans.state  pushleft :: monad m => pipe (either l a) (statet [l] m) r pushleft = forever $     o <- await     case o of         right -> yield         left l ->             stack <- lift             lift $ put (l : stack) 

at upstream end we'll on top of stack yield. if there isn't one, we'll await value upstream , yield it.

popleft :: monad m => pipe (either l a) (statet [l] m) r popleft = forever $     stack <- lift     case stack of         [] -> await >>= yield . right         (x : xs) ->             lift $ put xs             yield (left x) 

now can write loopleft. compose upstream , downstream pipes pipe composition popleft >-> hoist lift p >-> pushleft. hoist lift turns pipe b m r pipe b (t m) r. distribute turns pipe b (t m) r t (pipe b m) r. pipe b m r run whole statet computation starting empty stack []. in pipes.lift there's nice name evalstatep combination of evalstatet , distribute.

import pipes.lift  loopleft :: monad m => pipe (either l a) (either l b) m r -> pipe b m r loopleft p = flip evalstatet [] . distribute $ popleft >-> hoist lift p >-> pushleft