Repository navigation
Symbolic loops (like "scan" in Theano) #208
Description
Activity
Our white paper mentions a number of control flow operations that we've experimented with -- I think once we're happy with its API and confident in its implementation we will try to make it available through the public API -- we're just not quite there yet. It's still early days for us :)
I see. Great news that that's in the pipeline. Can't wait until it's publicly available. ;-)
- added a commit that references this issue
on Jan 3, 2016 +1
See the highly alpha and unsupported control_flow_ops.While, the
TensorArray python class, and nn.dynamic_rnn. All at HEAD.
On Feb 16, 2016 7:48 AM, "Dmitrij Koniajev" [email protected]
wrote:+1
—
Reply to this email directly or view it on GitHub
#208 (comment)
.Any update or progress schedule?
You can now use control_flow_ops.{map, foldl, foldr} with forward and backprop, and you can call these functions from inside each others' lambdas. For RNN, you can use dynamic_rnn which does the same. If you have more complex eneds you can comment them here or use control_flow_ops.While and TensorArray. I'm marking this as fixed.
Indeed not, but I believe {map_fn, foldl, foldr} are ready to be added by
referencing them in the header. We'll get a CL out soon.On Wed, Mar 9, 2016 at 9:21 AM, Vijay Vasudevan [email protected]
wrote:@ebrevdo https://github.com/ebrevdo: are those functions:
- public
- documented?
—
Reply to this email directly or view it on GitHub
#208 (comment)
.A toy RNN example with these functions would go a long way; especially since they do not directly correspond to the
theano.scanfunctionality.See the implementation of dynamic_rnn for a comprehensive example.
On Mar 9, 2016 6:59 PM, "rakeshvar" [email protected] wrote:A toy RNN example with these functions would go a long way; especially
since they do not directly correspond to the theano.scan functionality.—
Reply to this email directly or view it on GitHub
#208 (comment)
.6 remaining items
@rdipietro I tried to extend your code to a vanilla GRU. Also removed the clipping therefore and replaced Gradient Descent with Adagrad.
Consequently convergence is now much faster.
Can you please verify the code ?
http://paste.ubuntu.com/16534925/
(I know it's much dirty!)Reacted by Rakeshvara R.A.How to use map and scan together??????? Anyone have any idea?
x=tf.constant(
[[[1,2,3],[10,20,30]],
[[1,2,3],[10,20,30]],
[[1,2,3],[10,20,30]],
[[1,2,3],[10,20,30]]])def sum(x): return tf.scan(lambda y, z : tf.add(y,z), x)
tf.map_fn(sum ,a )Then errors pop up!!!! Is there any way using these together?
@rdipietro I have been through your example notebook but how to use it for batch????The implementation of both scan and map_fn uses TensorArray. Nesting scan inside map_fn is essentially nested while loops. Unfortunately, there is a known bug when TensorArray and nested while loops are used together. We have been working on a fix.
Ok thanks.
I am using scan to deal with different sequence length. But having problem to deal with batch! @rajarsheem can you write your solution for my above problem??? If I use for loop then its same like online learning! No need to use batch then! So no need to use GPU then!
Please Google solve this nested while loops. ...
You can see here for a more complex example, with a batch size > 1. But you'll need to either a) write not-so-clean code that's efficient (as they do in TensorFlow officially) or b) write clean code that is less efficient (this is what I do; I just wrap shorter sequences in time until all sequences are the same length, which simultaneously makes short sequences "count" just as much loss wise as long sequences).
@rdipietro Thanks a lot. I will go through the your example!
@rdipietro can you verify my graph, whether it is set up right ? (it is a very very simple example)
The problem of scan and map together is solved by the latest release of Tensorflow.
- added a commit that references this issue
on Oct 30, 2019 - added a commit that references this issue
on Nov 4, 2019 - added a commit that references this issue
on Nov 19, 2019 - added a commit that references this issue
on Dec 6, 2019
I've been wondering if there are plans to add symbolic loops to TensorFlow because I feel like this is a major feature when it comes to variable length sequences. Finite unfolding (with bucketing) seems like a dirty hack to me and since (as far as I understand) TensorFlow is meant for deployment, too, how do you envision using it for seq2seq translation given that you don't know ahead of time how long the generated sequence will be?
Thanks,
Sigurd