Discussion: Synaptic 2.x

So, we want to make Synaptic more mature.
I've created a design draft to think on it.
https://github.com/cazala/synaptic/wiki/Design-draft:-2.x-API

Let's discuss.

The most significant changes in design:

Pre-built architectures are dropped. Examples will be provided and should be used instead.
They are actually an examples of "how-to" and so they are usually useless for real projects.
biggest API parts (activate, train etc) are async now.
Every "moving part" used by an API is placed separately, and so if NN is simply created and loaded - there will be no need in trainer, optimization objectives, optimizers and everything else required for training. This is very useful for a client run, when only activation is needed without any training.
Multiple back-ends are supported. Each of them has its own strengths and weaknesses.
API fingerprint is as small as possible now.
Lot of stuff is configurable, lot of staff is pre-built.

Important question btw. Do we actually need Neuron as an exposed entity?

I guess that depends on the underlying implementation. Neuron and Layers can be replaced by just Networks that handle everything. The fact that Synaptic has Neurons is because when I first read Derek Monner's paper he described a generalized unit, that could use the same algorithm and behave differently according just to their position in the topology of the network (i.e. a self-connected neuron acts as a memory cell, a neuron gating that self-connection acts as a forget gate but the same neuron gating the input of the memory cell would act as an input gate filtering the noise. And all those neurons would essentially follow the same algorithm internally). That's what I found really cool about that paper and that's why I coded first the Neuron. Then the Layer is just an array of them, and a Network is an array of Layers. The advantage of having the individual units is that you can connect them in any way easily and try new topologies (Like LSTM w/wo forget gates, w/wo peepholes, w/wo connections among the memory cells, etc). But I know that the approach that other NN libraries take is more like matrix math at network level, instead of having the individual units. This is probably way better for optimization/parallelization so I'm up for it, as long as we can keep an easy and intuitive API, that allows the user to create flexible/complex topologies.

I'm not totally clear on what it means to expose a neuron, but it was extremely important for my application to be able to clearly see the neurons (using toJSON). I trained the network using synaptic and implemented the results in another program (excel).

From my understanding it would be more important to expose an ability to override/specify neurones' activation function than to have direct access to neurones. In that way a developer can concentrate on implementing higher level functionality, e.g. by stacking layers or networks and having access to neurones' activation functions to implement custom networks.
That assumed there are basic networks are implemented (convolutional, recurrent, Bayesian etc).

I would love to contribute to the new version if any additional help is still required.

Regards,
Oleh Filipchuk

On 16 Sep 2016, at 14:50, jocooler [email protected] wrote:

I'm not totally clear on what it means to expose a neuron, but it was extremely important for my application to be able to clearly see the neurons (using toJSON). I trained the network using synaptic and implemented the results in another program (excel).

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.

@jocooler

@jocooler That's a great note. From my point of view - removal of Neuron will significantly reduce memory consumption. However, this is a good reminder that we do need a human-readable export.

@Jabher

Thanks for spearheading this thread @Jabher. From a user's perspective, I might add having improved documentation and examples. Especially since the site is down now (#141), it's harder to reference back to examples I've found in the past.

Specifically, I think for my use case an LSTM may be a more natural approach, but I am hesitant to test it because I have not trained one before and the mechanism for RNNs seems quite different. Having more than 1 example (preferably 3+) would help as users can pick the one that best matches their use-case. It might help to encourage users to contribute their own examples as well (maybe something I can do if I figure this out myself).

Another point should be on optimizations. I think a big reason people are using other libraries is due to limitations, especially in memory. It could help to have a short guide on the Wiki discussing how to set Node to allocate more memory before getting OOM, or using a mini batch approach for strategies that support it. Also, regarding exposing the Neuron, I may suggest something similar to compilers where the toJSON method can be human-friendly in debug mode and machine-friendly otherwise. I'm seeing my memory filled with Neurons when conducting a heap analysis.

@olehf

@olehf Convolutional (1d and 2d + max/avg pooling for them) are mentioned in design document, same for RNN - I think GRU and LSTM would be enough for average user.

Activation functions should definitely be passable to any layer, only issue we can encounter is a custom implementation of them (as we will need multiple implementations of one function) so we should keep as much as possible in design.

Help is (as usual) always appreciated. It is a lot of work to do, and any contribution will be significant. As soon as we will decide that we're good with this design, GH project board will be created with tasks to do.

@schang933

@schang933 there's new temp website - http://caza.la/synaptic/. Old one will be back soon.

Speaking about examples - this is a good concern, but first we should implement an API itself.
Good set of examples can be a https://github.com/fchollet/keras/tree/master/examples (did I mention that Keras is documented in a great way?).

Another point should be on optimizations. I think a big reason people are using other libraries is due to limitations, especially in memory. It could help to have a short guide on the Wiki discussing how to set Node to allocate more memory before getting OOM, or using a mini batch approach for strategies that support it.

That's totally correct, but we should keep in mind that Node.js is not the only runtime target. Multiple browsers (latest Edge, Firefox and Chrome) with WebGL/WebCL and workers are also a runtime targets, and nodejs-chakra is also a preferrable option as it looks like it do not have memory limitations at all (it should be checked).

About Neuron exposion and keeping human-readable output - I totally agree.

Hey, thanks for this great library! It is really nice that its development is still running.

I am not an ML expert, and synaptic has helped me so much to understand how the neural networks work ,thanks especially to the prebuilt architectures. If you are dropping them from the core repo, please move them to a separate one, maybe with the examples, because it makes really simple for ML non-experts to get initiated and running.

Cheers

Most JavaScript related libraries are not actively maintained any more. Hope this one can keep going!
Hope one day I can have enough knowledge contributing on this library.
Cheers!

import {
  WebCL,
  AsmJS,
  WorkerAsmJS,
  CUDA,
  OpenCL,
  CPU
} from 'synaptic/optimizers';

I think those shall be separated packages. In order to keep the original package as small and simple as possible.

That will help us to create browser bundles.
Will reduce the scope of the main project, helping us to create specific tests for each optimiser.

I agree but it should be both options for easy installation
On Sep 25, 2016 16:10, "Agustin Mendez" [email protected] wrote:

import {
WebCL,
AsmJS,
WorkerAsmJS,
CUDA,
OpenCL,
CPU
} from 'synaptic/optimizers';

I think those shall be separated packages. In order to keep the original
package as small and simple as possible.

That will help us to create browser bundles.

Will reduce the scope of the main project, helping us to create
specific tests for each optimiser.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
#140 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMo8NrmvKGffvCAvsxLnVOqZlqxyUsIKks5qtnLVgaJpZM4J6oXe
.

@menduz

@menduz I'd suggest multiple modules + 1 meta-module ("synaptic" itself) to expose them all in that case?

There's a nice line in TensorFlow docs:
https://www.tensorflow.org/versions/r0.11/api_docs/index.html

Over time, we hope that the TensorFlow community will develop front ends for languages like Go, Java, JavaScript, Lua, R, and perhaps others. With SWIG, it's relatively easy to develop a TensorFlow interface for your favorite language.

ASAP we actually need only few functions, some incomplete port of TF would not be so hard to use, and it will deal with most of server-side performance issues, so we will actually be able to train with good speed, I think.

I'd love to see reinforcement network implemented.
More utils will be cool - I couldn't even find any npm module that normalize numbers. JS is missing many important tools to work with data.
In this section I could happily contribute.
some options:

normalizeNum(min, max, num) =>  0->1 // (curried?),
deNormalizeNum(min, max,  0->1) => num // (curried?), 
normalizeNumericalArray ([num]) => [0->1] // min and max from array
normalizeCategoricalArr([string]) => [0|1] //based on uniqueness. 
etc...

@playground

@playground As of right now I don't have anything with those libraries.
@buckle2000 ok.

Also there is a new Math library being built for JavaScript for machine learning because the default Math library in JavaScript and NodeJS is very erroneous. And it only offers a few mathematical functions and advanced trigonometric functions and other n-dimentional functions are not present and the ones that are present are using the default math library in JavaScript and NodeJS and hence every library on npm on mathematics is erroneous and not safe for Machine Learning. And they only use upto float32 which is pretty limited when it comes to ML. So, check out this math library which is being built very industry standard. And every mathematical functions have a citation paper associated with it to prove its accuracy.

https://github.com/stdlib-js/stdlib

Hello and Thanks for the library,
I'm porting the underlying algorithm on microcontroller (c language), supporting Perceptron an LSTM. The goal is to build and train a network using synaptic over decent cpu for pattern recognition and/or time series analysis. And finally upload the trained model to the edge. In this process, the exchange format will be central. I'm quite happy with json, but wanted to know how the json v2 will look like. Also, if you can introduce a Version properties such { version:"2.x", neurons:[...],..}.
Ready to help with somes topics if you need.

So this project is officially dead?

Can someone help me on how to run the "paint an image" demo locally on my system?

if i understand correctly, you want to reproduce the demo on localhost. For the purpose, you have the code of the demo at
http://caza.la/synaptic/scripts/controllers/paint-an-image.js
you simply have to embed/modify it into local html page within synaptic lib referenced locally.
The graphical presentation or the use of any framework such Angular is up to you.

is it support cuda or gpu?

Overall, I was thoroughly impressed with the implementations provided in Synaptic in v1.
However, I didn't see any documentation in regards to Tensors.

Obviously, Tensors are not necessary, but, do help you follow the correct rules when implementing models. I think it would be a nice thing to add as a backlog, lower priority item.

Maybe even a configuration setting where you can configure Tensors on or off for added flexibility...
I feel this suggestion may be adding unneeded scope creep though.

@freelogic

@freelogic, check issue #245.

I believe this is pretty up in the air at the moment. But good integration with GPUs would be make or break for me migrating my work over to Synaptic.

@Jabher

@Jabher What's the current situation with v2?

Is project still alive?

is there gonna be a version 2...? ;-;

Use can try this url https://github.com/bigstepinc/jsonrpc-bidirectional בתאריך יום ב׳, 6 בספט׳ 2021, 22:01, מאת Martin M. ***@***.***

…

: is there gonna be a version 2...? ;-; — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#140 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ADFDYNXSNHY7MAJWAZ7MMADUAUFZHANCNFSM4CPKQXPA> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.

That's a URL to a bidirectional rpc library...

Navigation Menu

Repository navigation

Discussion: Synaptic 2.x #140

58 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Discussion: Synaptic 2.x #140

Description

Activity

Jabher commented on Sep 12, 2016

cazala commented on Sep 13, 2016

jocooler commented on Sep 16, 2016

olehf commented on Sep 16, 2016

Jabher commented on Sep 18, 2016

schang933 commented on Sep 19, 2016

Jabher commented on Sep 19, 2016

Jabher commented on Sep 20, 2016

arqex commented on Sep 20, 2016

hongbo-miao commented on Sep 24, 2016

menduz commented on Sep 25, 2016

yogevizhak commented on Sep 25, 2016

Jabher commented on Sep 25, 2016

Jabher commented on Oct 3, 2016

yonatanmn commented on Oct 8, 2016

58 remaining items

vickylance commented on Sep 3, 2017

pandaGaume commented on Jan 11, 2018

vickylance commented on Jan 21, 2018

dhrubesh commented on Jan 22, 2018

pandaGaume commented on Jan 22, 2018

freelogic commented on Mar 19, 2018

ChrisCates commented on Aug 3, 2019

ChrisCates commented on Aug 3, 2019

Berkmann18 commented on Aug 20, 2019

rumax commented on Mar 9, 2020

martinmaroyan commented on Sep 6, 2021

yogevizhak commented on Sep 6, 2021

martinmaroyan commented on Sep 14, 2021

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions