NN Visual API project

(a work in progress)

The overview

This project was an idea I had been considering for some time. I noticed I was rewriting a lot of code to achieve similar things.
It would instead be nice to have projects that were intuitive and quick to understand. The pytorch framework is something many are familiar with, but there is a lot of boiler plate. Furthermore it is easier to move visual concepts to code if the visual objects can write the code themselves. These are both good motivators, but it is important to note that pytorch is widely used specifically used for its ability to deeply customize things, but at runtime and at “author time”. Because of this, I wanted to also incorporate a way to write code within the application, to fill in the gaps for the models that one would typically be writing for a research paper (i.e. something that is sufficiently novel will not have a way to out of the box implement it)

I figured I could also use this opportunity to explore javascript more deeply. I had been meaning to do this for a while, and what better way to acquaint myself with the language than to require it to work exactly as expected! The browser is a great place to test out style as well, so javascript was a natural candidate for frontend behavior (especially since the MVC loop is baked into the browser, and is exactly what a real time application requires)

The architecture

While it is tempting to immediately start writing something up, knowing how you want it to work, from a high level is important for many reasons. In my opinion, the most important of these is destination. Having a destination in terms of functionality enables test driven development, which enables API development, which enables principled implementation of functionality, and narrows the selection for design patterns being implemented.

  • Need a way to keep track of relationships for writing the code
  • Need a way to keep track of object properties
  • Need a way to keep track of events
  • Need a way to add new objects

These were solved as described JSON objects easily map to python dictionaries. Dictionaries that are not flat contain a lot of structure in their hierarchical organization, and this is perfect when describing objects being built. So long as fields were used consistently and the top level json layout was itself consistent, the code would be easily extensible and flexible. Signals were immediately incorporated, since signals with events are a popular (and rightfully so) design pattern for user interfaces. I built out my own signalling system as well, since I wanted to understand the characteristics of the design pattern and its constraints.

There were two tiers of signals. One was overall application signalling, and one was dedicated to the backend compiling behavior of the models. This was useful because it didnt mix responsbilities of the overall architecture. The top level was dedicated to the overall application state management, and one of those application states is interfacing with the model construction. This was the reason for this method of implementation.

The frontend required graphical and user experience oriented design. I proceeded initially with hard coded objects, that I eventually moved to more extensible design. This was motivated after I noticed how much overlap there was in the intrinsic structuring of data, and that extensions to this would take similar form. . The design process was honestly pretty rushed, but it was principled. Json is used often in extensible design because its structure is self describing and is “relatively” easy to parse. I figured it would be useful to make extensibility a first class citizen in the design and implementation, since 1) the runtime performance of the application is not of utmost performance, so json oriented extensions should be fine, especially since the browser hosts the rendering of the frontend. Because of this, i created a text box that would send instructions to an ipython kernel backend, which shared the same data path as signals. This is fine since the python instructions were, themselves, sent in signals packaged via json data as well.
This is however a work in progress, since i quickly realized how difficult it could be for building out the frontend behavior.

The frontend behavior was