idavidg

https://data-information-meaning.blogspot.com/2020/12/14-kohenen-feature-maps-new-embedded.html Kohenen Feature Maps are an interesting blend of two processes, simultaneously creating a model and imposing on that model a new metric space, it works by taking a simple KMeans learning function and blend it with a graph. ... This is an important point, may be I should expand on it now, but I will get back to it again. Just quickly mention, two aspects of creating the embedded space. 1) It echoes the original data, things that are similar in the original space are similar in the new embedded space, sounds good right, not really, it keeps the lower level connected to the higher level, there is less learning going on, since the higher levels are not free from the constraints of the lower level. 2) When the data distribution changes, the model is locked into the original metrics (true that Joshua Benjio tries to avoid this with 'attention' but why create a problem and then try to work around it)

https://data-information-meaning.blogspot.com/2021/01/165-phase-transitions-measure-of.html Neat right, KMeans spreads its representations equally across the entire dataset, minimising the global loss of information The thing to notice with faddc is that the representation is very stable up to a point, at a certain point there is a dramatic shift in the representation.

https://data-information-meaning.blogspot.com/2021/01/155-multi-scale-defines-learning.html A car is a complex system, that provides a unique service that its parts cannot provide. With this new language I can communicate ideas that are unique, and can not be communicated at the lower level. The whole is greater than the sum of its parts, a car provides a language to describe concepts that are more than just wheels and chassis. The greater context, multiple people utilise cars, enables multiple instances of cars to be correlated and new conceptual ideas to emerge. Thus, within the higher level language of cars I can talk about traffic, multiple cars attempting to service a destination in a timely manner.

https://data-information-meaning.blogspot.com/2021/01/16-phase-transitions-measure-of-learning.html Phase transitions, a measure of learning Phase transitions demonstrate a loose coupling. A tight coupling like in KMeans mirror the distortion at each level. A loose coupling enables the higher level to move at a different pace, disjoint, from the lower level. This separation between levels, indicates that the levels represent different descriptions of the data, they speak different languages.

https://data-information-meaning.blogspot.com/2021/01/155-multi-scale-defines-learning.html Why have I been so focused on multi-scale data? Because learning can only occur when there are multiple scales. Why is that true, is sounds nonsensical, what is the relationship between the scale of a set of data and learning that data. ... Learning creates a new descriptive space, an embedding, that is unique and distinct from the original descriptive space of the data. A model is created to map from the original descriptive space to the new descriptive space. The result of this mapping is a new representation that cannot coexist within the original framework.

https://data-information-meaning.blogspot.com/2020/12/15-agglomerative-online-clustering.html Wouldn't it be nice if there was an algorithm that created a representation while streaming data, not sensitive to initial starting conditions and able to capture the scale of the data... Agglomerative online clustering, here is a link to my paper: https://www.cs.huji.ac.il/~werman/Papers/guedalia_etal99.pdf ... So what to do when the limit is reached, consolidate memory, i.e. learn! That is step two in my algorithm, merge the two closest centroids. This frees up a memory allocation. More importantly it creates an abstraction of the original data locally. With a goal of minimising the local loss of information. ...

https://data-information-meaning.blogspot.com/2020/12/13-kmeans-as-means-to-describe-failed.html KMeans is finding the representation that minimizes the global distortion, i.e. if I can only communicate a single point, what is the message (the single point) that best represents the data? The average (mean) of all the data. Why, since the distortion of the original data is minimized, the distance from the average data point to the other data points is the smallest. This is true since the data has a Gaussian distribution, so the center is more populated than the edges. ... Here can be clearly seen that KMeans chooses to communicate the average message, which dilutes the uniqueness of each independent element. Global maximization is a failed strategy, it is not scale sensitive, since it treats the data as if there is a global single scale. Another way to say this, I am trying to learn a particular distinction but constrained not to forget the general idea. Or, KMeans attempts to memorize the data, if it can't memorize it in its entirety it will minimize the loss of memorization! not the loss of learning the data. Memorization is a failed strategy

https://data-information-meaning.blogspot.com/2020/12/12-data-information-and-meaning.html Learning increases information in a system. Why is communication necessary for learning? It would seem to be unrelated? Communication provides two aspects, a constraint and a method of mapping two systems to each other to create meaning. Some basic definitions first. Data: a recording of sensory perceptions, 'observations' might better to describe a sensory perception Sensory receptive field: field of external stimuli that results in an observation by a sensor Information: a measure of signs (data and their relationships) in a system, how many signs are present in a system, typically calculated either as independent information (Shannon) or dependent information (LZW complexity information) or prescriptive information (Kolmogorov information)

https://data-information-meaning.blogspot.com/2020/12/memorization-learning-and-classification.html Memorization - Store the contents of a set of observations Learning - When a constraint is imposed with a requirement to communicate outside the system, then learning occurs, a new representation of the observations is necessary, more efficient, abstractions occur. Classification - A judgment statement, good/bad, a category is labeled with a quality While learning provides a better quantity, more efficient representation, classification provides a quality. Notes: 1. Short term memory is constrained and communicated to long term memory typically at night, most learning occurs at night 2. Maimonidies in the Guide opens with the distinction between truth-falsity vs. good-bad