Nick Mark, MD @nick

**Philo Sophies** @Philosophies@troet.cafe · Apr 14

**Philo Sophies** @philosophies@planetearth.social · Mar 30

**RoundSparrow** @RoundSparrow@mastodon.social · Jun 26, 2024 *

**RoundSparrow** @RoundSparrow@mastodon.social · Jun 26, 2024 *

**Matt Willemsen** @mattotcha@mastodon.social · Dec 21, 2023

Continued thread

**Victoria Stuart** @persagen@mastodon.social · Sep 11, 2023 *

Sep 11, 2023 *

Victoria Stuart @persagen@mastodon.social

Addendum 10

1 Wide Feedforward All You Need
https://arxiv.org/abs/2309.01826

* 2 non-embedding components in transformer architecture: attention; feed forward network
* attention captures interdependencies betw. words regardless of position
* FFN non-linearly transforms ea. input token independently
* FFN (sig. fract. parameters) highly redundant
* modest drop in accuracy removing FFN on decoder layers & sharing single FFN across encoder

#ML #transformers #NeuralNetworks

**Victoria Stuart** @persagen@mastodon.social · Jul 11, 2023 *

Jul 11, 2023 *

Victoria Stuart @persagen@mastodon.social

Absorbing Phase Transitions in Artificial Deep Neural Networks
https://arxiv.org/abs/2307.02284

To summarize, we believe that the this work places the order-to-chaos transition in the initialized artificial deep neural networks in the broader context of absorbing phase transitions, & serves as the first step toward the systematic comparison between natural/biological & artificial neural networks.
...

arXiv.orgAbsorbing Phase Transitions in Artificial Deep Neural NetworksTheoretical understanding of the behavior of infinitely-wide neural networks has been rapidly developed for various architectures due to the celebrated mean-field theory. However, there is a lack of a clear, intuitive framework for extending our understanding to finite networks that are of more practical and realistic importance. In the present contribution, we demonstrate that the behavior of properly initialized neural networks can be understood in terms of universal critical phenomena in absorbing phase transitions. More specifically, we study the order-to-chaos transition in the fully-connected feedforward neural networks and the convolutional ones to show that (i) there is a well-defined transition from the ordered state to the chaotics state even for the finite networks, and (ii) difference in architecture is reflected in that of the universality class of the transition. Remarkably, the finite-size scaling can also be successfully applied, indicating that intuitive phenomenological argument could lead us to semi-quantitative description of the signal propagation dynamics.

#NeuralNetworks #MeanFieldTheory #SignalPropagation

**Phil McAleer** @mcaleerp@mastodon.social · Mar 4, 2023

**The vOICe vision BCI** @seeingwithsound@mas.to · Feb 2, 2023

Feb 2, 2023

The vOICe vision BCI @seeingwithsound@mas.to

Distinct early and late neural mechanisms regulate feature-specific sensory adaptation in the human visual system https://www.pnas.org/doi/10.1073/pnas.2216192120 fatigue and sharpening "at different points in the sensory processing cascade, likely reflecting distinct #feedforward and #feedback interactions"; #neuroscience

Recent searches

Search options

Administered by:

Server stats:

#feedforward