University of  Rome  "La Sapienza"

DIET -  Department of Information Engineering, Electronics and Telecommunications  

 

Course Info: 

Neural Networks

Master (Laurea Magistrale) in: Artificial Intelligence and Robotics

 

Prof. Aurelio Uncini  (info: aurelio.uncini _AT_ uniroma1.it)

 

Information

 

Course begins  September, 28 2016,

ends December 22.          

 

Lesson timetable

Wedneday     8:30-10:10 A4

Wedneday   10:15-11:45 A4

Thursday      8:30 -10:00 A7

 

Room A4: Via Ariosto

Room A7: Via Eudossiana

 


Tools  - FTP download

Links

International labs link

 

Course description

 

This course introduces neural networks and others soft computing methods which, unlike hard computing, are tolerant of imprecision, uncertainty and partial truth. Topics include: neural networks model, architectures, mathematical property and learning algorithms; optimization algorithms for soft computing methods; introduction to fuzzy-set; application on intelligent data analysis, patterns recognition, multi-sensors data-fusion, blind source separation.

 

The student acquires basic and specific knowledge related to the discipline. In particular, it is able to implement (and evaluate the performance) of complex systems for pattern recognition and dynamic data processing.

 

Final exam modalities

 

After the study of the course material and a preliminary discussion with the teacher, the examination consists in the discussion of an assigned home work.

 

The course will be taught in English

 


Course syllabus

Introduction to computational and biological inspired intelligent circuits (ICs)

ICs classification according to: input-output characteristic, learning paradigms and rules, the aggregation and topology. ICs applications: static data processing, dynamic data processing.

 

Fundamentals of Linear Adaptive Systems

Normal equations in Wiener-Hopf and Yule-Walker notations. Least-Squares (LS) method and variants: weighted LS, regularized LS. Adaptive first and second order algorithms: steepest-descent (SDA) and least mean square (LMS) algorithms, recursive LS (RLS) and variants, affine projection (APA) and exponentiated gradient algorithms. General law of adaptation. Multi-channel extension. Convergence property and stochastic performance evaluations, for the first and the second order algorithms.

 

Human brain and biologically inspired intelligent circuits

The biological brain. McCulloch and Pitts’s neuron model. Neuron models and classification problem. Activation functions. Nonlinear classification functions. Definition of artificial neural networks (ANN). ANN topologies, learning paradigms and rules. Mathematical properties of ANN.

 

Supervised learning by error correction

Multi layers perceptron (MLP) and back-propagation (BP) learning algorithm. Learning in dynamic neural networks. Signal-flow-graph approach for gradient calculation of adaptive dynamic systems. Autoencoders.

   

Regularized networks, Extreme Learning Machine and Reservoir Computing

Functions Approximation and Interpolation. Tikhonov Regularization Theory and Radial Basis Function networks. RBF Learning. Self Regularizing Networks Extreme Learning Machine and Reservoir Computing .

  

Statistical learning theory of Vapnik and Support Vector Machine

Loss function and risk functional. The VC-dimension and rate convergence bounds of the learning processes. Bias vs variance dilemma. Support vector machine (SVM) definition  and learning algorithms.

 

Dynamic stochastic neural networks

Hopfield networks, Boltzmann machine, competitive learning, Kohonen’s self organizatoin feature maps, adaptive resonance theory.

 

Deep neural networks

Gradient diffusion and vanish gradient problem. Autoencoder and Stacked Autoencoder. Restricted Boltzmann machine,  Convolutional neural networks. Neural networks for big-data problems.

 

Information theory based unsupervised learning

Principal component analysis (PCA). Independent component analysis (ICA). Learning algorithms for PCA and ICA. Natural gradient learning algorithms. Blind Signal Separation. Infomax and Amari algorithms.

 

Soft computing and data mining

Optimization problems with noisy or unknown performance surface. Global optimization algorithms – heuristic approach: simulated annealing and tabu search; Darwinian approach: genetic and evolutionary algorithms; swarm intelligence based algorithms: ant colony optimization, particles swarm optimization. Introduction to fuzzy logic, fuzzy set and fuzzy set operations. Symbolic vs. linguistic knowledge representation

 

Applications

Static and dynamic pattern recognition, complex dynamic system modelling, Deep Learning on Big-Data, time-series filtering and forecasting, multi sensors data-fusion.

 



References

Text books and papers

  • A. Uncini, Computational and Machine Intelligence, Lecture notes ed. 2015. available (only) at A4Z copy center, (via della Polveriera, 13, 00184 Roma).

  • A. P. Engelbrecht, Computational Intelligence: An Introduction, Second Edition, John Wiley & Sons, 2008.
  • Li Deng and Dong Yu, Deep Learning Methods and Applications, Foundations and Trends in Signal Processing 7:3-4, 2014
  • Y. LeCun, Y. Bengio, G. Hinton, Deep Learning, Nature, May 2015
 

 

Further reading books

  • S. Haykin, “Neural Networks”, MacMillan College Publishing Company, NY, 2009.

  • Thomas Weise, “Global Optimization Algorithms Theory and Applications”, University of Kassel, http://www.it-weise.de/

  • R.O. Duda e P.E. Hart, “Pattern Classification and Scene Analysis”, J. Wiley & Sons, 1973 (MAT 68-1973-03IN, ING2 EL.0069).

  • J.-S.R. Jang, C.-T. Sun, E. Mizutani, “Neuro-Fuzzy and Soft Computing”, Prentice Hall, 1997.

  • A. Uncini, Fundamentals of Adaptive Signal Processing - Springer, Febrary 2015.