Physical systems, concepts and principles are increasingly being used in devising novel and robust machine learning architectures. We illustrate this point with examples from two ML domains: sequence modeling and graph representation learning. In both cases, we demonstrate how physical concepts such oscillators and multi-scale dynamics can lead to ML architectures that not only mitigate problems that plague these learning tasks but also provide competitive performance.