The rapid growth of on-device artificial intelligence has transformed how we use smart devices. By running machine learning models locally on edge systems like smartphones, smart cameras, and home automation nodes, we can achieve instant decisions, save network bandwidth, and protect user privacy. However, modern deep learning models require vast amounts of memory and computational power. This makes them difficult to deploy on small edge hardware, which runs on limited battery power and has restricted memory. To resolve this issue, this paper presents a practical guide to 'Green Machine Learning' techniques designed to run neural networks efficiently on low-power devices. We investigate three key optimization methods: reducing number precision (quantization), removing redundant model parameters (pruning), and training compact models using larger systems as guides (knowledge distillation). Using these methods, we evaluate an optimized image recognition model on two real physical systems representing different edge levels: a high-performance single-board computer and a resource-constrained microcontroller. Our findings show that combining pruning with 8-bit quantization reduces model size by up to 84.5% and improves inference speed by 4.3 times with less than a 1.2% drop in accuracy. On microcontrollers, these compression techniques reduced energy consumption by over 90%, allowing the model to fit and execute within very small memory boundaries. Finally, we discuss the trade-offs between performance and accuracy, and outline future directions such as dynamic security and runtime adaptations.