Using AI to Cut Diesel Emissions
Deep Reinforcement Learning Slashes NOx Emissions to Near Zero
Partner with Us for Success
Get expert guidance from MathWorks Consulting on tailored solutions to help you achieve your goals.
Nitrogen oxides, a byproduct of burning fossil fuels, are among the toughest regulated pollutants and one of the most damaging to human health. The chemicals, collectively called “NOx,” make up the brown smog that, despite tighter air quality controls, still hovers over some city skylines.
In diesel engines for industrial engines, selective catalytic reduction (SCR) catalysts have been mainly used in the 56 kW or higher output range in response to the NOx regulation levels of the EPA Tier 4 Final regulations that have been applied since 2014. Regulations will continue to become stricter, and further reductions in NOx levels will be required, resulting in more complex and advanced systems that will require more development time than ever before. Therefore, further development of efficiency will be required. At Yanmar America, a company that produces agricultural equipment, energy systems, and industrial engines, engineers are pioneering the use of deep reinforcement learning to refine the technique.
“We need a further 90% reduction to meet expected new standards for engines over 56 kW, so that we have near-zero NOx coming out of the tailpipe of our SCR engines,” says Martin Muinos, a research and development engineer at Yanmar America.
The California Air Resources Board sets standards for limiting pollutants from off-road diesel engines. Its newly proposed Tier 5 standards, which will be phased in over the next decade, require manufacturers to cut NOx emissions by 90% in some power categories.
Yanmar’s 4TN107FTT engine, equipped with SCR technology. (Image credit: Yanmar America)
A few years ago, Yanmar America engineers began workshopping ways to meet this requirement in their new engines. The engines were already equipped with SCRs, which minimize NOx emissions through a chemical reaction between ammonia and NOx. Ammonia is introduced into the exhaust upstream of the SCR catalyst through a diesel exhaust fluid (DEF) injector. DEF is a urea-water solution, which breaks down into ammonia in the hot exhaust gases. Aided by a catalyst, the ammonia breaks down the NOx into nitrogen gas, a natural component in the atmosphere.
But some NOx still makes it through this cleaning process, and ammonia, itself an unhealthy pollutant, can leak into the air. This unfortunate drawback is called ammonia slip. “You want to try to have as much ammonia as possible stored on the SCR to increase your NOx conversion capability,” says Muinos. “There are challenges with that because ammonia storage is temperature dependent.” For instance, quick increases in exhaust gas temperature can result in ammonia slip.
Diagram depicting the process of minimizing NOx emissions using SCRs. (Image credit: Yanmar America)
The size of the SCR is another limiting factor. “SCR performance depends heavily on size, meaning they don’t have infinite space for ammonia injections,” says Shota Nomura, a testing engineer at Yanmar America.
From Testbench to Simulation
Typically, SCR development involves a lengthy calibration process to ensure appropriate NOx reduction, which can total over 240 work hours for the SCR calibration project. “There are 20 or more calibration maps that outline the SCR controls, and we need data to calibrate those maps,” explains Muinos. “To collect this data, you need to run an engine testbench for a few weeks at a time, calibrate the maps, and then validate the performance.”
“Simulink is like a sandbox for controls development.”
“There is no autonomous way to calibrate the system,” says Nomura. “Since SCR adjusts several maps that affect each other, performance is affected by the adjustment. To adjust it while taking into consideration the optimal balance based on the test results, a large amount of manual adjustment work was required.
To reduce the time and costs of physical testing, the team at Yanmar switched to a model-in-the-loop approach, using Simulink® and separate catalyst simulation software, to model their SCR. “If we did not have Simulink, then we would have to propose a change to our controls, submit that to our engine control unit (ECU) supplier, have them develop the control logic, have them develop the software for the controller, and then test it on an actual testbench,” says Muinos. “But Simulink is like a sandbox for controls development.”
Using the sandbox, the team quickly realized that their current SCR control methods wouldn’t be enough to meet the emerging Tier 5 standard, which aims to slash NOx emissions to near zero.
A screenshot of the Reinforcement Learning Training Monitor available in Reinforcement Learning Toolbox. It depicts the progress graph while training a DRL model that Yanmar used for this project. (Image credit: Yanmar America)
Nomura wondered if AI, specifically deep reinforcement learning, could help. “But I’m a mechanical engineer,” says Nomura. “I don’t have a lot of AI knowledge.”
Since the team was already using MATLAB® and Simulink, Nomura explored online resources from MathWorks and used MathWorks Consulting Services to see whether deep reinforcement learning could help them determine the best DEF dosing rate for their SCR.
Reinforcement learning is a type of AI that learns by taking in information from the environment to produce a desired result, learning by trial and error. The reinforcement learning agent is “rewarded” when it produces a desired result.
“The thought process behind using reinforcement learning is we are going to use the AI to find the best dosing profiles or best ammonia storage profiles for emission test cycles,” says Muinos. “Then we will study the results and develop controls to target those profiles.”
“Before, we didn’t have any way to determine the maximum performance,” says Nomura.
Using Reinforcement Learning Toolbox™ and Deep Learning Toolbox™, MathWorks consultants helped Nomura choose from the different reinforcement learning algorithms provided and settle on the right one for Yanmar’s project.
“After we worked with the team and analyzed the advantages and drawbacks of Yanmar’s suggested approach, we understood that the deep Q-network (DQN) algorithm would help them develop a good reinforcement learning agent,” says Mohammad Muquit, a principal technical consultant at MathWorks.
Simulink scope screenshot depicting the amount of NOx emission while simulating with Yanmar’s system default value (yellow line) vs. the DRL model developed in this project (blue line). (Image credit: Yanmar America)
Learning from Experience
Equipped with the DQN algorithm, the Yanmar team began “teaching” their deep reinforcement learning (DRL) agent to find the optimal ammonia dosing profile for SCR performance.
With the model output, the team could see how the agent’s suggested calibration values matched up with their calibration values from existing engines. Reinforcement Learning Toolbox visualizations helped them understand the model results.
“With no experience in AI, I didn’t expect to be able to complete a reinforcement learning project … but MathWorks Consulting Services helped me handle the necessary parameters and learn how reinforcement learning works.”
“We’re comparing the results from the reinforcement learning agent to our calibration results and trying to understand why they’re different,” says Muinos. “Once we have a good understanding of why the reinforcement learning agent is producing better emission results, we can understand what controls to implement. It gives us a better starting point for calibration.”
After running the simulation, which takes around 30 minutes, the agent provided an optimal dosing profile, which resulted in a 60% improvement in NOx emission reductions, according to Nomura.
The agent halved the calibration time, shrinking the project’s total hours by 30%. “It took only six months—we didn’t expect such a short development timeline,” says Nomura. “This went way beyond my expectations.”
The time savings and reduction in physical testing also resulted in 41% cost savings compared to the normal manual calibration process.
“With no experience in AI, I didn’t expect to be able to complete a reinforcement learning project,” says Nomura. “But MathWorks Consulting Services helped me handle the necessary parameters and learn how reinforcement learning works.”
Right now, the AI-calibrated SCR tool provides Yanmar engineers with a better starting point for calibration when developing their engines’ SCRs. The next step is rapid control prototyping. The Yanmar team recently acquired the hardware to put their modeling results to the test. The team plans to automate this process further. Currently, the DRL agent searches for optimal values, but humans still need to incorporate those values into the ECUs. In future projects, the Yanmar engineers hope to use AI to automatically generate initial calibration values for the ECUs, bypassing the manual adjustments.
MATLAB and Simulink were instrumental for this project, and the team plans to continue to use them and other related toolboxes for future development. “We wouldn’t be this far without MATLAB or Simulink,” says Muinos. “They’re some of the best tools I've used for controls development.”
Partner with Us for Success
Get expert guidance from MathWorks Consulting on tailored solutions to help you achieve your goals.
Read Other Stories
AUTOMATED DRIVING / ROBOTICS
Applying Proven Robotics Techniques to Automobile Development Processes
Renault Advances ADAS Prototyping with ROS Toolbox and Simulink Integration
AI / CONTROL SYSTEMS
Designing Smarter Electrical Equipment Embedded with AI
Combining Model-Based Design with AI in Electrical Systems Improves Industry Operations
ENGINEERED SYSTEMS
Dyson Accelerates New Product Development with System-Level Simulation
Model-Based Design Used to Develop a First-Generation Product
