Zerina Kapetanovic
How do we produce 70% more food by the year 2050 to feed the world’s ever-growing population?
Zerina Kapetanovic, a 5th-year graduate research assistant in the University of Washington Department of Electrical & Computer Engineering (UW ECE), is determined to find solutions to this and other critical environmental dilemmas. Her thesis work on “Low-Power Communication for Environmental Sensing Systems” recently earned Kapetanovic a Microsoft Research Dissertation Grant (MRDG) to help support her ongoing doctoral research on this topic.
Advised by Joshua Smith, the Milton and Delia Zeutschel Professor in Entrepreneurial Excellence at UW ECE and a professor at the Allen School of Computer Science & Engineering, Kapetanovic has been instrumental in the development of cutting-edge wireless and backscatter techniques, battery-free sensing modules, and end-to-end connected Internet of Things (IoT) hubs for such forms of low-power communication. Working together, these technologies and data-driven methods allow for far greater precision and more sustainable solutions to agricultural and environmental issues than previously available. As farmers around the world tackle the looming threat of food shortages resulting from expanding global populations, reduction in arable land and numerous other problems related to climate change, these types of solutions are more imperative and meaningful than ever.
“I am deeply passionate about using technology to solve environmental issues, and my goal is to develop new techniques and solutions that make the Internet of Things (IoT) more practical for resource-constrained environments,” explains Kapetanovic. “In particular, my focus is on low-power communication and sensing.”
The 2020-21 Microsoft Research Dissertation Grant recipients
Kapetanovic is also a member of Professor Smith’s Sensor Systems Lab, which centers on the development of similar technologies for sensing, robotics, wireless power and medical devices. “Zerina is an incredibly productive graduate student,” says Smith. “In addition to her strong research program at the UW, she has also been a key member of another major research program at Microsoft. I’m not sure how she does so much!”
Now in its fourth year, the Microsoft Research Dissertation Grant awards up to $25,000 for the 2020–21 academic year to support Ph.D. students who are underrepresented in the field of computing and are currently completing their doctoral thesis work at North American universities. The grant hopes to provide research funding opportunities to students who self-identify as a woman, African American, Black, Hispanic, Latinx, American Indian, Alaska Native, Native Hawaiian, Pacific Islander and/or person with a disability, increasing the pathway of diverse and talented individuals pursuing advanced degrees in computing-related fields.
Awardees are also invited to partake in the Microsoft PhD Summit, a two-day workshop that will be hosted virtually this autumn by Microsoft Research. Fellows have the opportunity to meet with both Microsoft researchers and other top students to share their research.
“I am very honored and thrilled to receive the Microsoft Research Dissertation Grant,” says Kapetanovic. “It will help me tremendously with completing my research at the University of Washington, giving me an uninterrupted block of time to extend my core research, write my dissertation and help cover the cost of hardware development for my research. I am especially grateful to my advisor, Josh Smith, and Ranveer Chandra, my mentor at Microsoft, both of whom have guided and supported me throughout my PhD career.”
The grant is awarded through a highly competitive process. Of the 230 applications submitted this year, the highest number yet for the program, only 10 students were selected to receive the award.
Just beat it
No stranger to Microsoft and its research programs, Kapetanovic has worked on Microsoft’s FarmBeats team as a research intern on three separate occasions–since its early inception in 2015, and again in 2016 and 2019. Providing farmers with long-term, automated and sustainable methods of agriculture through ultra-low-power, networked communications devices at a large scale has been Kapetanovic’s primary area of focus. By integrating her expertise with several new technologies developed at FarmBeats, Kapetanovic helped create and deploy various sensor and communications systems that enable farmers to accurately monitor soil moisture, temperature, humidity levels, address fertilizer and pesticide overuse and, ultimately, increase their crop yields at reduced costs.
Zerina Kapetanovic checks the FarmBeats ground sensors at Dancing Crow Farm in Carnation, WA. The sensor box features 8 different sensors that collect real-time data on variables that might affect the growth of crops, including wind speed and direction, soil and atmospheric moisture and temperature, barometric pressure, CO2 levels, and rain and light exposure. (Photo by Michael Victor)
“While my thesis focuses on enabling low-power communication for environmental sensing systems,” says Kapetanovic, “it especially deals with two major challenges: resources constraints and scale. For example, devices deployed in remote locations often lack power and internet connectivity. This becomes even more challenging when considering the sizes of forests, farms and oceans, which all require large-scale sensing systems.”
So far, the farmers who have partnered with FarmBeats in their 6-month pilot programs have reaped huge benefits from the system’s added precision and insights, reducing their use of crop spraying by up to 90%, while saving tens of thousands of dollars in the process.
3 key innovations: connectivity, low power and scale
The FarmBeats system uses a multi-pronged approach to collect and analyze data from several sources. Since farms and forests can often span thousands of acres, setting up a network of commercially available in-ground sensors to provide sufficiently comprehensive data-gathering coverage at a large scale is prohibitively expensive for many farms. In response to this dilemma, Farmbeats uses a far more spread-out network of different sensors (hundreds, rather than tens, of meters of space between them), as well as a small fleet of flying drones with downward-facing smartphones attached in order to survey the farmland below. Combined with the in-ground sensor data, the optical and infrared (IR) image data captured by the smartphone cameras is later compiled and analyzed to create precision maps of soil moisture, temperature, pH and humidity levels. “Micro-climate” predictions of wind direction, wind speed, ambient temperature and humidity are also provided. These resulting data-driven agricultural predictions give farmers more accurate insights than ever before into what, when and where to plant or spray, leading to far greater sustainability overall.
FarmBeats Precision Maps: (a) A 40MPixel orthomosaic created from a 3-min. flight over 2-acre area of farmland. The system infers dense sensor measurements from very few sensors deployed on the farm (white circles). (b) The predicted soil moisture map (on a scale of 1-5). The top left region in the image was correctly predicted to have high moisture despite no moisture sensors present on that section of farmland. (c) The predicted pH map. The system identified the entire field as slightly acidic, but the bottom left/center as more acidic than the rest. (d) The predicted soil temperature map (in Fahrenheit).
Because many farms are too rural to directly access either the Internet or a reliable power source, the data that’s collected is often too limited to be of any use or is unable to reach the farmer’s local computer entirely. To address this constraint, the FarmBeats system relies on an ingeniously clever, yet relatively simple, solution to relay this information to farmers — harnessing the availability of unused television broadcast signals that are being continuously transmitted through the air. A solar-powered TV White Spaces (TVWS) base station takes advantage of the under-leveraged frequencies between television signals in the lower part of the MHz spectrum. What would typically be seen on older, analog TVs simply as static “fuzz” is repurposed by the TVWS base station to send information at broadband speeds using off-the-shelf Wi-Fi routers and antennae, thereby connecting and extending internet connectivity to rural areas.
Kapetanovic played a central role in developing a sensor module prototype to interface with the network of sensors and cameras, whose batteries are also solar-powered, as well as a smart IoT hub that connects this information with other drone data. Power from the hub ensures that the entire system operates uninterrupted, even during harsh weather conditions. Since lower frequency waves are able to travel much farther than higher ones, farms that are spread out over long distances, covered by dense forests, or are in different locations from the farmers’ homes, are able to be economically and effectively connected to both internet and power sources using these technologies.
Once the farm’s data has been collected and received by the IoT hub, the TVWS base station transmits it to the farmer’s local computer, which is used to compress the information before uploading it to Microsoft’s cloud-based platform. Shrinking the data from gigabytes (Gb) down to mere kilobytes (kb) before it’s uploaded makes the entire process of consolidating, analyzing and storing the information faster and much more efficient overall. In the cloud, all of the necessary data aggregation, visualization and analytics is done through Microsoft’s machine learning and artificial intelligence (AI) and vision algorithms. The subsequent maps and recommendations on how and when to best optimize resources to increase production are then made available to the farmers to download onto their smartphone devices from the cloud when needed.
The FarmBeats system uses sensor, drone, TV Whites Space, Edge and cloud solutions to provide insights to farmers and enable data-driven agricultural techniques.
Farms that are unable to afford the high initial costs of flying drones to monitor their farmland, such as those in developing countries for instance, can still utilize an alternative solution developed by the FarmBeats team. With this method, the smartphone is attached to the bottom of a large helium-filled balloon rather than a remote-controlled drone. An operator then manually walks the balloon along the farmland while the smartphone’s camera surveys the ground beneath it to collect the necessary data and imagery.
Although she does not have a background in the field of agriculture per se, Kapetanovic says she became interested in her assigned role at FarmBeats after quickly recognizing that her combined technical expertise and environmental interests aligned perfectly with FarmBeats’ mission and needs. Kapetanovic’s motivation and commitment to this groundbreaking (quite literally at times) research is readily apparent. In her dissertation, she explains, “Over the last decade the Internet of Things (IoT) has been changing the world, from enabling connected electronics, to smart homes and smart agriculture. I’m extremely excited about the role that IoT plays when it comes to environmental monitoring, which is now increasingly relevant in the times of the climate change crisis and the need to achieve biodiversity conservation.”
The FarmBeats system caught the attention of renowned environmentalist, global public health philanthropist and Microsoft co-founder Bill Gates. Gates wrote an article on his personal blog describing his impressions of the FarmBeats project. He is seen here (center) standing next to Zerina (bottom left) while looking up at the smartphone camera tethered to a FarmBeats balloon. Shown at right, a FarmBeats balloon being tested at one of the partnering farms.
One challenge Kapetanovic and others are working on overcoming with these systems is when they are deployed amidst dense forest canopies or on overcast days. Under such conditions, reduced amounts of sunlight could potentially cause the solar-powered IoT hub and its connected routers, sensors and antennae to lose power. To help mitigate this issue and ensure connectivity is maintained throughout a farming season, the entire FarmBeats system has been designed to be weather-aware, with an on/off switch dictated by daily cloud coverage to better conserve power when necessary.
The FarmBeats system has recently been developed into a commercially available service, Azure FarmBeats (part of Azure Marketplace). Microsoft’s Airband team has been busy deploying it on farms in several states around the USA. Additionally, with millions of Americans still lacking access to commercially available high-speed internet service (including nearly one-quarter of people living in rural areas), FarmBeats’ TVWS communication technology is also being used to connect those communities, businesses and individual consumers who live and work in remote areas of the country. The Airband team also currently has deployments on farms in India and various countries in Europe and Africa, with upcoming plans to expand their efforts into China.
The techniques Kapetanovic has helped develop are already being applied to other, non-agricultural, environmental scenarios where collecting camera and sensor data is tremendously useful, such as monitoring air pollution in urban areas, monitoring animals’ migratory patterns, and alerting authorities to illegal animal poaching or deforestation.
“My past and ongoing research focuses on addressing these challenges by developing systems like FarmBeats to enable data-driven agriculture, enabling low-power communication using backscatter techniques, and using machine learning to improve the performance of mainstream IoT solutions,” says Kapetanovic.
A TV whitespace base station and FarmBeats sensor box deployed on Nelson Farm in Spokane, WA.
Glazed over
The “data hiding” techniques behind Kapetanovic and her team’s latest low-power IoT communication research project, Glaze, sound like something straight out of a John le Carré spy novel. The technologies behind them, however, are very much grounded in reality.
The Glaze module and receiver prototype hardware: (left) the Glaze Transmitter module prototype using off-the-shelf components; (right) the Glaze Receiver prototype’s printed circuit board.
Combining a transmitter module and receiver/decoder module, the Glaze system introduces tiny perturbations to existing TV, FM radio frequency (RF) or data signals (Wi-Fi, Bluetooth), allowing it to effectively overlay additional data on top of these transmissions (“glazing them over” with new information, so to speak) without distorting or degrading them in any noticeable way. Whereas the TVWS method utilizes unused television frequencies, Glaze is able to take advantage of the already-occupied range of signals being transmitted. This technique, when coupled with other backscatter solutions, can effectively create an end-to-end system of low-power communication.
Glaze can encode data not only by introducing small amounts of attenuation to a pre-existing TV or RF signal, but also by detecting the presence or absence of Wi-Fi data packets being sent over a network. Because the consumer communications devices being used have inherent built-in tolerances to certain levels of noise, Glaze’s tiny alterations to the signals do not produce any perceptible change to the integrity of the originally intended signal (a song or podcast, for example) when it is received by a device like an FM radio. Glaze’s receiver, however, is sensitive enough to accurately decode and interpret the packets of data transmitted by the Glaze module.
Similar data hiding techniques have been demonstrated previously by other researchers using the visible spectrum of television broadcast signals. With this method, individual pixels of a TV image are brightened or dimmed at levels imperceptible to the human eye. These minuscule deviations from the original source image can still be detected and interpreted as extra, useful information by an appropriate receiver/decoder, however.
Such glazing techniques can easily take advantage of these existing broadcast signals since they are constantly being transmitted all around us. With the ability to travel tens of miles, Glaze’s use of these types of low frequencies has many potential real-world applications. As mentioned, lower frequencies tend to travel farther than higher ones, while also penetrating obstacles like walls much more easily, as anyone who’s ever experienced the low, chest-thumping bass frequencies invading their living rooms during a loud car stereo drive-by is well aware. In urban settings, therefore, this system could one day be used to interact with and provide updates to existing IoT smart devices in people’s homes or offices, as one example.
Glaze System. The Glaze module connects to the antenna port of a preexisting wireless transmitter to overlay data. The transmitted signal is then received by a Glaze receiver and legacy receivers.
The Glaze system provides several useful advantages over other existing or proposed backscatter or data hiding solutions in terms of cost, maintenance and scale. Glaze does not require any sort of modification to the existing transmission hardware or wireless access points (AP). Instead, it simply adds its supplementary transmitter module to existing antennae ports and employs a Glaze receiver on the other end that can decode a transmission. Not needing to upgrade existing infrastructure makes Glaze an incredibly low-cost and versatile solution to implement, especially over large-scale scenarios like sprawling metropolitan cities or large swaths of farmland or forests.
Additionally, Glaze employs an ultra-low power design. In other systems, the overhead costs associated with single-use battery replacement and routine maintenance can present major obstacles to sustainability. The Glaze system, however, utilizes a low-power receiver with various passive components. Such a design has the potential to work as an entirely battery-free energy model, harvesting the energy it needs to run from ambient light or RF signals alone.
And crucially, Glaze provides a downlink communication solution to complement existing backscatter solutions, which nearly always lack a low-power downlink capability. This functionality allows for a complete end-to-end low-power IoT communications setup for both receiving and sending information, making for a much more sustainable and economical peer-to-peer data gathering and feedback device solution.
Most recently, Kapetanovic has been using machine learning in an attempt to further improve the communication efficiency of low power wide area networks (LPWAN), such as the popular LoRa LPWAN solution.
The novel coronavirus (COVID-19) pandemic has limited the regular access to research labs for UW ECE students over the past several months. Undeterred, Zerina continues to pursue her research projects at her home lab setup. Pictured above, she has been conducting wireless experiments using two horn antennae. (photo by Zerina Kapetanovic)
While being awarded a Microsoft Research Dissertation Grant is Kapetanovic’s most recent recognition of her many accomplishments, she was also among four female UW ECE students selected to speak at the Grace Hopper Celebration for Women in Computing Conference in 2019. This gave Kapetanovic an opportunity to present her FarmBeats research on low-power communication to a much broader audience, while also expanding her professional network.
“It was a really great experience,” says Kapetanovic. “I was able to network with so many women in tech and learn about their work. It was also inspiring to see such a huge group of women all in one place pursuing careers in STEM fields.”
In addition to being featured on Bill Gates’ GatesNotes, FarmBeats has also been featured on the BBC, in The Economist, and was highlighted by Microsoft CEO Satya Nadella as one of the ten projects that most inspired him in 2017.
We congratulate Kapetanovic and look forward to seeing what she does next in her ongoing research efforts to help solve the myriad environmental and communication challenges facing people around the world. Visit Kapetanovic’s personal website for more information on her research projects and follow her on Twitter and Instagram.
Learn more about the Microsoft Research Dissertation Grant and this year’s awardees, along with each of their respective academic institutions and dissertations.
Story by Ryan Hoover | UW ECE News