View source for Howto:Troubleshooting Memory Issues

{{Stub}}

{{Troubleshooting}}
== Introduction ==
{{Main article|Resource Tracking for FlightGear}}

Hopefully you already have a feel for how much memory your system uses when running Flightgear. Maybe you run close to full, maybe you have lots of free memory, or maybe you are somewhere in between? You can use the tools provided by your operating system to get a feel for that, but remember:

* Flightgear is just one program on your system and other programs contribute to overall memory usage
* Changes to settings can have a significant impact on how much memory Flightgear uses
* Weather conditions and terrain will affect memory consumption as you fly
* Different versions of Flightgear place different demands on your system memory

[[File:Gnome System Monitor.png|thumb|Screenshot of Gnome System Monitor]]

In an ideal world, we would all have systems with enough memory to run Flightgear with all the eye candy turned on, fly the most complex aircraft models and have detailed scenery stretching out for a hundred miles and more. But it's not an ideal world, and we have to make fit Flightgear into our systems. Most of us should expect to compromise with some settings to run Flightgear smoothly in the memory we have, and to adjust the level of compromise as we run new versions with more features. More features usually consume more memory.

{{cquote|The memory consumption is about 0.7 GB + c1(Aircraft) + c2(Terrain) * visibility^2

where c1(Aircaft) mesaures how heavy your aircraft is in memory and c2(Terrain) measures what vertex density the terrain has. Dependent on what the visibility is and what terrain you're running and what LOD settings you're using for trees, buildings and objects, the total consumption can be below 3 GB even for a heavy airplane (30 km visibility in 1.0 World Scenery) or equally well exceed 12 GB (160 km visibility in CORINE-based Europe).

There is no 'normal' - it depends almost exclusively on your settings.<ref>{{cite web |url=http://forum.flightgear.org/viewtopic.php?f=68&t=25325#p231432
|title= Re: RC 3.4
|author=Thorsten |date= 2015-02-06 06:08}}</ref>|Thorsten}}

{{cquote|I have to recalibrate my settings with every version since 3.0. Am on 3.2 now. The setting that influences memory absolutely most for me is LOD Bare. So I take a resource heavy plane like 777 and fly over the alps at my personal max cruising alt fl 400. All the time i watch total os memory consumption that it stays below 90%. I use total os memory because i also run Firefox for mpmap. That should be included in the total. This calibrating may take several overflights over the alps until i get it right. Too much memory consumption is obviously wrong. Too little gives me an unnecessary short LOD Bare. After such calibration i am fine anywhere with the new version.<ref>{{cite web |url=http://forum.flightgear.org/viewtopic.php?f=68&t=25325&start=15#p231569 
|title= Re: RC 3.4 memory consumption
|author=KL-666 |date= 2015-02-07 16:38}}</ref>|KL-666}}

Having to adjust your settings to fit Flightgear to your system is not a memory issue. That's just the way it is. You should make use of the tools provided by your operating system to understand how much memory you are using and make adjustments to scale down the memory used by Flightgear. Or, if you have lots of spare memory, see what you can enable. Ideally make one change at a time and note the results.

The rest of this article is about troubleshooting memory growth while Flightgear is running. If your memory usage is basically flat during a long flight, perhaps with some small variations due to weather conditions and terrain, you are not seeing a memory issue that indicates a problem with Flightgear itself. On the other hand, if you see a continuous growth in memory usage, either smooth or stepped it's something that needs to be investigated. Especially, if you are not using any heavy features (think aircraft/scenery) or even see memory occupancy growing massively despite sitting idle on the runway or circling a few hundred feet above the ground/sea.

== Techniques ==

=== Flightgear Memory Usage ===

Operating system tools may give some indication of memory growth while Flightgear is running, but to create a useful bug report or to analyze the problem more closely, you need to track the memory usage of Flightgear in isolation.

==== Linux and Mac OS ====

<!-- this can be simplified by using pidof fgfs 

To see swap space used by fgfs :
# grep --color VmSwap /proc/$(pgrep fgfs)/status
-->

On Linux and Mac OS, you can use the <code>ps</code> command in a simple timed loop:
<syntaxhighlight lang="bash">
while true; do ps -o vsz $(pgrep fgfs) | tail -1; sleep 60; done
</syntaxhighlight>

This will print the current size (virtual memory size) of the Flightgear process in kilobytes, and will do so every 60 seconds. Press <tt>Ctrl-C</tt> to quit. You can adjust the sleep time to create more or less output during a flight.

If it makes more sense to send the output to a file, you can do so, but change the filename each time to keep results separate.
<syntaxhighlight lang="bash">
while true; do ps -o vsz $(pgrep fgfs) | tail -1 > myfile.txt ; sleep 60; done
</syntaxhighlight>
For visualization of the results, it is easy to paste this raw data into a spreadsheet for charting.

==== Windows ====
{{Note|This should be written by someone on Windows, familiar with [https://technet.microsoft.com/en-us/sysinternals/bb795533.aspx Sysinternals tools]}}

* [https://msdn.microsoft.com/en-us/library/windows/desktop/dd744766(v=vs.85).aspx background]
* task manager
* sysinternals process explorer
* sysinternals [http://blogs.technet.com/b/askperf/archive/2010/08/13/introduction-to-the-new-sysinternals-tool-rammap.aspx RAMMap9
* [http://www.drmemory.org/ DrMemory]

{{WIP}}

=== Creating a Test Case ===

As discussed in the [[#Introduction|Introduction]], the memory consumed by Flightgear is hugely dependent on your settings and the aircraft you use. It also changes throughout a flight due to factors such as weather conditions and terrain. For useful analysis of memory growth while Flightgear is running, you need to capture data using a consistent test case, where the variables contributing to memory usage are minimized. Essentially what you need is a repeatable run of Flightgear that shows a memory issue and the same run of Flightgear where you change one thing and the memory issue disappears. It also helps to exclude as many potential factors (subsystems and features) as possible - for instance, by trying different locations, different aircraft, different FDMs to see if changing any of these has any noticeable impact on memory occupancy. That isolates the problem and creates the basis of a good bug report.

It is also generally helpful to check if the bug seems affected by/related to changing simulation rate and/or pausing/unpausing the simulation, because tha twill help us exclude/include certain subsysttems, but also commits when running git bisect

If you can, please try to disable as much as possible, while using the least complex combination of aircraft/location which still exhibits the problem - that will greatly help us narrow down the issue, while also making sure that there's plenty of resources left for running  a debugger/profiler or a leak-checking tool like valgrind/address sanitizer to understand what is going on

For the same reason it is helpful to see if disabling rendering of certain scene elements is affecting the bug or not ([[Draw masks]]), as well as tinkering with different threading modes.

For example, if you believe that Advanced Weather causes memory growth, you to be able to consistently repeat a flight without Advanced Weather and show, using the techniques listed in [[#Flightgear Memory Usage|Flightgear Memory Usage]] above, that memory usage is flat. Then you need to repeat that flight, with the only difference being that your turn Advanced Weather on, and show that memory usage grows.

It may be that you have already created an informal test case that does the job. For better bug reporting, consider making a more stringent test case using:

* [[Troubleshooting_crashes#Minimal_startup_profile|Minimal Startup Profile]] 
* disabling parts of the scenegraph using draw-masks: <nowiki>
--prop:/sim/rendering/draw-mask/terrain=0
--prop:/sim/rendering/draw-mask/aircraft=0
--prop:/sim/rendering/draw-mask/models=0
--prop:/sim/rendering/draw-mask/clouds=0</nowiki>
* [[Route manager]] routes/flight plans (keep the file and submit with your bug report)
* [[Instant_Replay#FG_2.6.0_and_later|Flight recorder tapes]]
* [[Nasal]] scripts for creating scripted flights

What you do depends on the problem you are trying to report or solve. It may be that a simple solution like flying over the ocean with manual weather input set to an empty METAR string does the job. The bottom line is that you need a test that you can repeat without the problem, change one thing and show that the problem occurs.

{{Startup Profile|name=minimal|version=3.2}}

=== Flightgear Version ===

Because Flightgear development is concentrated on the latest version, and bugs in old versions are not usually fixed, you need to run the latest development version to make a useful bug report. In addition, it is important that you use the HELP/ABOUT dialog to post the exact version of FlightGear, including its dependencies. This dialog provides two buttons for taking a screen shot and for copying relevant data to the clipboard respectively (add screen shot).

{{WIP}}

=== Causes of Memory Growth ===

If you have discovered a memory issue, where memory grows as a result of a particular feature of Flightgear, and you can reproduce that problem with a consistent test case, there could be two reasons why it is happening:

* Memory Leak
* Memory Management Problems

The difference between them is subtle, but understanding the nature of the memory issue is important to help get the problem resolved.

A memory leak happens when Flightgear allocates some memory and then loses track of it. It's a bit like buying a car and then losing the key. So you go and buy another. Eventually your street fills up with cars that you own but can't drive. This might sound stupid, but in some programming languages it's easily done and not uncommon.

Memory management problems are not quite the same. You go out and buy a car, you don't lose the key, but still go out and buy another before selling the first one. You have access to all the cars but the end result is the same as a memory leak: your street is full of cars.

==== Identifying Memory Leaks ====

Memory leaks can be identified by rebuilding Flightgear using leak checking.
{{WIP}}

==== Identifiying Memory Management Problems ====

There is no specific tool to identify these, other than you have a repeatable test case where memory usage grows and grows but the leak checker doesn't indicate a problem. This can still be reported in the bug, but it's useful to say that you checked for memory leaks and didn't find any, so always check for memory leaks when you see a memory issue and indicate on your bug report that the leak checker came up with nothing.

=== Bisecting ===
{{WIP}}

== Case Study ==
{{WIP}}

[[Category:Troubleshooting]]