Msr-tools
Jenifer Griffard
1) Which software guide should I read to understand which Xeon Phi MSR does what?
2) Any sample code I can refer?
3) I have used turbostat and looking to read all the MSR read by this tool. The source code of this tool is a bit large, so any pointers will help.
On one of my Haswell systems, the decimal output is similar to yours, and is not very enlightening. The hex output clearly shows more structure. In the last example I pipe the 64-bit hex output through a little tool that I wrote to print out the bit positions and bits.
The last case is a number, not a bit field, so using the decimal output for these temperature values might make interpretation easier. (Note that I am using the "-u" descriptor because these bit fields are interpreted as unsigned integers.)
MSRs are documented in the Intel Software Developer Manuals, specifically in volume 4: Model-specific registers (Xeon Phi 72xx documentation starts on PDF page 298 of the linked version of the document)..
Do you think, I can get the power, temperature etc data using RAPL sysfs directly (I did investigate the system I have, but still need to validated the data I am getting). If not, then reading MSR is good approach?
The interface to the RAPL functionality uses MSRs. If you look at the kernel driver for the sysfs interface, you will see that it eventually resolves its way to an MSR read. The same is true for any sysfs function that interfaces with hardware functionality that is implemented via MSRs. (Most hardware configuration is done through MSRs. Some is done through PCI configuration space, but this mostly applies to the "uncore" devices.)
Some aspects of the documentation can be frustrating. Many sections of the documentation (particularly in Volume 3 of the Intel Architectures Software Developers Manual) refer to MSRs by name only, and you have to look up these names (in Volume 4 of the Intel Architectures Software Developers Manual) to obtain the MSR numbers. It can be tricky to search PDFs for the longer MSR names, since PDF searches don't match on strings that include line breaks (e.g., caused by narrow columns in tables). The names of the MSRs are not always consistent -- some MSRs are referred to with the "MSR_" prefix in some cases and with the "IA32_" prefix in other places. Sometimes a specific MSR number will have different names on different processors -- even when the functionality appears to be identical.
The msr-tools "rdmsr" and "wrmsr" programs are definitely the most reasonable way to access the MSRs for occasional use. (The alternative is writing loadable kernel modules, which has a much steeper learning curve.)
What I don't know (neither the document specifies), is the unit of this output, and whether any post processing is required? I guess, things will get complicated when I move with others sensors related to package power etc.
Appreciate your detailed answer (your answers are always helpful!).
Quick question: If I do diff to get absolute temperature, then how can that be considered accurate? Running this "rdmsr -p 0 -u -f23:16 0x1a2" and then this "rdmsr -p 0 -u -f22:16 0x19c", and then taking diff will give me the readinds, but it won't be real time? Am I correct?
The "Thermal Activation Temperature" in MSR_TEMPERATURE_TARGET (0x1A2) should never change for a particular processor. It is different across processor models, but for each model it looks like it is programmed at the factory to a fixed value.
There are several places that you can get temperature data. IA32_THERM_STATUS (0x19C) has a scope of "core" on most Intel processors, but it has a scope of "module" on Xeon Phi x200. "Module" is a potentially confusing label to use here, but at the beginning of Section 2.17 of Volume 4 of the SWDM the text explains that "module" is the same as "tile" (i.e., a processor pair) for the Xeon Phi x200. You can also get the temperature using IA32_PACKAGE_THERM_STATUS (0x1B1), which provides a single temperature for the entire chip. It looks like the value provided by IA32_PACKAGE_THERM_STATUS is the maximum of the values of the various sensors on the package. Not all of the sensors are in cores, so it is possible to get "package" temperatures that are higher than any of the core temperatures.
I find it easier to use the MSR interface than sysfs interfaces, since the Linux kernel developers always seem to want to "simplify" or abstract these interfaces. It is painfully difficult to work through the kernel source code to find out what a sysfs device is actually doing at the lowest level -- which is the only level that is documented. Using the MSR interface directly allows me to skip a lot of irritating Linux kernel detective work....
There was an attempt to build an "MSR-safe" kernel extension that would allow user access to "safe" MSRs -- e.g., programming the performance counters and reading most of the MSRs -- but I think that project was dropped due to a lack of ongoing funding.
My biggest gripe about the current MSR interface is that it requires crossing into the kernel for every MSR read and requires an interprocessor interrupt for every MSR read on any core that is not the currently executing core. This makes reading lots of MSRs very expensive. One of my performance monitoring codes takes about 2 milliseconds to read a subset of the available counters on all cores of a Xeon Phi x200, and most of the time is spent in cross-processor MSR reads. The /dev/cpu/*/msr interface does allow a "length" parameter that can be bigger than 8 Bytes, but instead of reading multiple MSRs, it simply reads the same MSR multiple times and returns only the final result. I am guessing that Intel's "sep" driver is able to batch the performance counter reads into a smaller number of kernel crossings, because I have seen VTune results with 2 millisecond sampling granularity -- definitely not practical with the 2 millisecond overhead of my code.
It would not be too hard to write a kernel module that would dump all the performance counters in a single call. I don't know how hard it would be to "batch" the transactions so that one InterProcessor Interrupt would read all the target MSRs on a core, but that does not seem too scary. The hard part (based on my experience) is figuring out how to write all the testing code to make sure that the kernel copy back to user space does not accidentally use kernel privileges to overwrite memory that it should not overwrite. In a previous project, a kernel device driver would happily kill the system if the user passed the kernel a bad pointer. In that case no one else had access to the device driver, so the security flaw was not a problem, but it feels like it would take a fair amount of study to figure out how to make this sort of interface safe to deploy in an environment with thousands of users.
Intel does not verify all solutions, including but not limited to any file transfers that may appear in this community. Accordingly, Intel disclaims all express and implied warranties, including without limitation, the implied warranties of merchantability, fitness for a particular purpose, and non-infringement, as well as any warranty arising from course of performance, course of dealing, or usage in trade.
"There are a series of MSRs that tell you if a given CPU supports the '1' setting of unrestricted guest and a number of other virtualization features. The specific MSR is IA32_VMX_PROCBASED_CTLS2 and I believe unrestricted guest is bit 7 in the secondary proc controls VMCS field."
i don't know if powershell or wmic will show it. I know win7 32bit has debug(which might possible show it? know debug can show some registers, I have once used debug to move a value into the AX register), though i'm on win7 64bit and that doesn't have debug.
I am assuming that there is some tool available in Windows to read and write MSRs as there is one for Linux msr-tools. The tool gives you two command rdmsr and wrmsr. One can read the value of MSRs using rdmsr command. Here is link to one such tool for Window.
A model-specific register (MSR) is any of various control registers in the x86 system architecture used for debugging, program execution tracing, computer performance monitoring, and toggling certain CPU features.
With the introduction of the Pentium processor, Intel provided a pair of instructions (RDMSR and WRMSR) to access current and future "model-specific registers", as well as the CPUID instruction to determine which features are present on a particular model. Many of these registers have proven useful enough to be retained. Intel has classified these as architectural model-specific registers and has committed to their inclusion in future product lines.[5]
Reading and writing to these registers is handled by the rdmsr and wrmsr instructions, respectively. As these are privileged instructions, they can be executed only by the operating system. Use of the Linux msr kernel module creates a pseudo file "/dev/cpu/x/msr" (with a unique x for each processor or processor core). A user with permissions to read and/or write to this file can use the file I/O API to access these registers. The msr-tools[6] package provides a reference implementation.
Documentation regarding which MSRs a certain processor implementation supports is usually found in the processor documentation of the CPU vendor. Examples for rather well-known MSRs are the memory type range registers (MTRRs) and the address-range registers (ARRs).
ff7609af8f