[dpdk-dev] [PATCH v1 3/4] examples/power: allow vms to use lcores over 63

[Hunt, David]

Hi Anatoly,

On 10/12/2018 1:06 PM, Burakov, Anatoly wrote:
> On 22-Nov-18 5:02 PM, David Hunt wrote:
>> Extending the functionality to allow vms to power manage cores beyond 
>> 63.
>>
>> Signed-off-by: David Hunt <david.hunt at intel.com>
>> ---
>>   examples/vm_power_manager/channel_manager.c | 59 ++++++++-------------
>>   examples/vm_power_manager/channel_manager.h | 30 ++---------
>>   examples/vm_power_manager/channel_monitor.c | 56 +++++++------------
>>   examples/vm_power_manager/vm_power_cli.c    |  4 +-
>>   4 files changed, 48 insertions(+), 101 deletions(-)
>>
>> diff --git a/examples/vm_power_manager/channel_manager.c 
>> b/examples/vm_power_manager/channel_manager.c
>> index 5af4996db..3d493c179 100644
>> --- a/examples/vm_power_manager/channel_manager.c
>> +++ b/examples/vm_power_manager/channel_manager.c
>> @@ -49,7 +49,7 @@ static bool global_hypervisor_available;
>>    */
>>   struct virtual_machine_info {
>>       char name[CHANNEL_MGR_MAX_NAME_LEN];
>> -    rte_atomic64_t pcpu_mask[CHANNEL_CMDS_MAX_CPUS];
>> +    uint16_t pcpu_map[CHANNEL_CMDS_MAX_CPUS];
>>       struct channel_info *channels[CHANNEL_CMDS_MAX_VM_CHANNELS];
>>       char channel_mask[POWER_MGR_MAX_CPUS];
>>       uint8_t num_channels;
>> @@ -79,7 +79,7 @@ update_pcpus_mask(struct virtual_machine_info 
>> *vm_info)
>>       virVcpuInfoPtr cpuinfo;
>>       unsigned i, j;
>>       int n_vcpus;
>> -    uint64_t mask;
>> +    uint16_t pcpu;
>>         memset(global_cpumaps, 0, CHANNEL_CMDS_MAX_CPUS*global_maplen);
>>   @@ -120,26 +120,23 @@ update_pcpus_mask(struct virtual_machine_info 
>> *vm_info)
>>           vm_info->info.nrVirtCpu = n_vcpus;
>>       }
>>       for (i = 0; i < vm_info->info.nrVirtCpu; i++) {
>> -        mask = 0;
>> +        pcpu = 0;
>>           for (j = 0; j < global_n_host_cpus; j++) {
>>               if (VIR_CPU_USABLE(global_cpumaps, global_maplen, i, j) 
>> > 0) {
>> -                mask |= 1ULL << j;
>> +                pcpu = j;
>
> Not sure what this does.
>
> Initial code goes through all CPU's, marks all that are usable into 
> mask (i.e. code implies there can be several), and then sets up this 
> mask.
>
> Now, you're going through all CPU's, store *one* arbitrary CPU index, 
> and set the map up with that single index.
>
> That doesn't look like an equivalent replacement.
>

Good catch. mask was changed to an array, so I need to set element in 
the loop, not outside it. Fixed in next rev.


>>               }
>>           }
>> -        rte_atomic64_set(&vm_info->pcpu_mask[i], mask);
>> +        vm_info->pcpu_map[i] = pcpu;
>>       }
>>       return 0;
>>   }
>>     int
>> -set_pcpus_mask(char *vm_name, unsigned int vcpu, char *core_mask)
>> +set_pcpu(char *vm_name, unsigned int vcpu, unsigned int pcpu)
>
> See patch 1 comments - it would've made this change easier to parse if 
> set_pcpu was removed there, instead of here.
>
I couldn't change it back there, as the calling functions had not been 
updated. Now that they are gone, I can remove it in this patch.


>>   {
>>       unsigned i = 0;
>>       int flags = VIR_DOMAIN_AFFECT_LIVE|VIR_DOMAIN_AFFECT_CONFIG;
>>       struct virtual_machine_info *vm_info;
>> -    char mask[POWER_MGR_MAX_CPUS];
>> -
>> -    memcpy(mask, core_mask, POWER_MGR_MAX_CPUS);
>>         if (vcpu >= CHANNEL_CMDS_MAX_CPUS) {
>>           RTE_LOG(ERR, CHANNEL_MANAGER, "vCPU(%u) exceeds max 
>> allowable(%d)\n",
>> @@ -166,17 +163,16 @@ set_pcpus_mask(char *vm_name, unsigned int 
>> vcpu, char *core_mask)
>>           return -1;
>>       }
>>       memset(global_cpumaps, 0 , CHANNEL_CMDS_MAX_CPUS * global_maplen);
>> -    for (i = 0; i < POWER_MGR_MAX_CPUS; i++) {
>> -        if (mask[i] == 1) {
>> -            VIR_USE_CPU(global_cpumaps, i);
>> -            if (i >= global_n_host_cpus) {
>> -                RTE_LOG(ERR, CHANNEL_MANAGER, "CPU(%u) exceeds the 
>> available "
>> -                        "number of CPUs(%u)\n",
>> -                        i, global_n_host_cpus);
>> -                return -1;
>> -            }
>> -        }
>> +
>> +    VIR_USE_CPU(global_cpumaps, i);
>> +
>> +    if (pcpu >= global_n_host_cpus) {
>> +        RTE_LOG(ERR, CHANNEL_MANAGER, "CPU(%u) exceeds the available "
>> +                "number of CPUs(%u)\n",
>> +                i, global_n_host_cpus);
>> +        return -1;
>>       }
>
> Some comments on what the above code does would have been nice. Why 
> the check was removed?


function changed from set_pcpus_mask to the name of what used to be the 
wrapper function, and in doing so we changed the mask parameter to a 
pcpu (int) parameter, so there's not need for a loop any more. However, 
there was an error where I was calling VIR_USE_CPU with the incorrect 
value, so that's now fixed.  The check was removed because we're no 
longer checking for '1's in an array, we are being passed the actual 
pcpu to use.

Also, there were an addition of a spinlock around reading the pcpu 
config data.

>
>> +
>