<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1"> </head> <body> <div>Thanks Andrew. Changes are fine with me</div> <div></div> <hr style="display:inline-block;width:98%" tabindex="-1"> <div id="divRplyFwdMsg" dir="ltr">From: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru> Sent: Thursday, October 6, 2022 2:06 PM To: Ferruh Yigit <ferruh.yigit@amd.com>; Thomas Monjalon <thomas@monjalon.net>; Ray Kinsella <mdr@ashroe.eu> Cc: dev@dpdk.org <dev@dpdk.org>; Jerin Jacob Kollanukkaran <jerinj@marvell.com>; Sunil Kumar Kori <skori@marvell.com> Subject: [EXT] Re: [PATCH v4] ethdev: support congestion management <div> </div> </div> <div class="BodyFragment"> <div class="PlainText">External Email ---------------------------------------------------------------------- @Jerin, could you confirm if my minor changes are OK for you and, if so, I'll proceed with applying the patch. On 10/4/22 12:02, Andrew Rybchenko wrote: > From: Jerin Jacob <jerinj@marvell.com> > > NIC HW controllers often come with congestion management support on > various HW objects such as Rx queue depth or mempool queue depth. > > Also, it can support various modes of operation such as RED > (Random early discard), WRED etc on those HW objects. > > Add a framework to express such modes(enum rte_cman_mode) and > introduce (enum rte_eth_cman_obj) to enumerate the different > objects where the modes can operate on. > > Add RTE_CMAN_RED mode of operation and RTE_ETH_CMAN_OBJ_RX_QUEUE, > RTE_ETH_CMAN_OBJ_RX_QUEUE_MEMPOOL objects. > > Introduce reserved fields in configuration structure > backed by rte_eth_cman_config_init() to add new configuration > parameters without ABI breakage. > > Add rte_eth_cman_info_get() API to get the information such as > supported modes and objects. > > Add rte_eth_cman_config_init(), rte_eth_cman_config_set() APIs > to configure congestion management on those object with associated mode. > > Finally, add rte_eth_cman_config_get() API to retrieve the > applied configuration. > > Signed-off-by: Jerin Jacob <jerinj@marvell.com> > Signed-off-by: Sunil Kumar Kori <skori@marvell.com> > Signed-off-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru> > --- > v3..v4: Andrew Rybchenko > - rebase > - remove eth_check_err() and use eth_err() instead > - minor fixes in description to avoid "This patch" and "Added". > - correct position in release notes > v2..v3: > - Rename rte_cman.c to rte_ethdev_cman.c > - Move lib/eal/include/rte_cman.h to lib/ethdev/rte_cman.h > - Fix review comments (Andrew Rybchenko) > - Add release notes > > v1..v2: > - Fix review comments (Akhil Goyal) > > rfc..v1: > - Added RED specification (https://urldefense.proofpoint.com/v2/url?u=http-3A__www.aciri.org_floyd_papers_red_red.html&d=DwICaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=1DGob4H4rxz6H8uITozGOCa0s5f4wCNtTa4UUKvcsvI&m=kISGqWlRwNHLBHvKCTkVQBoHyBW79qMFQEJNcoDZXb8X4GKLwHVeeE_D3Lh082fx&s=HZKFgpOQ0CtqvK-_Zc-QZn87gBY4UrscDXgFihsXjvQ&e= ) link > - Fixed doxygen comment issue (Min Hu) > > doc/guides/nics/features.rst | 12 ++ > doc/guides/nics/features/default.ini | 1 + > doc/guides/rel_notes/release_22_11.rst | 6 + > lib/ethdev/ethdev_driver.h | 25 ++++ > lib/ethdev/ethdev_private.h | 3 + > lib/ethdev/meson.build | 2 + > lib/ethdev/rte_cman.h | 55 +++++++++ > lib/ethdev/rte_ethdev.c | 2 +- > lib/ethdev/rte_ethdev.h | 164 +++++++++++++++++++++++++ > lib/ethdev/rte_ethdev_cman.c | 101 +++++++++++++++ > lib/ethdev/version.map | 4 + > 11 files changed, 374 insertions(+), 1 deletion(-) > create mode 100644 lib/ethdev/rte_cman.h > create mode 100644 lib/ethdev/rte_ethdev_cman.c > > diff --git a/doc/guides/nics/features.rst b/doc/guides/nics/features.rst > index b4a8e9881c..70ca46e651 100644 > --- a/doc/guides/nics/features.rst > +++ b/doc/guides/nics/features.rst > @@ -727,6 +727,18 @@ Supports configuring per-queue stat counter mapping. > ``rte_eth_dev_set_tx_queue_stats_mapping()``. > > > +.. _nic_features_congestion_management: > + > +Congestion management > +--------------------- > + > +Supports congestion management. > + > +* **[implements] eth_dev_ops**: ``cman_info_get``, ``cman_config_set``, ``cman_config_get``. > +* **[related] API**: ``rte_eth_cman_info_get()``, ``rte_eth_cman_config_init()``, > + ``rte_eth_cman_config_set()``, ``rte_eth_cman_config_get()``. > + > + > .. _nic_features_fw_version: > > FW version > diff --git a/doc/guides/nics/features/default.ini b/doc/guides/nics/features/default.ini > index f7192cb0da..a9c0008ebd 100644 > --- a/doc/guides/nics/features/default.ini > +++ b/doc/guides/nics/features/default.ini > @@ -60,6 +60,7 @@ Tx descriptor status = > Basic stats = > Extended stats = > Stats per queue = > +Congestion management = > FW version = > EEPROM dump = > Module EEPROM dump = > diff --git a/doc/guides/rel_notes/release_22_11.rst b/doc/guides/rel_notes/release_22_11.rst > index 44f9a30c6a..0ffa004a9e 100644 > --- a/doc/guides/rel_notes/release_22_11.rst > +++ b/doc/guides/rel_notes/release_22_11.rst > @@ -78,6 +78,12 @@ New Features > Added new rte_flow action which allows application to re-route packets > directly to the kernel without software involvement. > > +* **Added support for congestion management for ethdev.** > + > + Added new APIs ``rte_eth_cman_config_init()``, ``rte_eth_cman_config_get()``, > + ``rte_eth_cman_config_set()``, ``rte_eth_cman_info_get()`` > + to support congestion management. > + > * **Updated Intel iavf driver.** > > * Added flow subscription support. > diff --git a/lib/ethdev/ethdev_driver.h b/lib/ethdev/ethdev_driver.h > index 8cd8eb8685..e1e2d10a35 100644 > --- a/lib/ethdev/ethdev_driver.h > +++ b/lib/ethdev/ethdev_driver.h > @@ -1094,6 +1094,22 @@ typedef int (*eth_rx_queue_avail_thresh_query_t)(struct rte_eth_dev *dev, > uint16_t *rx_queue_id, > uint8_t *avail_thresh); > > +/** @internal Get congestion management information. */ > +typedef int (*eth_cman_info_get_t)(struct rte_eth_dev *dev, > + struct rte_eth_cman_info *info); > + > +/** @internal Init congestion management structure with default values. */ > +typedef int (*eth_cman_config_init_t)(struct rte_eth_dev *dev, > + struct rte_eth_cman_config *config); > + > +/** @internal Configure congestion management on a port. */ > +typedef int (*eth_cman_config_set_t)(struct rte_eth_dev *dev, > + const struct rte_eth_cman_config *config); > + > +/** @internal Retrieve congestion management configuration of a port. */ > +typedef int (*eth_cman_config_get_t)(struct rte_eth_dev *dev, > + struct rte_eth_cman_config *config); > + > /** > * @internal A structure containing the functions exported by an Ethernet driver. > */ > @@ -1309,6 +1325,15 @@ struct eth_dev_ops { > eth_rx_queue_avail_thresh_set_t rx_queue_avail_thresh_set; > /** Query Rx queue available descriptors threshold event */ > eth_rx_queue_avail_thresh_query_t rx_queue_avail_thresh_query; > + > + /** Get congestion management information */ > + eth_cman_info_get_t cman_info_get; > + /** Initialize congestion management structure with default values */ > + eth_cman_config_init_t cman_config_init; > + /** Configure congestion management */ > + eth_cman_config_set_t cman_config_set; > + /** Retrieve congestion management configuration */ > + eth_cman_config_get_t cman_config_get; > }; > > /** > diff --git a/lib/ethdev/ethdev_private.h b/lib/ethdev/ethdev_private.h > index cc9879907c..acb4b335c8 100644 > --- a/lib/ethdev/ethdev_private.h > +++ b/lib/ethdev/ethdev_private.h > @@ -37,6 +37,9 @@ struct rte_eth_dev_callback { > > extern rte_spinlock_t eth_dev_cb_lock; > > +/* Convert all error to -EIO if device is removed. */ > +int eth_err(uint16_t port_id, int ret); > + > /* > * Convert rte_eth_dev pointer to port ID. > * NULL will be translated to RTE_MAX_ETHPORTS. > diff --git a/lib/ethdev/meson.build b/lib/ethdev/meson.build > index 47bb2625b0..39250b5da1 100644 > --- a/lib/ethdev/meson.build > +++ b/lib/ethdev/meson.build > @@ -8,6 +8,7 @@ sources = files( > 'ethdev_trace_points.c', > 'rte_class_eth.c', > 'rte_ethdev.c', > + 'rte_ethdev_cman.c', > 'rte_flow.c', > 'rte_mtr.c', > 'rte_tm.c', > @@ -19,6 +20,7 @@ sources = files( > ) > > headers = files( > + 'rte_cman.h', > 'rte_ethdev.h', > 'rte_ethdev_trace.h', > 'rte_ethdev_trace_fp.h', > diff --git a/lib/ethdev/rte_cman.h b/lib/ethdev/rte_cman.h > new file mode 100644 > index 0000000000..297db8e095 > --- /dev/null > +++ b/lib/ethdev/rte_cman.h > @@ -0,0 +1,55 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright(C) 2022 Marvell International Ltd. > + */ > + > +#ifndef RTE_CMAN_H > +#define RTE_CMAN_H > + > +#ifdef __cplusplus > +extern "C" { > +#endif > + > +#include <rte_bitops.h> > + > +/** > + * @file > + * Congestion management related parameters for DPDK. > + */ > + > +/** Congestion management modes */ > +enum rte_cman_mode { > + /** > + * Congestion based on Random Early Detection. > + * > + * <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_Random-5Fearly-5Fdetection&d=DwICaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=1DGob4H4rxz6H8uITozGOCa0s5f4wCNtTa4UUKvcsvI&m=kISGqWlRwNHLBHvKCTkVQBoHyBW79qMFQEJNcoDZXb8X4GKLwHVeeE_D3Lh082fx&s=iyRkqzJBjwu2oS-YMRhtxJOpsK8JWJG5gEH_1DC5pwM&e=" data-auth="NotApplicable"> https://urldefense.proofpoint.com/v2/url?u=https-3A__en.wikipedia.org_wiki_Random-5Fearly-5Fdetection&d=DwICaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=1DGob4H4rxz6H8uITozGOCa0s5f4wCNtTa4UUKvcsvI&m=kISGqWlRwNHLBHvKCTkVQBoHyBW79qMFQEJNcoDZXb8X4GKLwHVeeE_D3Lh082fx&s=iyRkqzJBjwu2oS-YMRhtxJOpsK8JWJG5gEH_1DC5pwM&e=</a> > + * <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.aciri.org_floyd_papers_red_red.html&d=DwICaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=1DGob4H4rxz6H8uITozGOCa0s5f4wCNtTa4UUKvcsvI&m=kISGqWlRwNHLBHvKCTkVQBoHyBW79qMFQEJNcoDZXb8X4GKLwHVeeE_D3Lh082fx&s=HZKFgpOQ0CtqvK-_Zc-QZn87gBY4UrscDXgFihsXjvQ&e=" data-auth="NotApplicable"> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.aciri.org_floyd_papers_red_red.html&d=DwICaQ&c=nKjWec2b6R0mOyPaz7xtfQ&r=1DGob4H4rxz6H8uITozGOCa0s5f4wCNtTa4UUKvcsvI&m=kISGqWlRwNHLBHvKCTkVQBoHyBW79qMFQEJNcoDZXb8X4GKLwHVeeE_D3Lh082fx&s=HZKFgpOQ0CtqvK-_Zc-QZn87gBY4UrscDXgFihsXjvQ&e=</a> > + * @see struct rte_cman_red_params > + */ > + RTE_CMAN_RED = RTE_BIT32(0), > +}; > + > +/** > + * RED based congestion management configuration parameters. > + */ > +struct rte_cman_red_params { > + /** > + * Minimum threshold (min_th) value > + * > + * Value expressed as percentage. Value must be in 0 to 100(inclusive). > + */ > + uint8_t min_th; > + /** > + * Maximum threshold (max_th) value > + * > + * Value expressed as percentage. Value must be in 0 to 100(inclusive). > + */ > + uint8_t max_th; > + /** Inverse of packet marking probability maximum value (maxp = 1 / maxp_inv) */ > + uint16_t maxp_inv; > +}; > + > +#ifdef __cplusplus > +} > +#endif > + > +#endif /* RTE_CMAN_H */ > diff --git a/lib/ethdev/rte_ethdev.c b/lib/ethdev/rte_ethdev.c > index 0c2c1088c0..e4eb17221b 100644 > --- a/lib/ethdev/rte_ethdev.c > +++ b/lib/ethdev/rte_ethdev.c > @@ -642,7 +642,7 @@ rte_eth_dev_get_port_by_name(const char *name, uint16_t *port_id) > return -ENODEV; > } > > -static int > +int > eth_err(uint16_t port_id, int ret) > { > if (ret == 0) > diff --git a/lib/ethdev/rte_ethdev.h b/lib/ethdev/rte_ethdev.h > index a21f58b9cd..8df1cdfad0 100644 > --- a/lib/ethdev/rte_ethdev.h > +++ b/lib/ethdev/rte_ethdev.h > @@ -160,6 +160,7 @@ extern "C" { > #define RTE_ETHDEV_DEBUG_TX > #endif > > +#include <rte_cman.h> > #include <rte_compat.h> > #include <rte_log.h> > #include <rte_interrupts.h> > @@ -5314,6 +5315,169 @@ typedef struct { > __rte_experimental > int rte_eth_dev_priv_dump(uint16_t port_id, FILE *file); > > +/* Congestion management */ > + > +/** Enumerate list of ethdev congestion management objects */ > +enum rte_eth_cman_obj { > + /** Congestion management based on Rx queue depth */ > + RTE_ETH_CMAN_OBJ_RX_QUEUE = RTE_BIT32(0), > + /** > + * Congestion management based on mempool depth associated with Rx queue > + * @see rte_eth_rx_queue_setup() > + */ > + RTE_ETH_CMAN_OBJ_RX_QUEUE_MEMPOOL = RTE_BIT32(1), > +}; > + > +/** > + * @warning > + * @b EXPERIMENTAL: this structure may change, or be removed, without prior notice > + * > + * A structure used to retrieve information of ethdev congestion management. > + */ > +struct rte_eth_cman_info { > + /** > + * Set of supported congestion management modes > + * @see enum rte_cman_mode > + */ > + uint64_t modes_supported; > + /** > + * Set of supported congestion management objects > + * @see enum rte_eth_cman_obj > + */ > + uint64_t objs_supported; > + /** > + * Reserved for future fields. Always returned as 0 when > + * rte_eth_cman_info_get() is invoked > + */ > + uint8_t rsvd[8]; > +}; > + > +/** > + * @warning > + * @b EXPERIMENTAL: this structure may change, or be removed, without prior notice > + * > + * A structure used to configure the ethdev congestion management. > + */ > +struct rte_eth_cman_config { > + /** Congestion management object */ > + enum rte_eth_cman_obj obj; > + /** Congestion management mode */ > + enum rte_cman_mode mode; > + union { > + /** > + * Rx queue to configure congestion management. > + * > + * Valid when object is RTE_ETH_CMAN_OBJ_RX_QUEUE or > + * RTE_ETH_CMAN_OBJ_RX_QUEUE_MEMPOOL. > + */ > + uint16_t rx_queue; > + /** > + * Reserved for future fields. > + * It must be set to 0 when rte_eth_cman_config_set() is invoked > + * and will be returned as 0 when rte_eth_cman_config_get() is > + * invoked. > + */ > + uint8_t rsvd_obj_params[4]; > + } obj_param; > + union { > + /** > + * RED configuration parameters. > + * > + * Valid when mode is RTE_CMAN_RED. > + */ > + struct rte_cman_red_params red; > + /** > + * Reserved for future fields. > + * It must be set to 0 when rte_eth_cman_config_set() is invoked > + * and will be returned as 0 when rte_eth_cman_config_get() is > + * invoked. > + */ > + uint8_t rsvd_mode_params[4]; > + } mode_param; > +}; > + > +/** > + * @warning > + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice > + * > + * Retrieve the information for ethdev congestion management > + * > + * @param port_id > + * The port identifier of the Ethernet device. > + * @param info > + * A pointer to a structure of type *rte_eth_cman_info* to be filled with > + * the information about congestion management. > + * @return > + * - (0) if successful. > + * - (-ENOTSUP) if support for cman_info_get does not exist. > + * - (-ENODEV) if *port_id* invalid. > + * - (-EINVAL) if bad parameter. > + */ > +__rte_experimental > +int rte_eth_cman_info_get(uint16_t port_id, struct rte_eth_cman_info *info); > + > +/** > + * @warning > + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice > + * > + * Initialize the ethdev congestion management configuration structure with default values. > + * > + * @param port_id > + * The port identifier of the Ethernet device. > + * @param config > + * A pointer to a structure of type *rte_eth_cman_config* to be initialized > + * with default value. > + * @return > + * - (0) if successful. > + * - (-ENOTSUP) if support for cman_config_init does not exist. > + * - (-ENODEV) if *port_id* invalid. > + * - (-EINVAL) if bad parameter. > + */ > +__rte_experimental > +int rte_eth_cman_config_init(uint16_t port_id, struct rte_eth_cman_config *config); > + > +/** > + * @warning > + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice > + * > + * Configure ethdev congestion management > + * > + * @param port_id > + * The port identifier of the Ethernet device. > + * @param config > + * A pointer to a structure of type *rte_eth_cman_config* to be configured. > + * @return > + * - (0) if successful. > + * - (-ENOTSUP) if support for cman_config_set does not exist. > + * - (-ENODEV) if *port_id* invalid. > + * - (-EINVAL) if bad parameter. > + */ > +__rte_experimental > +int rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *config); > + > +/** > + * @warning > + * @b EXPERIMENTAL: this API may change, or be removed, without prior notice > + * > + * Retrieve the applied ethdev congestion management parameters for the given port. > + * > + * @param port_id > + * The port identifier of the Ethernet device. > + * @param config > + * A pointer to a structure of type *rte_eth_cman_config* to retrieve > + * congestion management parameters for the given object. > + * Application must fill all parameters except mode_param parameter in > + * struct rte_eth_cman_config. > + * > + * @return > + * - (0) if successful. > + * - (-ENOTSUP) if support for cman_config_get does not exist. > + * - (-ENODEV) if *port_id* invalid. > + * - (-EINVAL) if bad parameter. > + */ > +__rte_experimental > +int rte_eth_cman_config_get(uint16_t port_id, struct rte_eth_cman_config *config); > + > #include <rte_ethdev_core.h> > > /** > diff --git a/lib/ethdev/rte_ethdev_cman.c b/lib/ethdev/rte_ethdev_cman.c > new file mode 100644 > index 0000000000..4a1bdd7bd0 > --- /dev/null > +++ b/lib/ethdev/rte_ethdev_cman.c > @@ -0,0 +1,101 @@ > +/* SPDX-License-Identifier: BSD-3-Clause > + * Copyright(C) 2022 Marvell International Ltd. > + */ > + > +#include <stdint.h> > + > +#include <rte_errno.h> > +#include "rte_ethdev.h" > +#include "ethdev_driver.h" > +#include "ethdev_private.h" > + > +/* Get congestion management information for a port */ > +int > +rte_eth_cman_info_get(uint16_t port_id, struct rte_eth_cman_info *info) > +{ > + struct rte_eth_dev *dev; > + > + RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV); > + dev = &rte_eth_devices[port_id]; > + > + if (info == NULL) { > + RTE_ETHDEV_LOG(ERR, "congestion management info is NULL\n"); > + return -EINVAL; > + } > + > + if (dev->dev_ops->cman_info_get == NULL) { > + RTE_ETHDEV_LOG(ERR, "Function not implemented\n"); > + return -ENOTSUP; > + } > + > + memset(info, 0, sizeof(struct rte_eth_cman_info)); > + return eth_err(port_id, (*dev->dev_ops->cman_info_get)(dev, info)); > +} > + > +/* Initialize congestion management structure with default values */ > +int > +rte_eth_cman_config_init(uint16_t port_id, struct rte_eth_cman_config *config) > +{ > + struct rte_eth_dev *dev; > + > + RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV); > + dev = &rte_eth_devices[port_id]; > + > + if (config == NULL) { > + RTE_ETHDEV_LOG(ERR, "congestion management config is NULL\n"); > + return -EINVAL; > + } > + > + if (dev->dev_ops->cman_config_init == NULL) { > + RTE_ETHDEV_LOG(ERR, "Function not implemented\n"); > + return -ENOTSUP; > + } > + > + memset(config, 0, sizeof(struct rte_eth_cman_config)); > + return eth_err(port_id, (*dev->dev_ops->cman_config_init)(dev, config)); > +} > + > +/* Configure congestion management on a port */ > +int > +rte_eth_cman_config_set(uint16_t port_id, const struct rte_eth_cman_config *config) > +{ > + struct rte_eth_dev *dev; > + > + RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV); > + dev = &rte_eth_devices[port_id]; > + > + if (config == NULL) { > + RTE_ETHDEV_LOG(ERR, "congestion management config is NULL\n"); > + return -EINVAL; > + } > + > + if (dev->dev_ops->cman_config_set == NULL) { > + RTE_ETHDEV_LOG(ERR, "Function not implemented\n"); > + return -ENOTSUP; > + } > + > + return eth_err(port_id, (*dev->dev_ops->cman_config_set)(dev, config)); > +} > + > +/* Retrieve congestion management configuration of a port */ > +int > +rte_eth_cman_config_get(uint16_t port_id, struct rte_eth_cman_config *config) > +{ > + struct rte_eth_dev *dev; > + > + RTE_ETH_VALID_PORTID_OR_ERR_RET(port_id, -ENODEV); > + dev = &rte_eth_devices[port_id]; > + > + if (config == NULL) { > + RTE_ETHDEV_LOG(ERR, "congestion management config is NULL\n"); > + return -EINVAL; > + } > + > + if (dev->dev_ops->cman_config_get == NULL) { > + RTE_ETHDEV_LOG(ERR, "Function not implemented\n"); > + return -ENOTSUP; > + } > + > + memset(config, 0, sizeof(struct rte_eth_cman_config)); > + return eth_err(port_id, (*dev->dev_ops->cman_config_get)(dev, config)); > +} > diff --git a/lib/ethdev/version.map b/lib/ethdev/version.map > index 3651ceb234..857d21be1c 100644 > --- a/lib/ethdev/version.map > +++ b/lib/ethdev/version.map > @@ -287,6 +287,10 @@ EXPERIMENTAL { > rte_mtr_meter_vlan_table_update; > > # added in 22.11 > + rte_eth_cman_config_get; > + rte_eth_cman_config_init; > + rte_eth_cman_config_set; > + rte_eth_cman_info_get; > rte_flow_async_action_handle_query; > rte_mtr_meter_policy_get; > rte_mtr_meter_profile_get; </div> </div> </body> </html>