[dpdk-dev] [RFC v5] regexdev: introduce regexdev subsystem

[Ori Kam]

Hi Xiang,

May I ask when do you plan to add the Hyperscan code to the DPDK?

> -----Original Message-----
> From: dev <dev-bounces at dpdk.org> On Behalf Of Wang Xiang
> Sent: Monday, March 2, 2020 9:05 AM
> To: Ori Kam <orika at mellanox.com>
> Cc: jerinj at marvell.com; dev at dpdk.org; pbhagavatula at marvell.com; Shahaf
> Shuler <shahafs at mellanox.com>; hemant.agrawal at nxp.com; Opher Reviv
> <opher at mellanox.com>; Alex Rosenbaum <alexr at mellanox.com>;
> dovrat at marvell.com; pkapoor at marvell.com; nipun.gupta at nxp.com;
> bruce.richardson at intel.com; yang.a.hong at intel.com; harry.chang at intel.com;
> gu.jian1 at zte.com.cn; shanjiangh at chinatelecom.cn;
> zhangy.yun at chinatelecom.cn; lixingfu at huachentel.com; wushuai at inspur.com;
> yuyingxia at yxlink.com; fanchenggang at sunyainfo.com;
> davidfgao at tencent.com; liuzhong1 at chinaunicom.cn;
> zhaoyong11 at huawei.com; oc at yunify.com; jim at netgate.com;
> hongjun.ni at intel.com; j.bromhead at titan-ic.com; deri at ntop.org;
> fc at napatech.com; arthur.su at lionic.com; Thomas Monjalon
> <thomas at monjalon.net>
> Subject: Re: [dpdk-dev] [RFC v5] regexdev: introduce regexdev subsystem
> 
> Hi Ori,
> 
> Comments below.
> 
> Thanks,
> Xiang
> 
> On Thu, Feb 27, 2020 at 03:08:35PM +0000, Ori Kam wrote:
> > From: Jerin Jacob <jerinj at marvell.com>
> >
> > Even though there are some vendors which offer Regex HW offload, due to
> > lack of standard API, It is diffcult for DPDK consumer to use them
> > in a portable way.
> >
> > This _RFC_ attempts to standardize the RegEx/DPI offload APIs for DPDK.
> >
> > This RFC crafted based on SW Regex API frameworks such as libpcre and
> > hyperscan and a few of the RegEx HW IPs which I am aware of.
> >
> > RegEx pattern matching applications:
> > * Next Generation Firewalls (NGFW)
> > * Deep Packet and Flow Inspection (DPI)
> > * Intrusion Prevention Systems (IPS)
> > * DDoS Mitigation
> > * Network Monitoring
> > * Data Loss Prevention (DLP)
> > * Smart NICs
> > * Grammar based content processing
> > * URL, spam and adware filtering
> > * Advanced auditing and policing of user/application security policies
> > * Financial data mining - parsing of streamed financial feeds
> > * Application recognition.
> > * Dmemory introspection.
> > * Natural Language Processing (NLP)
> > * Sentiment Analysis.
> > * Big data databse acceleration.
> > * Computational storage.
> >
> > Request to review from HW and SW RegEx vendors and RegEx application
> > users to have portable DPDK API for RegEx.
> >
> > The API schematics are based cryptodev, eventdev and ethdev existing
> > device API.
> >
> > Signed-off-by: Jerin Jacob <jerinj at marvell.com>
> > Signed-off-by: Pavan Nikhilesh <pbhagavatula at marvell.com>
> > Signed-off-by: Ori Kam <orika at mellanox.com>
> > ---
> > V5:
> >  * Remove unused iov struct.
> > V4:
> >  * Replace iov with mbuf.
> >  * Small ML comments.
> > V3:
> >  * Change subject title.
> > V2:
> >  * Address ML comments.
> > ---
> > +
> > +#define RTE_REGEX_DEV_SUPP_PCRE_GREEDY_F (1ULL << 6)
> > +/**< RegEx device support PCRE Greedy mode.
> > + * For example if the RegEx is 'AB\d*?' then '*?' represents zero or
> unlimited
> > + * matches. In greedy mode the pattern 'AB12345' will be matched
> completely
> > + * where as the ungreedy mode 'AB' will be returned as the match.
> > + * @see struct rte_regex_dev_info::regex_dev_capa
> > + */
> > +
> 
> Hyperscan actually supports "match all" semantic, neither greedy nor
> ungreedy,
> which is different from PCRE. In the case above, AB, AB1, ..., AB12345 will all
> be returned as matches. Do HW solutions support this?

No our HW doesn't support this.
Jerin, does Marvell HW support this?

> Can we add a new flag like RTE_REGEX_DEV_SUPP_PCRE_MATCHALL_F?
> Similarly, we can define a flag RTE_REGEX_PCRE_RULE_MATCHALL_F so
> Hyperscan
> users have to set this flag during rule compile.
> 

Sure,

> > +#define RTE_REGEX_DEV_SUPP_PCRE_LOOKAROUND_ASRT_F (1ULL << 7)
> > +/**< RegEx device support PCRE Lookaround assertions
> > + * (Zero-width assertions). Example RegEx is '[a-z]+\d+(?=!{3,})' if
> > + * the given pattern is 'dwad1234!' the RegEx engine doesn't report any
> matches
> > + * because the assert '(?=!{3,})' fails. The pattern 'dwad123!!!' would return
> a
> > + * successful match.
> > + * @see struct rte_regex_dev_info::regex_dev_capa
> > + */
> > +
> > +
> > +/**
> > + * RegEx device information
> > + */
> > +struct rte_regex_dev_info {
> > +	const char *driver_name; /**< RegEx driver name. */
> > +	struct rte_device *dev;	/**< Device information. */
> > +	uint16_t max_matches;
> > +	/**< Maximum matches per scan supported by this device. */
> > +	uint16_t max_queue_pairs;
> > +	/**< Maximum queue pairs supported by this device. */
> > +	uint16_t max_payload_size;
> > +	/**< Maximum payload size for a pattern match request or scan.
> > +	 * @see RTE_REGEX_DEV_CFG_CROSS_BUFFER_SCAN_F
> > +	 */
> > +	uint32_t max_rules_per_group;
> > +	/**< Maximum rules supported per group by this device. */
> > +	uint16_t max_groups;
> > +	/**< Maximum groups supported by this device. */
> > +	uint32_t regex_dev_capa;
> > +	/**< RegEx device capabilities. @see RTE_REGEX_DEV_CAPA_* */
> > +	uint64_t rule_flags;
> > +	/**< Supported compiler rule flags.
> > +	 * @see RTE_REGEX_PCRE_RULE_*, struct rte_regex_rule::rule_flags
> > +	 */
> > +	uint8_t max_scatter_gather;
> > +	/**< The max supported number of buffers that can
> > +	 * be used in a single ops. The total size of all elements
> > +	 * must be less then max_payload_size.
> > +	 */
> 
> s/then/than
> 

Will fix.

> > +};
> > +
> > +/**
> > + * @warning
> > + * @b EXPERIMENTAL: this API may change without prior notice.
> > + *
> > + * Retrieve the contextual information of a RegEx device.
> > + *
> > + * @param dev_id
> > + *   The identifier of the device.
> > + *
> > + * @param[out] dev_info
> > + *   A pointer to a structure of type *rte_regex_dev_info* to be filled with
> the
> > + *   contextual information of the device.
> > + *
> > + * @return
> > + *   - 0: Success, driver updates the contextual information of the RegEx
> device
> > + *   - <0: Error code returned by the driver info get function.
> > + *
> > + */
> > +__rte_experimental
> > +int
> > +rte_regex_dev_info_get(uint8_t dev_id, struct rte_regex_dev_info
> *dev_info);
> > +
> > +/* Enumerates RegEx device configuration flags */
> > +#define RTE_REGEX_DEV_CFG_CROSS_BUFFER_SCAN_F (1ULL << 0)
> > +/**< Cross buffer scan refers to the ability to be able to detect
> > + * matches that occur across buffer boundaries, where the buffers are
> related
> > + * to each other in some way. Enable this flag when to scan payload size
> > + * greater struct struct rte_regex_dev_info::max_payload_size and/or
> > + * matches can present across scan buffer boundaries.
> 
> s/struct/than
> 

Will fix.

> > + *
> > + * @see struct rte_regex_dev_info::max_payload_size
> > + * @see struct rte_regex_dev_config::dev_cfg_flags,
> rte_regex_dev_configure()
> > + * @see RTE_REGEX_OPS_RSP_PMI_SOJ_F
> > + * @see RTE_REGEX_OPS_RSP_PMI_EOJ_F
> > + */
> > +
> > +
> > +/* Enumerates RegEx response flags. */
> > +#define RTE_REGEX_OPS_RSP_PMI_SOJ_F (1 << 0)
> > +/**< Indicates that the RegEx device has encountered a partial match at the
> > + * start of scan in the given buffer.
> > + *
> > + * @see RTE_REGEX_DEV_CFG_CROSS_BUFFER_SCAN_F
> > + */
> > +
> Hyperscan supports cross buffer scan and only reports true matches instead of
> partial matches. Can we have users to config this partial match capability?
> 

Do you mean something like this:
RTE_REGEX_OPS_RSP_PMI_FOJ_F
/**< Indicates that the RegEx device has encountered a full match in the current buffer.
  * The match was started in previous buffer.
  */

> > +#define RTE_REGEX_OPS_RSP_PMI_EOJ_F (1 << 1)
> > +/**< Indicates that the RegEx device has encountered a partial match at the
> > + * end of scan in the given buffer.
> > + *
> > + * @see RTE_REGEX_DEV_CFG_CROSS_BUFFER_SCAN_F
> > + */
> > +