[dpdk-ci] [PATCH v2 10/10] tools: skip the IDs we already fetched

[Ali Alnubani]

> -----Original Message-----
> From: Thomas Monjalon <thomas at monjalon.net>
> Sent: Tuesday, October 12, 2021 9:45 AM
> To: Ali Alnubani <alialnu at nvidia.com>
> Cc: ci at dpdk.org; jerinj at marvell.com; ferruh.yigit at intel.com;
> david.marchand at redhat.com; juraj.linkes at pantheon.tech
> Subject: Re: [PATCH v2 10/10] tools: skip the IDs we already fetched
> 
> 11/10/2021 21:30, Ali Alnubani:
> > From: Thomas Monjalon <thomas at monjalon.net>
> > > 21/09/2021 16:35, alialnu at nvidia.com:
> > > > From: Ali Alnubani <alialnu at nvidia.com>
> > > >
> > > > Store the IDs we already fetched in a file and don't
> > > > run 'callcmd' again for them.
> > >
> > > We store all IDs. Should we manually remove olds one from time to time?
> > >
> >
> > Do you have a suggestion for when should we clear this file? Maybe each
> time the script starts?
> 
> Yes at each start, we can remove the very old entries, like more than 10
> hours old.

Wouldn't this require using and depending on a database (at least an SQLite one) to additionally store timestamps to use for queries?
Do you have a simpler way in mind to keep track of the IDs and their timestamps to process later based on how long they have been in the file? Maybe keep only up to a certain number of IDs in this file (up to a 1000 maybe)?

> 
> > > We need an explanation about the strategy, why it is needed.
> > > I think it is because filtering by date is not enough.
> > > In order to not miss any patch, we should request a date earlier
> > > than the previous fetch and skip those already fetched.
> >
> > The reason this change was made isn't because filtering by date is not
> enough, it's because
> > I want to avoid feeding the same ID to 'callcmd' more than once.
> > This can happen if a patchwork ID was created between recording
> date_now and fetching the API.
> > I don't think we are missing any IDs, even without this change.
> >
> > > Where the "earlier date" is defined?
> >
> > There are 2 variables, "date_now", which is recorded right before fetching
> from the API, and then gets written
> > to the file, and "since", which is the last date that was written to the file.
> 
> OK please update the commit log.
> 

Updated commit messages in v3.