[dpdk-ci] [PATCH v2 10/10] tools: skip the IDs we already fetched

[Thomas Monjalon]

18/10/2021 10:04, Ali Alnubani:
> From: Thomas Monjalon <thomas at monjalon.net>
> > 11/10/2021 21:30, Ali Alnubani:
> > > From: Thomas Monjalon <thomas at monjalon.net>
> > > > 21/09/2021 16:35, alialnu at nvidia.com:
> > > > > From: Ali Alnubani <alialnu at nvidia.com>
> > > > >
> > > > > Store the IDs we already fetched in a file and don't
> > > > > run 'callcmd' again for them.
> > > >
> > > > We store all IDs. Should we manually remove olds one from time to time?
> > >
> > > Do you have a suggestion for when should we clear this file? Maybe each
> > time the script starts?
> > 
> > Yes at each start, we can remove the very old entries, like more than 10
> > hours old.
> 
> Wouldn't this require using and depending on a database (at least an SQLite one) to additionally store timestamps to use for queries?
> Do you have a simpler way in mind to keep track of the IDs and their timestamps to process later based on how long they have been in the file? Maybe keep only up to a certain number of IDs in this file (up to a 1000 maybe)?

Yes we can store each entry as a line starting with timestamp:
	timestamp ID