NIST update - Allow to stop and continue later

[richardzaat]

Is your feature request related to a problem? Please describe.
I'm always frustrated when I install a new version of Dependency Check and need to download the complete NIST library. Now it stops at 220.000 downloads and just does not continue until I cancel the whole thing

Describe the solution you'd like
A clear and concise description of what I want to happen is that the next time I update the database it continuous where it left off. Same as when the database was already filled and it only downloads the delta. Why not do so now?

Describe alternatives you've considered
I've considered replacing the empty database with one from a previous version of DC, but does not feel like a good solution.

Additional context
n/a

[marcelstoer]

I've considered replacing the empty database with one from a previous version of DC, but does not feel like a good solution.

Why? Only rarely should it be necessary to start with an empty database. Besides, a new ODC tool version doesn't necessarily require a new database version.

It is strongly recommended that you cache the database one way or the other. Various approaches are documented at https://jeremylong.github.io/DependencyCheck/data/index.html#The_NVD_Database.

[kantipenko]

Why? Only rarely should it be necessary to start with an empty database. Besides, a new ODC tool version doesn't necessarily require a new database version.

It is strongly recommended that you cache the database one way or the other.

We used cashing approach, after upgrade to 12.1.0 we can not have even one first successful update from NVD API. Maximum it went was 86% of records. After that time it just hangs for hours.
Recent attempt example

[marcelstoer]

A more efficient way to initialize a cold database is to start with the dependency check data files (see documentation I posted for details).

• First, get the feed files as a basis.
• Then fetch the latest delta from the NVD API.
• Then offer the thusly created database to the actual build jobs.

$ dependency-check.bat --updateonly --nvdDatafeed=https://dependency-check.github.io/DependencyCheck_Builder/nvd_cache/nvdcve-{0}.json.gz
$ dependency-check.bat --updateonly --nvdValidForHours=0 --nvdApiKey="$NVD_API_KEY"

[richardzaat]

Nice one, Will definitely try that one. My way of thinking was that if initialization of the database happens after the first batch of 10 or 20K, then from then onwards additional blocks of updates van be treated individually like now when you update once a day for instance. But definitely will try the suggestion. Many thanks Op wo 26 feb. 2025 17:09 schreef Marcel Stör ***@***.***>:

…

A more efficient way to initialize a cold database is to start with the dependency check data files (see documentation I posted for details). - First, get the feed files as a basis. - Then fetch the latest delta from the NVD API. - Then offer the thusly created database to the actual build jobs. $ dependency-check.bat --updateonly --nvdDatafeed=https://dependency-check.github.io/DependencyCheck_Builder/nvd_cache/nvdcve-{0}.json.gz $ dependency-check.bat --updateonly --nvdValidForHours=0 --nvdApiKey="$NVD_API_KEY" — Reply to this email directly, view it on GitHub <#7449 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AECVQYMMBCCE6MM65DDJ5ND2RXRNDAVCNFSM6AAAAABXTF7GNOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBVGUZDINJYHA> . You are receiving this because you authored the thread.Message ID: ***@***.***> [image: marcelstoer]*marcelstoer* left a comment (dependency-check/DependencyCheck#7449) <#7449 (comment)> A more efficient way to initialize a cold database is to start with the dependency check data files (see documentation I posted for details). - First, get the feed files as a basis. - Then fetch the latest delta from the NVD API. - Then offer the thusly created database to the actual build jobs. $ dependency-check.bat --updateonly --nvdDatafeed=https://dependency-check.github.io/DependencyCheck_Builder/nvd_cache/nvdcve-{0}.json.gz $ dependency-check.bat --updateonly --nvdValidForHours=0 --nvdApiKey="$NVD_API_KEY" — Reply to this email directly, view it on GitHub <#7449 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AECVQYMMBCCE6MM65DDJ5ND2RXRNDAVCNFSM6AAAAABXTF7GNOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMOBVGUZDINJYHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[franzgranlund]

I have the same problem. It always stops at [INFO] Downloaded 220,000/283,171 (78%).

[kantipenko]

A more efficient way to initialize a cold database is to start with the dependency check data files (see documentation I posted for details).

Many Thanks! This approach have worked for us.

• First, get the feed files as a basis.
  Would you recommend we add it to scheduled run or only needed on the initial DB seeding?
• Then fetch the latest delta from the NVD API.
  At the moment we have only this one

[marcelstoer]

Would you recommend we add it to scheduled run

We do this every night. This ensures that every build job that runs during the day uses the same database.

[marcelstoer]

I am closing this as @jeremylong closed my request (asking for the same feature) for the same reasons I stated above: #7180 (comment)