Data sources
Where the data comes from.
These are the public sources the collector uses. Each one has a verified access path and a licence basis that allows our use.
We store exact source responses and stable record keys. The panel pipeline extracts facts from SEC documents, links exact identifiers, and publishes source-linked facts and neutral summaries. Relationship edges and full OFAC or GLEIF resolution are pending. The raw archive stays private and is not published.
Sources
Public sources, no credentials.
Each source is listed with what we collect, how we access it, and the licence basis we rely on. Attribution for each source follows the table.
| Source | What is collected | Access | Licence basis | Last verified |
|---|---|---|---|---|
| SEC EDGAR U.S. Securities and Exchange Commission | Submissions metadata and recent filing records for the AI universe. | data.sec.gov submissions JSON; optional companyfacts JSON | U.S. government public domain | 2026-10-21 |
| CFTC Commitments of Traders U.S. Commodity Futures Trading Commission | Latest Commitments of Traders report across the configured datasets. | Socrata SODA API at publicreporting.cftc.gov | U.S. government public domain | 2026-10-21 |
| Federal Reserve Board H.15 Board of Governors of the Federal Reserve System | H.15 selected interest rates, full-release ZIP. | Data Download Program H.15 full-release ZIP | Attribution required | 2026-10-21 |
| OFAC SDN Advanced U.S. Treasury, Office of Foreign Assets Control | SDN Advanced sanctions list XML. | OFAC Sanctions List Service XML export | CC0 | 2026-10-21 |
| GLEIF LEI Records Global Legal Entity Identifier Foundation | LEI records, sampled pages; parent relationships are pending. | GLEIF LEI API | CC0 | 2026-10-21 |
| BLS API v2 (v1 unregistered mode) U.S. Bureau of Labor Statistics | CPI, PPI, and total nonfarm payroll observations. | BLS public API v2, no registration key | Attribution required | 2026-10-21 |
| U.S. Treasury Fiscal Data Bureau of the Fiscal Service, U.S. Treasury | Daily total public debt outstanding, latest rows. | Treasury Fiscal Data API | U.S. government public domain | 2026-10-21 |
| Federal Register API Office of the Federal Register (NARA) / GPO | Documents from BIS, OFAC, SEC, and CFTC, incremental. | Federal Register REST API | U.S. government public domain | 2026-10-21 |
| OpenFIGI Bloomberg / OpenFIGI | FIGI mappings for AI universe tickers. | OpenFIGI v3 mapping API | OpenFIGI terms | 2026-10-21 |
All listed sources had a verified access path on 2026-10-21. SEC filing records are keyed by accession. FRB H.15 and OFAC SDN are stored as raw files; their parsing comes later.
Collection rules
How the collector behaves.
Declared identity
Every request carries a declared User-Agent with a contact address. We do not rotate IP addresses or User-Agents.
Source rate limits
Each source has its own token bucket. SEC requests stay at or below 8 per second. Retries honour Retry-After.
Incremental append
Raw bodies are content-addressed. An unchanged body is not written twice. Records use stable keys, so a repeated run does not duplicate them.
Conditional GET
Where a source provides an ETag or Last-Modified value, we send a conditional request. OFAC returns HTTP 304. Some sources return the same body with HTTP 200; the hash still prevents a duplicate archive write. CFTC skips a dataset whose latest report date is already stored.
No aggregation
This stage collects raw material only. There is no scoring, ranking, entity resolution, or derived dataset.
Terms first
A source is enabled only when its terms allow our use and its access path has been verified. Sources that require a UserID are not used.
Excluded
What we do not collect.
| Source or category | Reason |
|---|---|
| BEA | Requires a UserID and account registration. Not collected. |
| Consolidated Screening List (CSL) | The documented bulk endpoints returned 404 and the API host had an expired certificate. Not collected. |
| FRED | Terms bar commercial use and storage. No connector exists. |
| Market prices | Not collected. No price data appears on this site. |
| Login-only and paywalled content | Not accessed. We do not scrape behind logins. |
| Social media and non-public information | Not used. |
Attribution
Source statements.
These statements apply wherever data from the source appears. See the data sources and licences page for the full list.
- SEC EDGAR: Source: U.S. SEC EDGAR. Not affiliated with or endorsed by the SEC.
- CFTC Commitments of Traders: Source: CFTC Commitments of Traders
- Federal Reserve Board H.15: Source: Board of Governors of the Federal Reserve System, H.15
- OFAC SDN Advanced: Source: OFAC SDN List
- GLEIF LEI Records: Contains LEI data from GLEIF.
- BLS API v2 (v1 unregistered mode): Source: U.S. Bureau of Labor Statistics. BLS.gov cannot vouch for the data or analyses derived from these data after the data have been retrieved from BLS.gov.
- U.S. Treasury Fiscal Data: Source: U.S. Treasury Fiscal Data
- Federal Register API: Source: Federal Register. FR.gov text is unofficial; link to govinfo for the official edition.
- OpenFIGI: FIGI identifiers: OpenFIGI. Descriptive fields: cite OpenFIGI/Bloomberg.
SECfilingAI is not affiliated with, endorsed by, or connected to the U.S. Securities and Exchange Commission or any other agency listed here. Source names are used only to identify the public data.