NOTE: This page is out-of-date, we now have most of the data in Microsoft Azure and the plan is to make this publicly available on a new permanent website. e.g. with public API's which will allow interaction with the data, plus a public collection of all the various resources we have collected along this journey.
Bob Pisani Gold Valt Tour - recorded off Cable TV (297 mb)
Important Notes:
The project has the following parts:
Phase 1. Data Accumulation and Design (status: completed 2011)
Downloading the published Bar List information from various locations, for archive. This phase includes the automated extraction of contents, storage in the database and subsequent indexing, storage of archives in the cloud environment. The maintenance and improvement of the process is ongoing.
Phase 2. Aggregation Mechanic (status: mature 2012)
Create a normalized data model for all bar lists, with a common list of refiners and brands. This stage gives us a core table with normalized unique bar signatures, as well as a basic 'ledger' which allows us to easily track vault movements (Universal Bar List, Universal Ledger).
Phase 3. Advanced Cleanup (status: done 2013)
Identification of patterns and grouping, cleaning of the data and adding more metadata (like age of bars, etc). This phase is completed once we have a table which allows us to identify runs and sequences of bars from refiners.
Phase 4. Detailed Pattern Forensics (status: achieved 2014)
Interpreting the results and drawing conclusions from the data. This is where we hope to do advanced stuff like look at volumes and throughput, dark bullion inventories and inter-vault movements. Hopefully this shows any correlation between the data and price movements (hopefully predictive).
Consolidation Period (2022 - 2030) :
Bibliography: I am using the following books as reference:
- Gold Refiners & Bars Worldwide (1991) by Nigel Desebrock [Grendon Research International] link
- GLD History History Book (2012) by Timothy Green [LBMA] link
Questions? Comments? Criticisms? Scorn, Threats, Inquiries, etc. Email: bullionbars at hotmail.com
Bob Pisani Gold Valt Tour - recorded off Cable TV (297 mb)
Important Notes:
- I graciously acknowledge the input provided by various people including Mark Hynes, Bron Suchecki, Costata, Victor the Cleaner, FOFOA and all my readers who have provided feedback in various forms.
- The database by itself (the warehousing of data) does not 'prove' the validity of the bar lists. However, it provides a base for addressing discussion at 'Level 1, 2 and 3'. For anyone wishing to critique the approach, we recommend you read this post about 'How Can We Trust the Bar List Data?' first, and then comment.
- The ETF bars only show us a portion of bullion on the market - more specifically 'Category 1 (of 7)', "Identified Bullion Inventories". For a full discussion of the other types of Bullion (which we do not track here), please refer to Larry's comprehensive Typology for Above Ground Silver Stocks.
- I generally recognize my work as the third major attempt to analyze SLV’s data. The first is Project Mayhem’s 2009 analysis; the second is Joshua Gibbon’s continued excellent work at http://about.ag/slv/. Joshua also provided copies of the early 2011 SLV bar lists, so thanks to him we have SLV data from before we started recording in mid-2011 (I do not count Mark Anthony's Seeking Alpha analysis because it was not sufficiently detailed to be classed as a proper study).
- To date, no one has refuted my analysis (apart from ignorant generalizations). I would actually prefer that people who are able, look at the data critically. I will do my best to (a) indicate where I think limit of reading errors exist, and (b) provide as much raw data as is practical so that the process is as transparent as possible. At the heart of the project is an indexing system which allows us to trace an individual record back to the original source document (which is in the archive).
- This is the LARGEST KNOWN PUBLIC REPOSITORY OF HISTORICAL BAR LISTS ON THE INTERNET. Attempts will be made to make the repository permanent.
- We expect that the project will eventually focus on Gold only, which is the only precious metal worth discussing ;p Of course we'll keep SLV warehousing going as well.
The project has the following parts:
Phase 1. Data Accumulation and Design (status: completed 2011)
Downloading the published Bar List information from various locations, for archive. This phase includes the automated extraction of contents, storage in the database and subsequent indexing, storage of archives in the cloud environment. The maintenance and improvement of the process is ongoing.
Phase 2. Aggregation Mechanic (status: mature 2012)
Create a normalized data model for all bar lists, with a common list of refiners and brands. This stage gives us a core table with normalized unique bar signatures, as well as a basic 'ledger' which allows us to easily track vault movements (Universal Bar List, Universal Ledger).
Phase 3. Advanced Cleanup (status: done 2013)
Identification of patterns and grouping, cleaning of the data and adding more metadata (like age of bars, etc). This phase is completed once we have a table which allows us to identify runs and sequences of bars from refiners.
Phase 4. Detailed Pattern Forensics (status: achieved 2014)
Interpreting the results and drawing conclusions from the data. This is where we hope to do advanced stuff like look at volumes and throughput, dark bullion inventories and inter-vault movements. Hopefully this shows any correlation between the data and price movements (hopefully predictive).
Consolidation Period (2022 - 2030) :
- Rebuild Data Warehouse using best practices and automation. (DONE)
- Rework the database into a 'NO SQL' stucture (DONE)
- Build an API which can allow others to easily get to the aggregated data results
- Build 'wiki' style allowing public to contribute to refiners and bars information
Deep Patterns Analysis using Advanced AI (TBD)
-
Phase 5. End Game (~2070)
Hand over the project to assistant AI to continue the work.
The goals of the database are:
(A) To give me experience in handling large data volumes and mathematical processing, as the basis for another (non-metals related) project.
(B) To prove or create a reasonable set of arguments for:
(i) ... the validity (or non-validity) of the record of inventory inside the primary bullion-based ETF's, hence proving or disproving the popular silver meme which argues that the ETF's are largely empty (i.e. address Level 1 arguments).
(ii) ... whether the contents of the ETF's are real or fake (i.e. Tunsten or Molybdenum), by looking at refiner production runs and metals origins. (i.e. address Level 2 arguments).
(iii) ... the existence (or not) of the supposed 'Dark Pool' bullion of the OTC market, which we surmise is used by the bullion banks to affect price movements.
Software Used:
- Microsoft SQL Server
- Custom-Built Software (C#)
- PdfPig Nuget Library (for PDF extraction)
- Windows Azure Storage (cloud archive and indexing)
Accessing the Historical Archives: The original SLV PDF file archives are preserved in cloud storage but are quite huge. To save bandwidth costs these will be compressed before I make them available to the public (but if anyone is interested in the data please email me). Clicking on each link gives a list of the files inside that particular cloud container - use the URL displayed to download the list resource.
List of Articles (apologies, not chronological):
The Earliest Known GLD Bar Lists - We only started collecting in 2011, but though some sleuthing we managed to track down some early lists. These provide a great reference point for our tracking and there are even some bars which are shown to have been in the vault since 20-November 2004!
How can We Trust the Bar List Data? When people say they don't trust the bar list data, what excactly does that actually mean? This is my first attempt to give a breakdown of argument in an effort to improve the resolution of questions - and end up explaining what questions CANNOT be answered using the database. I consider this the most important article of this series because it establishes a proper framework for investigation of how the data will square off against people's belief systems.
SLV Database 1 - Original premise and basis for the bullion bars project, including some basic analysis of the SLV data and ideas for the database.
SLV Database 2 - Basic analysis of SLV data to demonstrate (matched by Bar records) that some of the silver removed from SLV showed up again in subsequent 'add' events (first evidence of dark bullion).
SLV Database 3 - The most polished analysis which proves in great detail that over one quarter of the silver in the Perth Mint Allocated Pool Silver used to exist in SLV, and takes the same analysis to show that none of Sprott's PSLV bars used to exist in SLV. Discusses the implications from this.
SLV Database 4 - Another in-depth data exercise which compares the bar list data from Goldmoney and Bullionvault to conclude that some SLV bars jumped vaults. Includes a discussion of the different types of 'Vault Jumpers' as well as the implications, given that most results are pointing in one specific direction. Bad news for the 'SLV is Empty' crowd, since the amount of heavy research they must debunk, is growing substantially!
SLV Database 5 - Dark Bullion - All our investigations to date show a common element. This post dives straight into the definitions of how we can look at the warehoused data, to get more clarification on the volume of physical silver that sits behind SLV. The definitions and the chart required over 6-8 months of preparation and custom-created software and is our most sophisticated post yet.
Zero Hedge ZJ6752 - Bob Pisani takes a tour of the GLD vault and the bar he shows to the camera is not registered in the GLD inventory. Conspiracy? Not so fast. Because we have huge gobs of data available, we found the bar reasonably quickly and decided that the spin from the article was a storm in a teacup.
Bob Pisani Revisited (a fresh look) - Over a year later, we use our big database to successfully vindicate Bob Pisani's September 2011 documentary by identifying 10 new bars held in GLD from the video clip. Digital forensics at its best, and lots of gold bar pictures too! This is my favorite article, it was great fun and has the best pictures.
Database Confirms Max Keiser Guest Statement - Dominic Frisby tells a story about the head of gold at HSBC saying he never sees any bars with 'a Chinese stamp' on them. We check the database to see whether this statement holds true from the perspective of the millions of data points we have on record.
GLD ETF Mechanics 101 (Updated x2) - Whilst downloading the bar list from GLD, we discover a fax which describes the de-allocation process in the HSBC GLD vault. This is a great example of the bullion banking process in actual form.
Chasing Chinese Bars - the great hunt continues! In this article, we look at Julius Baer data and find no chinese-refined gold bars. Where are they? We look at possible explanations, as well as qualify what makes a good (or bad) bar list as a document source.
Got Any Leased Bars? - if you suspect you may have ownership of a bar which is leased, why not contact us and we'll check the database to see if it shows up (as a double claim of ownership) in any ETF. All information kept confidential of course, but the general idea is that your evidence combined with ours may give you 15 minutes of precious metals fame as well as monies from court cases or whatever.
A Quick GLD Dark Bullion Post - We start looking at how much returned Bullion is in the GLD additions. This is a follow-up to the SLV Database 5 article which outlined all the principles of the terminology involved with the dark bullion measurements. It was intended to be a sequel to the original observation of dark bullion in GLD, titled "An example of the Dark Bullion Effect (GLD)".
Eric Sprott's Third Bar List - Sprott's documents are starting to get a little more frequent - we look at the stuff no one else is looking at : the origin of all his new metal added to the funds. We discover the reason why no one else is looking at it - there is nothing of interest in the data - he sourced most of it from Canada.
Palladium, Platinum and Vodafone (original title) - We finally found some vault jumpers in Eric's funds, with the newly released bar lists for Palladium and Platinum. What does this mean for the Sprott in relation to other funds when he uses a common custodian for the metal? We explore, with HARD DATA.
Screwtape Looks at Silver Vault Inventories
- We investigate the SLV data to find a sustained movement of silver
going to the Brinks vault over a period of 10 months (nearly 100 million
ounces). The results only add more frustration to the 'silver shortage /
silver isn't there' meme set - is it really still credible that the silver was not real? Judge for yourself ...
Fed Gold and German Tanks - An innocent National Geographic documentary shows some gold bars, we get a glimpse of some serial numbers and make an educated guess about the bars we can't see, combining this information with the database run on the 6-digit Johnson Matthey bars to come up with some interesting visuals (first time we've used our custom charting software) and zero support for ideas of metal shortages. Anyway, why is the FED getting gold delivered?
Deep Patterns Analysis using Advanced AI (TBD)
-
Phase 5. End Game (~2070)
Hand over the project to assistant AI to continue the work.
The goals of the database are:
(A) To give me experience in handling large data volumes and mathematical processing, as the basis for another (non-metals related) project.
(B) To prove or create a reasonable set of arguments for:
(ii) ... whether the contents of the ETF's are real or fake (i.e. Tunsten or Molybdenum), by looking at refiner production runs and metals origins. (i.e. address Level 2 arguments).
(iii) ... the existence (or not) of the supposed 'Dark Pool' bullion of the OTC market, which we surmise is used by the bullion banks to affect price movements.
Software Used:
- Microsoft SQL Server
- Custom-Built Software (C#)
- PdfPig Nuget Library (for PDF extraction)
- Windows Azure Storage (cloud archive and indexing)
Accessing the Historical Archives: The original SLV PDF file archives are preserved in cloud storage but are quite huge. To save bandwidth costs these will be compressed before I make them available to the public (but if anyone is interested in the data please email me). Clicking on each link gives a list of the files inside that particular cloud container - use the URL displayed to download the list resource.
- Perth Mint Pool Allocated Silver: 2011, 2012
- Sprott Physical Silver & Gold Combined: 2010, 2011, 2012
- BullionVault (all metals): 2011, 2012, 2013, 2014
- Goldmoney (graphical): 2001, 2002, 2003
Goldmoney (PDF): 2004, 2005, 2006, 2007, 2008, 2009, 2010, 2011, 2012 - ETF Securities: 2011, 2012, 2013, 2014
- SLV iShares: 2007 (only 1), 2010 (last three months), 2011, 2012, 2013, 2014
- GLD SPDR Gold Trust:
- Partial records: 2004, 2005, 2006, 2009
- Daily Records: 2011 (half year only), 2012, 2013, 2014
 |
| "Nocturnal" |
The Earliest Known GLD Bar Lists - We only started collecting in 2011, but though some sleuthing we managed to track down some early lists. These provide a great reference point for our tracking and there are even some bars which are shown to have been in the vault since 20-November 2004!
How can We Trust the Bar List Data? When people say they don't trust the bar list data, what excactly does that actually mean? This is my first attempt to give a breakdown of argument in an effort to improve the resolution of questions - and end up explaining what questions CANNOT be answered using the database. I consider this the most important article of this series because it establishes a proper framework for investigation of how the data will square off against people's belief systems.
SLV Database 1 - Original premise and basis for the bullion bars project, including some basic analysis of the SLV data and ideas for the database.
SLV Database 2 - Basic analysis of SLV data to demonstrate (matched by Bar records) that some of the silver removed from SLV showed up again in subsequent 'add' events (first evidence of dark bullion).
SLV Database 3 - The most polished analysis which proves in great detail that over one quarter of the silver in the Perth Mint Allocated Pool Silver used to exist in SLV, and takes the same analysis to show that none of Sprott's PSLV bars used to exist in SLV. Discusses the implications from this.
SLV Database 4 - Another in-depth data exercise which compares the bar list data from Goldmoney and Bullionvault to conclude that some SLV bars jumped vaults. Includes a discussion of the different types of 'Vault Jumpers' as well as the implications, given that most results are pointing in one specific direction. Bad news for the 'SLV is Empty' crowd, since the amount of heavy research they must debunk, is growing substantially!
SLV Database 5 - Dark Bullion - All our investigations to date show a common element. This post dives straight into the definitions of how we can look at the warehoused data, to get more clarification on the volume of physical silver that sits behind SLV. The definitions and the chart required over 6-8 months of preparation and custom-created software and is our most sophisticated post yet.
Zero Hedge ZJ6752 - Bob Pisani takes a tour of the GLD vault and the bar he shows to the camera is not registered in the GLD inventory. Conspiracy? Not so fast. Because we have huge gobs of data available, we found the bar reasonably quickly and decided that the spin from the article was a storm in a teacup.
Bob Pisani Revisited (a fresh look) - Over a year later, we use our big database to successfully vindicate Bob Pisani's September 2011 documentary by identifying 10 new bars held in GLD from the video clip. Digital forensics at its best, and lots of gold bar pictures too! This is my favorite article, it was great fun and has the best pictures.
Database Confirms Max Keiser Guest Statement - Dominic Frisby tells a story about the head of gold at HSBC saying he never sees any bars with 'a Chinese stamp' on them. We check the database to see whether this statement holds true from the perspective of the millions of data points we have on record.
GLD ETF Mechanics 101 (Updated x2) - Whilst downloading the bar list from GLD, we discover a fax which describes the de-allocation process in the HSBC GLD vault. This is a great example of the bullion banking process in actual form.
Chasing Chinese Bars - the great hunt continues! In this article, we look at Julius Baer data and find no chinese-refined gold bars. Where are they? We look at possible explanations, as well as qualify what makes a good (or bad) bar list as a document source.
Got Any Leased Bars? - if you suspect you may have ownership of a bar which is leased, why not contact us and we'll check the database to see if it shows up (as a double claim of ownership) in any ETF. All information kept confidential of course, but the general idea is that your evidence combined with ours may give you 15 minutes of precious metals fame as well as monies from court cases or whatever.
A Quick GLD Dark Bullion Post - We start looking at how much returned Bullion is in the GLD additions. This is a follow-up to the SLV Database 5 article which outlined all the principles of the terminology involved with the dark bullion measurements. It was intended to be a sequel to the original observation of dark bullion in GLD, titled "An example of the Dark Bullion Effect (GLD)".
 |
| "Sprottwatch" |
Eric Sprott's Third Bar List - Sprott's documents are starting to get a little more frequent - we look at the stuff no one else is looking at : the origin of all his new metal added to the funds. We discover the reason why no one else is looking at it - there is nothing of interest in the data - he sourced most of it from Canada.
Palladium, Platinum and Vodafone (original title) - We finally found some vault jumpers in Eric's funds, with the newly released bar lists for Palladium and Platinum. What does this mean for the Sprott in relation to other funds when he uses a common custodian for the metal? We explore, with HARD DATA.
 |
| "Screwtales" |
Fed Gold and German Tanks - An innocent National Geographic documentary shows some gold bars, we get a glimpse of some serial numbers and make an educated guess about the bars we can't see, combining this information with the database run on the 6-digit Johnson Matthey bars to come up with some interesting visuals (first time we've used our custom charting software) and zero support for ideas of metal shortages. Anyway, why is the FED getting gold delivered?
---------------------------- putting it all together:
Bar Signatures Discussion 1
A Different View of GLD (Part 1 of 4) -
A Different View of GLD (Part 2 of 4) -
A Different View of GLD (Part 3 of 4) -
A Different View of GLD (Part 3 of 4) -
A Different View of GLD (Part 1 of 4) -
A Different View of GLD (Part 2 of 4) -
A Different View of GLD (Part 3 of 4) -
A Different View of GLD (Part 3 of 4) -
Cool stuff:
This animation shows pallet movement within the vault, it also proved some gold stays in the vault even though it gets de-allocated from GLD. [TODO: cleanup this reference ]
https://bullionbars.blob.core.windows.net/charts/20140211_gld_rough_animation_9x9_square.wmv
Bibliography: I am using the following books as reference:
- Gold Refiners & Bars Worldwide (1991) by Nigel Desebrock [Grendon Research International] link
- GLD History History Book (2012) by Timothy Green [LBMA] link
Questions? Comments? Criticisms? Scorn, Threats, Inquiries, etc. Email: bullionbars at hotmail.com
8 comments:
Warren,
I suppose you have the year for every bar in your data base. Concerning gold LGD bars, I wonder whether you can analyze the flow of very old bars into the market. For example, set a threshold of 1944 (Bretton Woods). All the Western CB gold should have been cast before that year.
Now I'd be interested in how the number of these bars that are visible to you changes, i.e. the rate at which new bars of that age show up.
I'd love to have your data base going back to the 1990s, but that's obviously wishful thinking as there were no ETFs with published bar lists at that time. I'd guess that lots of old bars appeared in the market until 1999, but much fewer since then. Still, cann you see spikes of old bars coming in at certain points?
Victor
Hi VTC,
For some bars the year of manufacture is known (it's part of the serial number) but generally refiners like to obfuscate their production quantities by making the number abstract. As Bron has said - it's a competitive business and making their yearly production easily accessible is not in their interest.
Having said that, some of these naming rules are known and the rest can be inferred based on appearance, but indexing the bars by year takes some time so it's not done currently. There are plans to do this however, since it's a good test for validating potential 'fake' production runs. For the ones we can't determine we have to come up with a way of recording the date range & certainty, along with an index of any relevant supporting documentation.
I'll add this as a metric of interest - 'new appearances' are a specific category we track and flag, hence what you want should be a matter of an sql query once the year index is populated. It may not happen in time for your research though - I'm presently swamped.
Old bars do exist in today's ETF's - for example Bron spotted that in GLD there are some bars branded from the Perth Branch of the Royal Mint (which later became Perth Mint), the fineness indicates the date of manufacture anywhere between 1928 and 1970.
Stay tuned. :)
Hi,
Not sure which physical funds your database covers. I could not see when I read various comments. I have various links to physical bar lists, I can send you if need be. Could not see any email address via which to contact you guys.
Like this one
http://www.bmgbullion.com/doc_bin/goldbarlist.pdf
Hi Manly, good point - have added a contact email address (bullionbars at hotmail.com), please get in touch and we'll compare notes. BMG Bullion haven't updated their list since March 2012, we only started recording that one in Dec 2011 (we have their silver and platinum list as well).
Regards,
Warren,
on 20 February 2013, GLD had a net loss of inventory of about 20 tons.
Can you take a look at your data base and see which exact amount of gold was allocated to GLD and which was deallocated from GLD (there might be some bars added and others removed for a net loss of inventory of 20 tons).
Victor
Hi Guys, maybe you are interested in a Danish bar list (that doesn't comply to industry standards but still).
https://www.gainesvillecoins.com/blog/denmark-releases-gold-bar-list
Hey Jan, thanks for that - I've added the Danish bar list to the download routine.
I should provide an update for this project since it's been a few years now. Throughout this entire time I have been downloading GLD and SLV documents to the point where I have many gigabytes to sift through. There are a few gaps where occasionally the document location changed and my code didn't raise the error, but I still have most of the data (now about 10 years worth).
As time progresses the internal structure of the documents change and each time I have to adjust the processing code to cater for the new document without errors. Life events have gotten in the way so the development of the filters fell by the wayside - I plan to just keep growing the archive and keep it safe from digital rot and later this decade I should have human or Artificial Intelligence assistants to help flesh out the catalogue and process every document. It is already possible to use machine learning to conduct probability calculations on bar signature matches - I have been learning how to use agents and neural nets in my programming and I've realised it's quite powerful - even to the point where we could create agents to trawl the internet for gold bar photos, recognise bar numbers and automatically include those photos in the index.
Most of the bar lists in my archive are stored in the Microsoft Azure cloud with Geo-redundant replication. My intent in the long-term is to expose the catalogue as a public resource but this will become more useful when the indexing is fully structured - I don't currently have the resources to build out this project but I'm working hard to make it happen. When the world does transition to freegold (or equivalent) I think the bar list archive will have some value to the public.
That sounds very promising. Tx!
Post a Comment