 |
| A typical defrag program - showing data allocations on a computer file system. The software would remove blocks from one location and write them in another. Some files did not move at all (and you could see which blocks stayed). |
 |
| Famous shot, from articles: ZJ6752 (original) and also my follow-up. Bob Pisani acheived something neither you or I could manage - getting television cameras inside the HSBC GLD vault. |
1. All bars get represented on the chart, in order of appearance in the spreadsheet. If a bar is new to GLD we add it to the next available 'slot', which becomes a fixed position for that signature. If the bar disappears from the inventory we keep the space there but remove the color. If the same bar comes back, we recolor the pixel again. Similar to the defrag software, we'll show change over time so the animated chart uses one GLD source document per frame.
2. We know that gold tends to move around in pallets of 80 bars (1 tonne), so we'll clump the bars together in blocks of 80 (i.e. approximately 8 x 10 pixels ) in an attempt to show up the pallet movements.
3. The fundamental color for a pixel representing a gold bar is Yellow . Current bars being Removed or Added will be shown Magneta and Cyan respectively, which indicate 'current activity'. So, in the basic chart animation, a bar being removed is indicated by a flash of magneta then becomes white in the next frame to show it was removed; likewise a bar being added shows as Cyan and in the next frame it becomes a yellow pixel to indicate it is now on the ledger.
Those conditions make the following animated chart. You may wish to switch to the high-res version to zoom in and see the detail and the dates. The video starts from mid-2011 which is when we started collecting the GLD bar lists. Please bear in mind there is limit of reading error with this chart - I estimate it could be as high as 5 %, but I'm constantly refining the techniques and using the results to scrub the data!
Other formats available:
- Original WMV file (slightly better quality - Windows format, 8 mb).
- Original Source Image Files used for animation (zip file with PNG images, 24 mb)
Please feel free to ask questions in the comments but bear in mind this is the 'Basic Version' defrag chart. For those of you who have already seen this chart, I am actively working on a more advanced version - the next article will extend the concept substantially: "A different view of GLD (part 3 of 4)". Finally just to comment that this chart succeeds better than I hoped in terms of telling a unique story - it is really neat seeing the pallets move about.
regards,
Warren James
Screwtape Files
14 comments:
Got some initial observations on my blog http://goldchat.blogspot.com.au/2014/03/gld-vault-defragmentation.html republished below:
1. During the redemptions in 2013, most of the bars are taken from the bottom, that is, the recently added stuff. Makes sense, this stuff is easier to access.
2. But note much of the redemptions are from all over the place. Some of that is explained by them "hunting" for 9999 bars as Warren discussed in emails. He will have a follow up post taking this animation analysis into more detail showing this.
3. The area just above the empty bottom area is really stubborn, something about those bars they don't redeem from, even though they are more recently added than the bars in the first half of the pic above that section.
Hey Bron, that goes into an area I'm really interested in - i.e. the question of 'what is the basis of pallet removal'? There are some really interesting patterns in the advanced version of the chart that suggest it's not just random.
I will make a note that bar signature changes do sometimes show as a 'strange removal', and as such I'm trying to iron these small errors out of the data so that the picture is clearer.
Specifically, I want to know which pallets leave the vault 'for good', which ones don't, and formulate some reasons around why that occurs. It's not easy to construct the argument, but we definitely have a good set of definitions these days. The next chart will show the dark bullion effect on this inventory, which is even more interesting.
This is really cool!
I wonder if a visual pattern would emerge if the pixels were re-arranged just so. I'm thinking along the lines of a periodic table of the GLD storeroom. Maybe a pattern would emerge based on how tall the pallets are stacked and in which room they are placed.
Thank you for this.
I think it is hard to judge by eye because of the big 'holes' in the stock of bars remaining. LIFO is obvious for logistical reasons, quality etc.
It was interesting to see this though as it was, and it will also be interesting if in the advanced version there is an actual 'smart' defragmentation done on the basis of longevity of bars in the vault.
Warren,
this project is getting more and more impressive.
Do I see people checking in their coats at the bottom, then quite some time later other people picking up theirs at random locations in the mess at the top?
There are also some arbitrageurs adding a strip of 10 to 20 coats at the bottom, then shortly afterwards, removing precisely those again.
Fantastic !
Victor
Note that as Warren only has mid 2011 as his starting point, the order of the bars comes from the order they are listed in that first bar list.
This order most likely does not related to the actual order the bars were added up to mid-2011. Therefore redemptions from that early stuff may appear all over the place because they are not reflecting actual first in order. I would guess if we had all the bar lists the redemptions would be more contiguous or show a more clearer LIFO pattern and more blocking.
@Victor, thanks. Yes, we get closer to being able to answer questions like that - essentially we can now 'palletize' the entire inventory so it's now a case of working out what other discoverable elements make a difference to the interpretation.
@Jesse, thanks for the idea on the advanced version. 'longetivity' is a dimension we have - I would just have to figure out how to incorporate that in a chart.
@Michael H, we do have some visibility on 'stacks' and we touch on this concept in the next article. It gets tricky because the information resolution is still too low to bring hard conclusions to the table, but it definitely shows some information about physical location as well as the different types of vault activity.
Also a quick note, when viewing this chart don't forget that whenever you see 'cyan additions' in the middle of the inventory - that is dark bullion returning and it's possible that metal never left the vault. The next chart highlights this a lot better. What I'm heading towards is a probability map of how much metal is still in the vault regardless of how much gets removed from the inventory - I hope we'll have that cracked in less than a year.
Hi Warren,
I second VtC's comment: "this project is getting more and more impressive."
However, rather than supporting the "coat-check room" view I think your data mining reinforces the argument that the gold market is, in fact, two markets held together by one price. GLD, the Fed's gold market activity and the rest of the LBMA apparatus appear to sit at the crossroads of the paper and physical markets.
This concept of separate markets was reinforced for me during that phase of redemption last year by the APs, the buyers of last resort for the GLD ETF shares. Those redeemed shares were "no bid". The right to acquire physical gold at a discount could not compete with the liquidity of the tangible physical product (presumably mainly in the form of kilo bars).
While the rout was playing out in the gold ETFs a large addition to the supply/flow of physical gold was easily absorbed elsewhere. To underscore the strength of demand this additional flow was absorbed without the assistance of some of the largest buyers of physical gold in recent years - GLD and the other ETFs.
Cheers
Warren, absolutely brilliant work. I look forward to part 3.
I wonder what we'd see if you were to give super sticky gold -- that is the Hotel California gold that can check in anytime it likes, but it can never leave -- its own color. I have no idea what insight it might provide, but sometimes along the discovery process I find you just have to ask random questions of the data and see what answers you get.
Raxco uses colours, and stacking in their fragmentation method on Perfect disk.
Files over a certain duration in age are blue, the intermediate in orange, and the newest in green. They tend to order them this way on the disk and fill up the blank areas. I think this is so that the disk will tend to be written in more compact areas, decreasing head search time.
This is why you do not see this type of defrag used on SSD's. Although they do have the space compression defrag. SSD's have no read/write head lag time.
@Jesse, I had a crack at ordering the results by the 'age' element - it's really difficult. Consider a bar which gets added early in 2011 and then leaves the inventory for good 4 days later ... which (x,y) dimension should that bar use on the graph? What will work though, is a color overlay (tint) on the stock existing ... i.e. as a bar in the inventory 'ages' it can take on a different hue - although technically this is already shown in the current chart - the stuff at the top which is still visible, is the oldest, the stuff at the bottom is the youngest. Anyone can feel free to email me bullionbars@hotmail.com if you have a technique which can make that work - I'm stumped.
@Aaron thanks - that's a decent idea although it's tricky because we only have two points we can really measure: when a bar appears and when it disappears. So identifying any 'can't leave' gold is abitrary depending on where you measure it from - and unfortunately we only have about 10 bar lists from pre-2011. What I can tell you though, is that it's the new Johnson Matthey bars which are most likely to 'move around' and the older vintage bars tend to move less. We're exploring that as a way of building a theory for 'basis of selection' and it's possible that a different view of the defrag chart (by refiner) may help show this predisposition. I'll see if I can add that to the set of charts in this next article.
Wow. Cobwebs growin' in here.
Fwiw, my dirt simple model says gold closed the week beneath it's 200 day, so get the hell out. No If's, And's, or But's.
And silver or miners? Even worse. As usual.
Thank you For Sharing!
Post a Comment