< previously, I talk about storage and processing matters >
Nothing that annoys me more than being misrepresented, so I thought I would just put this snippet into the public record where my intentions can be transparent, if I need to call on it. The bars list project has been an interesting journey - it goes where the data takes us. For myself it took about two years to finally articulate the different levels of argument.
First I just want to be clear about why I'm doing this analysis.
Originally I started it as part of a due-diligence exercise but it grew
into wanting to really sort the wheat from the chaff as far as bullion
stories go. The internet has enabled the mass distribution of ideas (often direct copies) and for
better or for worse it has amplified the signal-to-noise ratio, to the
point where anyone trying to understand precious metals using purely
online research faces a tough job - simply because search engine optimization will automatically direct you to (say) tfmetalsreport, kwn or similar.
My theory is that if the banks are
doing naughty things then the raw data has potential to expose that,
depending on how clever we are with the questions. The one myth which appears to be slain is the 'silver shortage' story (we see no evidence of that). It is unfortunate the
exercise to date has only revealed the BORING mechanics of bullion shipping and movement. Occasionally a good question comes across my desk which can be answered from the database and usually that finds its way into an article.
The reasons for writing the article of course, are two-fold. The first is that it provides me a conduit for focusing the analysis (i.e. I have to correctly parse the argument and material for presentation as well as check the figures) and secondly it provides a venue for discussion - others can weigh in with their opinion of what I have written and hence refine the analysis.
For some reason, the comments section of many of my database articles have been either the base for silver propaganda (as was the case with ultranerd in SLV Database #2), or as a platform for freegold bashing, thanks to folk like Gary and SMRI. Since I have put a fair amount of effort into the writeups, any forum sliding upsets me greatly. Perhaps one day we will have a proper freegold forum, but the color and the shape looks really different to what these folk thinks it looks like (oh well).
At time of writing, this is the total amount of data (database tables). Most of it is duplicated information, but nonetheless it's all data points.
dbo.WAREHOUSE_SILVER > 155 million rows
dbo.WAREHOUSE_GOLD > 50 million rows
dbo.Documents (original sources) = 8773
Friday, 12th April 2013
Creating new pages to continue this internal monologue seems a bit pointless, so my new idea is just to put it all into a big page, with snippets of diary entry. Again, the point of writing this side is simply to provide a backdrop against which the bar list story is being discovered. I don't care whether anyone reads it or not, however for someone out there tracing my steps it may seem interesting (or not).
I've just spent a month in Singapore being overworked on a project that my day job wanted done. It's been a nice break from precious metals but at the same time it has delayed the 'SLV Database 5' article that is written, just waiting on the charts from the data.
I feel somewhat guilty now, having done all the research which allows me a deeper look into the world of bullion, to not yet publishing the results. It's a kind of zen feeling that so far is somewhat private, while everyone else tries to make head or tail of the various ramblings that we see on the internet about what silver is doing or is supposed to do, naked shorts and all the crap which taken at face value seems to be precious metals commentary but if we are really honest with ourselves then we should see it for what it is, as expertly put by Bron Suchecki: http://goldchat.blogspot.com.au/2013/03/precious-metal-commentary-propaganda.html. Anyway, I feel a slight burden to share my new bits of knowledge with the world but at the same time feeling very comfortable that the research I have done, has catapulted my knowledge well beyond what I ever expected. In the background I am honored to be on an ongoing background discussion thread led by Kid Dynamite where I get some very interesting insights into market formation and stuff surrounding the metals.
While I've been away, a cloud computer has done the work of daily downloading the different source documents (bar list documents) that we use. When I get home I'm going to have to take all those files (there's about 500 of them) and start feeding them into the database processing system - the process is not yet 100% automated, but I'm also wrestling with the size of the database which keeps growing and will continue to do so. The other month I had an unfortunate incident where the database went offline and I had to run a bit of DBCC (database consistency checks) which takes ages and is not very well documented ... the tools are quite scary, if you're lucky it will be able to repair the index entries and tell you it fixed everything. If you're unlucky it will just say 'your data is gone, you are Royally Screwed unless you had a backup - you do have a backup don't you?.' Fortunately for me the corruption was only in a table which was a copy of other data. Anyway, if everything goes haywire, I can actually reprocess the entire archive of 10,000+ original source documents, however the time it would take to do all the processing is substantial and I'm not sure that my code would cope properly with the 'changes to document formats over time' aspect, since my code doing the processing always generally reflects the current modern configuration, and processing the old crappy documents would be a real pain.
The work I've done with the Very Large Database Management is a precursor for my next project which will involve solar energy. The data generated may be so large that I'll probably need to use Azure table storage for this one. The software support for 'big data' is evolving slowly and improving slowly, soon it will be state of the art.
I am generally interested to start comparing the last set of GLD and SLV lists. A lot of people in the metalsphere seem to be fixated on the falling inventories of GLD, and rightly so, since it's a big chunk of gold available to the market. If FOFOA is correct in his theories then the declining inventory and accompanying falling POG is an indicator of a precursor to Freegold (or something like that). I am reasonably sure that the bar list data contains some clues as to what is happening, but despite having the framework to perform analysis, the really tricky part is (still) about what questions to ask. The bar lists are effectively the 'bank statements' of the Bullion Banks, and I'm still amazed at the fact they are public (they need to be), and I'm still looking forward to the next set of data mining we can do on them, if only I can find the time to dedicate to it. My biggest problem is that because I have many skills, everyone wants a piece of me, and I actually have to fight hard for time dedicated to 'me'. This is actually my biggest bone of contention with the whole monetary system - with a single keystroke, some random person in some country in the world can create enough monetary units to commandeer my entire life of productive output several times over. In this sense, whilst-ever I am laboring in a social contract just to get control of those same monetary units, then they have me under control, good and proper. The same goes for any of you out there - the money masters have you in their grip and it's a very lucrative game for them, whilst ever they can do this and whilst-ever you agree to the arrangement. If you view this arrangement and take away the dollars component (i.e. viewing just the social contract arrangement irrespective of dollars) then it's sheer madness. The 'financial matrix' of which FOFOA spake, is very real. Breaking away from it is difficult because it's so pervasive and basically dominates our entire consciousness. Perhaps only by changing our thinking will we be able to break away from it.
Anyway, hopefully shortly I'll finish the charts on the dark bullion article and then we'll see what can be found there. My plan is still to publish the results for SLV, but keep the GLD results to myself (and a handful of selected people with whom I will discuss it).
Regards,
Warren James
Screwtape Files
Friday, 7th June 2013
So I managed to finish the SLV Database 5 article; a brilliant success of data patterns analysis involving charting some 200 million data points, but very little response or comment. What's even annoying is that the two GLD articles I wrote around it seem to have gotten more attention (if page views are anything to go by). I did try to discourage comment on it, which worked in general because there was very little of the bickering previously infesting the comments section on our blog, but it may also have killed the appetite for some other comments. Anyway, we did get some good discussion from it and now it's old hat I suppose except for the fact that the dark bullion measure gets adjusted every time we get new data so new charts will be somewhat regular and it should be able to show us new things.
I'm having to go back into recalibrating the downloads because I've realized that I've lost some data for ETF securities over the last couple of months - they changed their spreadsheet structure and it turns out that my program just ignored the data and then deleted it (oops). Crap, so I've had to spend some time fixing that up - I've now realised that nearly all data can be really important so now I'm on a mission to try and get as much as humanly possible.
I'm going to have to set a time limit on this - I figure that if there isn't a financial trading signal somewhere in the data in the next four months then I'm going to have to ditch the thing and concentrate on my other projects which are now being crowded out. The recent article and ease of which is provided by the new dark bullion matrix, really bumped things up into a new level. What I mean is that all the secrets that the data holds can surely be mine if I just get the 'next layer' off, and what a deep layer it's turning out to be - we just seem to generate more questions but they are of complex nature and it's great to be at this point - kind of fun in a stupid obsessive kind of way.
Work is still giving me lots of strife and it would be nice to break away from that.
Anyway, I've got to get back to it. I figure there's a pattern in the physical order of the spreadsheet sources and I'm going to try fish it out. But I have to sort out the downloading stuff first. Very slowly I'm turning the cloud system into a big auto download and process center. Soon it should be able to warehouse the data automatically with no input from me at all (i.e. no coaxing or hand holding).
I had to redo my computer - put Windows 7 back on, can't stand Windows 8.
Regards,
Warren
Thursday, 4th July 2013
So we managed to offend SugarLover and Duggo by relocating their comments and implementing a 'Sin Bin'. I feel sorry for them that they lost the rights to play their little games, but basically I liken it to the following: My dog has freedom of expression in the sense that he lives in the house with us as well as outside. If he takes a crap in one of the rooms then we will clean it up by default because we also live in this space and have a preference for no dog shit on the floor. If he craps on the lawn, we normally leave it there, since the birds get at it and it becomes part of the garden flora and fauna - part of nature. So we're not limiting the dog from doing what he does naturally, but we have automatic (and established) rules for whatever is necessary for making the house function normally. I think the strange thing would be if we DIDN'T do anything about it.
The irony is that for a number of weeks we had let it ride, before we (all) finally got pushed over a particular pain threshold. Worst of all were some of the Freegold people themselves, who (truth be said) were just as offensive as some of the anti-FOFOA stuff we saw written on the blog. In addition to that, there was some fake identities being used. Don't even get me started on Gary's persistent promotion of his 'Time Proven' sock puppet (the latest incarnation of "Fake Another"). I'm slightly insulted that he thinks we're that stupid. Regardless, the Freegold debate page now has a really good rolling structure, which I'm proud of, and I think will help reveal / distill the good points of any freegold discussions. Finally, because I know that this text is not read by many people and by the time people find it, it won't matter, Duggo, Suger Lover, get your own fucking blog. Yep, this is a common internet terminology: GYOFB, or just GYOB (the non-profane version). We've worked really hard to make this site a great place to hangout and discuss things, and I'll be damned if you think you can come in here and hijack the comments thread. The truth is that we don't particularly wish to discuss the topic any more, and yes, we will force the issue it if we so desire, it is our right. Hopefully the new dedicated freegold page format will help meet everyone's goals.
Right, moving on. Recently I re-evaluated my reasons for writing and discovered that I've actually met most of my goals in relation to why I started writing in the first place. As an artist, I pay a lot of attention to this aspect. However, currently we haven't found the patterns we want from the bar list data, I think that's just a time thing - i.e.work has ripped out my soul and I have really only been in maintenance mode for the database the last number of months. I will however, do my best for one last push to map out the exhaustive set of things that we can find. With the help of Costata, I recently did outline all the areas that we can still look at, and this will form the basis for the last set of articles that I will write, and this may indicate an exit time for me because I will move onto my next project (Solar Power). I really look forward to the future and I think that humankind has a bright future ahead of it. Least of all, don't forget that last week NASA determined that the sun is going to make this planet uninhabitable in about a billion years or so. That thought may put our current daily strivings into perspective. Imagine a time-lapse image of this where the sun finally pops like a bursting balloon and wipes out most of its planets. Life will go on in some form or another, but very little of this discussion about gold we're having now will be relevant to anything.
Regards,
Warren :)
22nd July 2013
My better half insisted I listen to something posted on Facebook - a prank call to a car company. Reverse engineering is a hobby of mine so naturally I got berated after
pointing out she just spent the better part of 1½ minutes staring at a
huge Mercedes Benz logo, while a perfectly scripted, perfectly recorded
conversation played out under the guise of 'lol, must listen to this'.
Being oblivious to guided machinations seems to be a desperately important global trait -
otherwise how would we come to grips with the realization that the
'Would I Lie to You' show is largely scripted; we prefer the illusion it
is all spontaneous wit and to be sure, life would otherwise lack a lot
of color.
In my research we found out about an internet company called 9thsphere.com (in Canada) who are linked with sprott.com. The firm also does social media advertising. I wonder how much of Sprott's budget goes to that. I figure I'll never really know, but I would really love to talk to any ex-sprott employees web people and find out.
Ok, back to number crunching. Work still has my soul @ the minute.
No comments:
Post a Comment