Hi,
I'm a first time poster, long time lurker.

I would like some advice on where to start at automatically ingesting
stats and how most of you are collecting and storing data. Flat files,
relational databases, excel files? Are you screen scraping, or are
there RSS feeds or spreadsheets publicly available that make life easier?

Currently, all of my stats are entered by hand into a relational DB,
which makes the end of the season rather busy, so I would really like
to start automating some of the bigger leagues if I can. Luckily, I
have programming skills, so it shouldn't be too intimidating, but I
just don't know where to start. I same some PBP files mentioned. Where
do these come from? Finally, I noticed that hockey DB appears to have
an agreement with PointStreak for some leagues. I suppose he has a
feed of some sort because he is the well known hockeyeb.com. Are there
such services available to the public?

Any advice is greatly appreciated.

James
www.losthockey.com

I apologize for the off-topic post, but...

If anyone is interested in starting a similar project for basketball,
I have a basketball database that I can make available.  The database
download from databasebasketball.com (the 2.0 version) was used as a
starting point, but my current version differs significantly from the
original:

- Updated through 2006-07 (except for draft data, which is only
current through 2004)
- Many corrections and additions, as the original database had LOTS of
errors and omissions.  For example, several hundred missing player
playoff records were added, many missing splits for multi-team players
were added, and a number of statistical errors have been fixed.
- Player and team IDs have been modified to correspond to
basketball-reference.com (a few player IDs may still differ)
- Lots of other changes - missing data now represented by nulls
instead of zeros, added proper coach IDs, added additional fields to
the team table, etc.

For a variety of reasons, I don't have the interest in running a
basketball database project myself.  So if anyone is interested in
taking this on, please contact me offline.

Doug

The Sports-Reference team has launched their hockey site today
(www.hockey-reference.com).  According to the site blog, they obtained
their data through a licensing deal with Dan Diamond and Associates.
The data is essentially NHL only at this point; they have data for
minor leagues and other major leagues, but only for players who played
in the NHL (so for example, there are no WHA standings or season
summaries).

Doug

I just uploaded a zip file (2008preview.zip) with a preliminary
version of the 2007-08 update.  This doesn't include all of the tables
that are in a normal release (and I left out the readme and notes
files), but if anyone is anxious to get their hands on 2007-08 regular
season data, this should get you by until the full update is released
(probably in June).

Note that the Master table has a bunch of new fields - height, weight,
shooting hand, three additional name fields, and six fields for death
information.

Doug

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the hockey-databank
group.

   File        : /hdb-2008-06-14.zip
   Uploaded by : dsreyn <dreynolds@...>
   Description : 2007-08 update

You can access this file at the URL:
http://groups.yahoo.com/group/hockey-databank/files/hdb-2008-06-14.zip

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/l/us/yahoo/groups/original/members/web/index.htmlfiles

Regards,

dsreyn <dreynolds@...>

Changes in the new release are as follows:

* 2007-08 update
* New fields in the Master table - name note, given name, nickname,
height, weight, shooting (or catching) hand, date and place of death
* The team IDs for the Vancouver Millionaires / Maroons have changed
to be consistent with NHL game results files from the Hockey Summary
Project
* Added data for the NHA playoff series of 1913-14, 1914-15, and 1916-17
* Added All-Rookie teams to the Awards table.  Also added Plus-Minus
award winners from 1982-83 through 1995-96; this award has gone by
several names, so it is now labeled "Plus-Minus" in the table.
* Added data for the NHA, PCHA, and WCHL to the TeamSplits and
TeamVsTeam tables
* And of course, various corrections

Doug

I found a small problem in the latest release - I had the wrong ID for
Evgeni Nabokov in the AwardsPlayers table.  Rather than post the fix
as another new release, I have patched the 6-14-08 release.  There's
one other small change, noting the recent death of Ray Getliffe in the
Master table.

Doug

--- In hockey-databank@yahoogroups.com, "dsreyn" <dreynolds@...> wrote:
>
> I found a small problem in the latest release - I had the wrong ID for
> Evgeni Nabokov in the AwardsPlayers table.  Rather than post the fix
> as another new release, I have patched the 6-14-08 release.  There's
> one other small change, noting the recent death of Ray Getliffe in the
> Master table.
>
> Doug
>


also check Jack and John Keating "Master" data

Databank Master.csv:
playerID -- firstName -- lastName -- firstNHL -- lastNHL -- birthYear
keatija01 -- Jack -- Keating -- 1931 -- 1932 -- 1916
keatijo01 -- John -- Keating -- 1938 -- 1939 -- 1908

Databank  Scoring.csv:
keatija01,1938,1,DET,NHL,L,1,1,0,1,2,,,,,,,,,,,,,,,,,,,,
keatija01,1939,1,DET,NHL,L,10,2,0,2,2,,,,,,,,,,,,,,,,,,,,
keatijo01,1931,1,NYA,NHL,C,22,5,3,8,6,,,,,,,,,,,,,,,,,,,,
keatijo01,1932,1,NYA,NHL,C,13,0,2,2,11,,,,,,,,,,,,,,,,,,,,

Hi,

I was wondering about the rules for using your data.  I am working on
a little query tool that allows "ordinary" people to get some reports.
  Am I ok to use the files here?

To give you an idea of what I mean, I will show you the link here:

http://www.hockeyhunter.com

This is very much a "beta" so please bear that in mind.

Thanks,
Ryan

There are no restrictions, so there's no problem using the files.  In
fact, I've been hoping that some additional projects like yours might
develop from this.  An acknowledgement (which I see you have on the
page) is appreciated, of course, but not required.

Doug

--- In hockey-databank@yahoogroups.com, "the_warrior_king"
<RyanLind@...> wrote:
>
> Hi,
>
> I was wondering about the rules for using your data.  I am working on
> a little query tool that allows "ordinary" people to get some reports.
>  Am I ok to use the files here?
>
> To give you an idea of what I mean, I will show you the link here:
>
> http://www.hockeyhunter.com
>
> This is very much a "beta" so please bear that in mind.
>
> Thanks,
> Ryan
>

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the hockey-databank
group.

   File        : /hdb-2009-01-15.zip
   Uploaded by : dsreyn <dreynolds@...>
   Description : Hockey database, Jan. 15, 2009

You can access this file at the URL:
http://groups.yahoo.com/group/hockey-databank/files/hdb-2009-01-15.zip

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/l/us/yahoo/groups/original/members/web/index.htmlfiles

Regards,

dsreyn <dreynolds@...>

No major changes this time - mostly lots of data corrections and
updates.  There's also an additional field in the Master table giving
the hockey-reference.com ID's.

Doug

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the hockey-databank
group.

   File        : /draft.zip
   Uploaded by : dsreyn <dreynolds@...>
   Description : Draft data

You can access this file at the URL:
http://groups.yahoo.com/group/hockey-databank/files/draft.zip

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/l/us/yahoo/groups/original/members/web/index.htmlfiles

Regards,

dsreyn <dreynolds@...>

I have added a file with draft data in the files area.  This may be
rolled in with the regular data releases in the future, but for now
I'm keeping it separate.

Doug

Hi - This is a great resource, thanks for putting it together! I am starting a
bit of research on
how long hockey players live, change in lifespan over time, and comparisons to
the national
(canada) average.

A couple of questions about the database:

Where do date of death records come from? I am curious about how much confidence
we can
put in these dates.

Some DoD fields are marked "9999" and others are blank including people born in
the 1890s
and presumably gone now. Is there a distinction here?

Cheers,

Chris Fletcher

The death info has been cobbled together from multiple sources.
Generally speaking, any source I have used (for the entire project) is
listed in the "references and acknowledgements" part of the readme
file included with each release.  The death information specifically
has generally come from one of the following:

- Total Hockey
- hockey-reference.com
- Find A Grave (www.findagrave.com)
- SIHR (www.sihrhockey.org) - they've posted annual lists of
hockey-related deaths for the last several years
- LostHockey.com (www.losthockey.com)

I have occasionally added recent deaths (the last year or so) after
seeing news stories on ESPN.com, the hockhist Yahoo group, HFBoards,
etc.  Though of course, these generally show up in the annual SIHR
lists.  In case you're wondering, as a general rule I avoid using
Wikipedia for anything that I can't corroborate elsewhere.

As far as reliability of the information is concerned, I'm fairly
confident the death info is fairly reliable.  However, keep in mind
that consulting multiple sources will sometimes yield different dates
for the same player.  There's also one case in particular that's
probably wrong.  There are two players in the database named Jack
McDonald - one born in 1887, the other in 1921.  Total Hockey and
Hockey-reference.com give a date of death for *both* as 1/24/58.  Two
players with the same name, born 34 years apart, both dying on the
same date?  Seems like an incredible coincidence (though Adams and
Jefferson both died on the 50th anniversary of the Declaration of
Independence, so I suppose anything is possible) - I suspect the
likely explanation is a mixup given the identical names.

The "9999" entries indicate that a player is known to be deceased, but
no specific details are known.  I have only marked players as deceased
in this way if they are indicated as such either in Total Hockey, or
in the player career totals lists in the Official NHL Guide & Record
Book.  There are a number of other players who can be presumed to be
deceased given their birth year, but I have not marked them as
deceased without confirmation.

Doug

--- In hockey-databank@yahoogroups.com, "chris.fletcher66"
<cfmletcher@...> wrote:
>
> Hi - This is a great resource, thanks for putting it together! I am
starting a bit of research on
> how long hockey players live, change in lifespan over time, and
comparisons to the national
> (canada) average.
>
> A couple of questions about the database:
>
> Where do date of death records come from? I am curious about how
much confidence we can
> put in these dates.
>
> Some DoD fields are marked "9999" and others are blank including
people born in the 1890s
> and presumably gone now. Is there a distinction here?
>
> Cheers,
>
> Chris Fletcher
>

Thanks Doug - I missed the readme and notes files, should have checked them
first.

There are bizarre coincidences out there. As I started looking for data on this
topic I
clicked first on the 1922-23 Canadiens to find two players, Odie and Sprague
Cleghorn-
brothers, with my maternal grandparent's last name. They brothers Cleghorn died
within
days of each other and are buried in the same cemetery as my maternal
grandmother -
Sheena Cleghorn. Spooky. I will post updates on our project as we work through
it.

Best,
Chris

--- In hockey-databank@yahoogroups.com, "dsreyn" <dreynolds@...> wrote:
>
> The death info has been cobbled together from multiple sources.
> Generally speaking, any source I have used (for the entire project) is
> listed in the "references and acknowledgements" part of the readme
> file included with each release.  The death information specifically
> has generally come from one of the following:
>
> - Total Hockey
> - hockey-reference.com
> - Find A Grave (www.findagrave.com)
> - SIHR (www.sihrhockey.org) - they've posted annual lists of
> hockey-related deaths for the last several years
> - LostHockey.com (www.losthockey.com)
>
> I have occasionally added recent deaths (the last year or so) after
> seeing news stories on ESPN.com, the hockhist Yahoo group, HFBoards,
> etc.  Though of course, these generally show up in the annual SIHR
> lists.  In case you're wondering, as a general rule I avoid using
> Wikipedia for anything that I can't corroborate elsewhere.
>
> As far as reliability of the information is concerned, I'm fairly
> confident the death info is fairly reliable.  However, keep in mind
> that consulting multiple sources will sometimes yield different dates
> for the same player.  There's also one case in particular that's
> probably wrong.  There are two players in the database named Jack
> McDonald - one born in 1887, the other in 1921.  Total Hockey and
> Hockey-reference.com give a date of death for *both* as 1/24/58.  Two
> players with the same name, born 34 years apart, both dying on the
> same date?  Seems like an incredible coincidence (though Adams and
> Jefferson both died on the 50th anniversary of the Declaration of
> Independence, so I suppose anything is possible) - I suspect the
> likely explanation is a mixup given the identical names.
>
> The "9999" entries indicate that a player is known to be deceased, but
> no specific details are known.  I have only marked players as deceased
> in this way if they are indicated as such either in Total Hockey, or
> in the player career totals lists in the Official NHL Guide & Record
> Book.  There are a number of other players who can be presumed to be
> deceased given their birth year, but I have not marked them as
> deceased without confirmation.
>
> Doug
>
> --- In hockey-databank@yahoogroups.com, "chris.fletcher66"
> <cfmletcher@> wrote:
> >
> > Hi - This is a great resource, thanks for putting it together! I am
> starting a bit of research on
> > how long hockey players live, change in lifespan over time, and
> comparisons to the national
> > (canada) average.
> >
> > A couple of questions about the database:
> >
> > Where do date of death records come from? I am curious about how
> much confidence we can
> > put in these dates.
> >
> > Some DoD fields are marked "9999" and others are blank including
> people born in the 1890s
> > and presumably gone now. Is there a distinction here?
> >
> > Cheers,
> >
> > Chris Fletcher
> >
>

Hi Doug,

I have been studying the tables in hdb-2009-01-15.zip for normalization.

During this process, I noticed possible typos in table abbrev.csv.  Should the
fullname entries, with respect to codes AVC, AVF, AVSF, AVQF and AVPRE, be AVCO
instead of AVRO?

Thanks and excellent work.


Regards,

Ron

Oops...not sure how I messed that up, but thanks for pointing it out; that will
be fixed in the next release.  The next "official" release will come after the
playoffs have ended (probably right after the annual awards are announced), but
I will probably put up an interim release next week once I get the regular
season statistics entered.

Doug

--- In hockey-databank@yahoogroups.com, "sarona_mb" <sarona_mb@...> wrote:
>
> Hi Doug,
>
> I have been studying the tables in hdb-2009-01-15.zip for normalization.
>
> During this process, I noticed possible typos in table abbrev.csv.  Should the
fullname entries, with respect to codes AVC, AVF, AVSF, AVQF and AVPRE, be AVCO
instead of AVRO?
>
> Thanks and excellent work.
>
>
> Regards,
>
> Ron
>

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the hockey-databank
group.

   File        : /2009preview.zip
   Uploaded by : dsreyn <dreynolds@...>
   Description : Preliminary 2008-09 release

You can access this file at the URL:
http://groups.yahoo.com/group/hockey-databank/files/2009preview.zip

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/l/us/yahoo/groups/original/members/web/index.htmlfiles

Regards,

dsreyn <dreynolds@...>

Hi Doug -

Thank you very much for your work!  I have attempted a similar venture many
times for my own use, but it always falls by the wayside due to limited time
and/or resources.

Some simple suggestions for the next update.  In the Scoring file, you can
populate PPG, SHG, GWG, and GTG = 0 for everyone who has 0 goals scored. 
Likewise, PPA and SHA = 0 for all players with 0 assists.  (And of course,
similar concept for the post-season stats as well.)

Also, for every team in history that had 0 ties in a season, all the players on
their team can be populated with 0 GTG's.  Similarly, in the playoffs, if a team
won 0 playoff games, then all the players on their team can be populated with 0
PostGWG's.

I know these are all pretty minor and simple, but they are certainly accurate
and are very easy to update.  I have always been fascinated with the concept of
complete and accurate statistical histories in sports, so I just love projects
like this.  If I can find the time, I wouldn't mind contributing other pieces of
data (such as uniform numbers, the respective components that make up +/-,
time-on-ice, and other data), though no guarantees.

Roger

These suggestions do seem logical, but there are times when it may be useful to
differentiate between "data not available" and "zero".  I assume that the blank
data fields indicate "DNA" in the current files.  It's probably better to just
let the end user of the files fill in the zeroes as you recommend.

--- In hockey-databank@yahoogroups.com, "hockeydad55" <hockeydad444@...> wrote:
>
> Hi Doug -
>
> Thank you very much for your work!  I have attempted a similar venture many
times for my own use, but it always falls by the wayside due to limited time
and/or resources.
>
> Some simple suggestions for the next update.  In the Scoring file, you can
populate PPG, SHG, GWG, and GTG = 0 for everyone who has 0 goals scored. 
Likewise, PPA and SHA = 0 for all players with 0 assists.  (And of course,
similar concept for the post-season stats as well.)
>
> Also, for every team in history that had 0 ties in a season, all the players
on their team can be populated with 0 GTG's.  Similarly, in the playoffs, if a
team won 0 playoff games, then all the players on their team can be populated
with 0 PostGWG's.
>
> I know these are all pretty minor and simple, but they are certainly accurate
and are very easy to update.  I have always been fascinated with the concept of
complete and accurate statistical histories in sports, so I just love projects
like this.  If I can find the time, I wouldn't mind contributing other pieces of
data (such as uniform numbers, the respective components that make up +/-,
time-on-ice, and other data), though no guarantees.
>
> Roger
>

Within "database land", the proper method to handle "data not available" is
through the use of NULL. Using NULL is unambiguous, where blanks are decidedly
not.
Some SQL database programs (older, less developed, but still) will either choke
on or incorrectly handle data containing spaces.

Regardless, I want to at least join hockeydad55 in saying thank you!

--- In hockey-databank@yahoogroups.com, "kmanter" <kmanter@...> wrote:
>
> These suggestions do seem logical, but there are times when it may be useful
to differentiate between "data not available" and "zero".  I assume that the
blank data fields indicate "DNA" in the current files.  It's probably better to
just let the end user of the files fill in the zeroes as you recommend.
>
> --- In hockey-databank@yahoogroups.com, "hockeydad55" <hockeydad444@> wrote:
> >
> > Hi Doug -
> >
> > Thank you very much for your work!  I have attempted a similar venture many
times for my own use, but it always falls by the wayside due to limited time
and/or resources.
> >
> > Some simple suggestions for the next update.  In the Scoring file, you can
populate PPG, SHG, GWG, and GTG = 0 for everyone who has 0 goals scored. 
Likewise, PPA and SHA = 0 for all players with 0 assists.  (And of course,
similar concept for the post-season stats as well.)
> >
> > Also, for every team in history that had 0 ties in a season, all the players
on their team can be populated with 0 GTG's.  Similarly, in the playoffs, if a
team won 0 playoff games, then all the players on their team can be populated
with 0 PostGWG's.
> >
> > I know these are all pretty minor and simple, but they are certainly
accurate and are very easy to update.  I have always been fascinated with the
concept of complete and accurate statistical histories in sports, so I just love
projects like this.  If I can find the time, I wouldn't mind contributing other
pieces of data (such as uniform numbers, the respective components that make up
+/-, time-on-ice, and other data), though no guarantees.
> >
> > Roger
> >
>

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the hockey-databank
group.

   File        : /hdb-2009-06-18.zip
   Uploaded by : dsreyn <dreynolds@...>
   Description : 2008-09 update

You can access this file at the URL:
http://groups.yahoo.com/group/hockey-databank/files/hdb-2009-06-18.zip

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/l/us/yahoo/groups/original/members/web/index.htmlfiles

Regards,

dsreyn <dreynolds@...>

A year or so ago, I posted a message saying that I had a basketball database
available (see message #98 in the group message archive).  I've decided to post
this to the files area - it's available as bkb090621.zip.  The hope, of course,
is that someone will start an "open source" basketball project similar to this
one.  But if not, it's there anyway for the taking.

I think the only substantial change since I posted about this previously is that
I added an awards table (actually, two - one for players, one for coaches).  So
the database is up to date through the 2006-07 season (except for the draft
table, which goes only through 2004).  So obviously, there are two seasons of
updates needed to get it current again.  I'm very unlikely to take it any
further at this point.

I don't want to generate a lot of off-topic discussion about basketball in this
group, so if anyone has questions about the database, please contact me offline.
But of course, if someone start a basketball-databank group, feel free to make
an announcement here.

Doug

Hi,
Most sources list Bobby Hull with 67 games played in the 1960-61 season, but in
compiling that season's lineups, I have discovered that he actually missed four
games, giving him only 66 games played for the season. In addition to being not
listed in the lineups for 12/24/60, 12/25/60, 1/18/61, and 1/21/61, the Chicago
paper's game stories mention his absence from these four games.
Not a big deal, but I just thought I'd mention it here.

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the hockey-databank
group.

   File        : /tran.zip
   Uploaded by : dsreyn <dreynolds@...>
   Description : Transaction database (preliminary)

You can access this file at the URL:
http://groups.yahoo.com/group/hockey-databank/files/tran.zip

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/l/us/yahoo/groups/original/general.htmlfiles

Regards,

dsreyn <dreynolds@...>

Hello,

This email message is a notification to let you know that
a file has been uploaded to the Files area of the hockey-databank
group.

   File        : /tricks.zip
   Uploaded by : dsreyn <dreynolds@...>
   Description : Hat tricks (incomplete)

You can access this file at the URL:
http://groups.yahoo.com/group/hockey-databank/files/tricks.zip

To learn more about file sharing for your group, please visit:
http://help.yahoo.com/l/us/yahoo/groups/original/general.htmlfiles

Regards,

dsreyn <dreynolds@...>

Just a quick note about the hat trick file I just uploaded - the columns are:

year
month
date
playerID
goals
league
team
opponent
site

This is very preliminary at this point - it's NHL only, and only complete from
about 1960 onward (lots of gaps in particular for earlier seasons for Boston).

Doug