James Merrill
“Jim“
Jeffords (I-VT)
An Unbiased Attempt
at Political Profiling
Gregory Alan Hildstrom
12/17/2005
Project Option 1: Write a Senator's Profile
Fall 2005
GVPT 170 Sect 6984
American Government
Professor Hrutka
Table of Contents
Introduction...................................................................................................................................... 3
Senate Voting Records.....................................................................................................................4
Voting Record Analysis................................................................................................................... 6
Conclusions.................................................................................................................................... 12
Sources and References................................................................................................................. 14
Appendix A: Batch File For Retrieving All URLs........................................................................ 15
Appendix B: Example Text File Containing Votes....................................................................... 16
Appendix C: C++ Source Code for stats program......................................................................... 17
Appendix D: Batch File for Calculating Statistics......................................................................... 22
Appendix E: Example Statistics File............................................................................................. 23
Appendix G: Snapshot of Project Folder....................................................................................... 24
Illustration Index
Illustration 1: Number of Senate Vote Topics................................................................................. 6
Illustration 2: Number of Votes and Issues Analyzed..................................................................... 7
Illustration 3: Correlation Agreement Example............................................................................... 8
Illustration 4: Correlation Disagreement Example.......................................................................... 8
Illustration 5: Correlation Split Example......................................................................................... 8
Illustration 6: Jeffords-Republican Correlations.............................................................................. 9
Illustration 7: Jeffords-Democrat Correlations.............................................................................. 10
Illustration 8: Republican-Democrat Correlations......................................................................... 11
Illustration 9: Summary Average Correlations.............................................................................. 12
Introduction
I was very intimidated by this assignment when I first read about it. A political profile was so far
outside of my realm of experiences that I hardly knew where to begin. I knew very little about
American government when I enrolled in this class, so terms like Republican, Democrat,
conservative, and liberal all put a bad taste in my mouth because they were not part of my
working vocabulary. I went to the Senate website and slowly looked through the list of senators.
Only one really stuck out to me: Jeffords, James- (I-VT). I have not been very politically active
or knowledgeable, so the only person in the Senate that is not a Republican or Democrat seemed
far more intriguing than anyone from my home state.
I then thought about ways to profile a senator without interjecting too much of my own opinions
and without being influenced by the opinions of others. Most political articles are biased in one
way or another, so I decided to go straight to the best source of information possible: actual
senate voting records. I decided that I should try to let the votes speak for themselves.
Jeffords was first elected to the senate in 1988 as a Republican. “In 2001, Senator Jeffords left
the Republican Party and became an independent.“ (“Senator Jim Jeffords“, 2005) He is
currently serving his third term in the Senate and “he has announced that he will retire at the end
of his current term, in January 2007.“ (“Jim Jeffords“, 2005)
The results of his voting records are extremely interesting. The controversy surrounding his
transition to an independent also raises many questions. What were his motives? Has he
accomplished his goals?
Senate Voting Records
What better way to profile a person than from his actions? After all, actions speak louder than
words, right? I usually take what people have to say about themselves with a grain of salt.
Listening to the self-opinion of others is always inherently biased positively, so I chose to start
my political analysis and data collection with Senate voting records. This is Jeffords' 17th year in
the Senate, so there are plenty of data somewhere. I wanted to come up with a way to compare
Jeffords voting records to Republicans and Democrats over an extended period of time. That
analysis should show his performance with cold, hard, unbiased numbers. Then I can try to
compare the facts and draw some conclusions about his political affiliations.
I performed many, many, many Google and library database searches on Senate voting records
and Jim Jeffords. I hoped that someone had already taken the time to tabulate years of votes in
spreadsheet form. I had no such luck, but I was already committed to this plan of attack and
analysis. The best source of information that I found was the Senate website, but each vote topic
for each year needed to be clicked on individually. There were thousands of vote topics over the
17 years I was interested in, so pointing, clicking, and copying all of the requisite results was not
an option. The data presented by the Senate website was also not in a very usable form; it just
listed the senators' votes with text in a standard HTML web page. The web pages are probably
generated by a database and some web script code, but I did not have access to the original
database.
I decided to write a series of scripts and small programs to mine as much data from the Senate
website as possible. I started by analyzing the URL that was generated when I clicked on a
particular year and vote topic. Each vote topic web page ended with the string
“?congress=101&session=1&vote=00001“, with the requisite numbers changed. Those three
variables, congress, session, and vote, tell the web script on the Senate web site which records to
extract from the database. All I needed to do was make thousands of web page requests and save
the results.
I decided to use the command-line tool wget, which can easily download the HTML for a given
URL. Because this tool is command-line driven it can be easily called from a script, batch file, or
program. I then created a spreadsheet that I used to create thousands of web page wget
commands using only a few clicks and drags of the mouse. I then pasted these automatically
generated commands into a batch file to execute automatically in bulk. The script created one
HTML file for each year that contained all votes from all senators. The 5773 web page requests I
made to the Senate web site can be seen in the figure 1 below along with the total number of
individual senator votes.
Total Senate Roll Call Votes 577300
Total Senate Issues 1989-2005 5773
Total Senate Issues 1989 312
Total Senate Issues 1990 326
Total Senate Issues 1991 280
Total Senate Issues 1992 270
Total Senate Issues 1993 395
Total Senate Issues 1994 329
Total Senate Issues 1995 613
Total Senate Issues 1996 306
Total Senate Issues 1997 298
Total Senate Issues 1998 314
Total Senate Issues 1999 374
Total Senate Issues 2000 298
Total Senate Issues 2001 380
Total Senate Issues 2002 253
Total Senate Issues 2003 459
Total Senate Issues 2004 216
Total Senate Issues 2005 350
Illustration 1: Number of Senate Vote Topics
Unfortunately, the Senate.gov web server is not the best server on the planet. Not all of my
requests were answered; even with the three-second delay I inserted between web page requests.
The total process required several hours of web page requests and most were successful; at least
90%. The scripts generated one HTML file with all of the votes for each year. I then copied all of
the text for each year out of a web browser and pasted it into one ASCII text file for each year.
This process removed any errant HTML tags left in the results after the scripts had completed.
Voting Record Analysis
This process left me with one ASCII text file for each of the last 17 years that contained every
vote from every senator, corresponding political affiliation, and state. I decided to compile the
average Republican vote and average Democrat vote for comparison with Jeffords' vote on each
vote issue or topic. To accomplish this, I assigned a value of 0 for “Nay“ and 1 for “Yea“. If all
of the Republicans voted exactly the same on a given issue, the average would be exactly 0 for
all opposed, or 1 for all in favor. If the Republicans were exactly split on the issue, the average
would be 0.5. If the average vote is greater than 0.5, the party was generally in favor. If the
average vote is less than 0.5, the party was generally opposed.
I wrote a small C++ program, called stats, that read a vote ASCII text file for a single year and
calculated the following statistics: number of non-votes, number of Republican votes, average
Republican vote, number of Democrat votes, average Democrat vote, number of other party
votes, average other party vote, and Jeffords' vote. I fed the outputs from the stats program back
into spreadsheets. The program only output the results if Jeffords' vote was a “Nay“ or “Yea“. If
Jeffords did not vote or if the issue was about guilt, it was not tabulated. The number of votes
and issues analyzed is summarized in figure 2.
Percentage of Total Senate Issues Analyzed 83.21
Total Senate Issues Retrieved and Analyzed 4804
Total Non-Votes Analyzed 12829
Total Republican Votes Analyzed 230820
Total Democrat Votes Analyzed 235117
Total Other Party Votes Analyzed 1361
Total Jeffords Votes Analyzed 3295
Illustration 2: Number of Votes and Issues Analyzed
This analysis resulted in several lists of numbers for each year: average Republican vote, average
Democrat vote, and Jeffords' vote. Any two data sets can be compared for linear correlation by
calculating the correlation coefficient. A correlation of 1 reflects perfect correlation or
agreement. A correlation of -1 reflects complete disagreement. A correlation of 0 reflects
perfectly split agreement and disagreement. Three example votes and corresponding correlation
coefficients are presented in the figures 3, 4, and 5. A correlation greater than 0 indicates general
agreement and a correlation less than 0 indicates general disagreement.
Vote Issue /Topic Example Republican Vote Example Jeffords Vote Correlation
1 1 1 1
2 1 1
3 0 0
4 1 1
5 0 0
6 0 0
7 0 0
8 1 1
Illustration 3: Correlation Agreement Example
Vote Issue /Topic Example Republican Vote Example Jeffords Vote Correlation
1 1 0 -1
2 1 0
3 0 1
4 1 0
5 0 1
6 0 1
7 0 1
8 1 0
Illustration 4: Correlation Disagreement Example
Vote Issue /Topic Example Republican Vote Example Jeffords Vote Correlation
1 1 0 0
2 1 1
3 0 1
4 1 1
5 0 1
6 0 0
7 0 1
8 1 1
Illustration 5: Correlation Split Example
I then calculated correlation coefficients between Jeffords vote and the average vote of the
Republicans and Democrats for each year. I also calculated correlation between the average
Republican vote and the average Democrat vote for each year. The results are intriguing to say
the least.
Figure 6 shows the correlation between Jeffords' vote and the average Republican vote by year
from 1989 to 2005. You can clearly see the migration from general agreement to general
disagreement during 2001 when he left the Republican party and became independent. Jeffords
was true to his word and voted according to his stated disagreement with the Republican party.
Correlation of Jeffeords Vote
With Average Republican Vote
0.7 ++------+-------+------+-------+-------+-------+------+-------+------++
+ + + + + +"~/data.csv" using 1:2+**A*** +
0.6 ++ A* ++
| * ** |
0.5 ++ * ++
| A * A** |
0.4 ++ * **A** * * ++
| * A* * * A |
0.3 ++ * ** A * ++
| * **A*** *A** * * |
0.2 ++ A* A* * **A * ++
| A* * |
0.1 ++ * ++
| * |
0 ++ * ++
| * |
-0.1 ++ * ++
| * |
-0.2 ++ A** **A ++
+ + + + + + + + * **A* +
-0.3 ++------+-------+------+-------+-------+-------+------+---A*--+------++
1988 1990 1992 1994 1996 1998 2000 2002 2004 2006
Illustration 6: Jeffords-Republican Correlations
Figure 7 shows the correlation between Jeffords' vote and the average Democrat vote by year
from 1989 to 2005. You can clearly see the increase in general agreement during 2001 when he
left the Republican party and became independent. During this time period, Jeffords' general
agreement with the average Democrat vote soared higher than ever.
Correlation of Jeffeords Vote
With Average Democrat Vote
0.9 ++------+-------+------+-------+-------+-------+------+-------+------++
+ + + + + +"~/data.csv" using 1:2+**A*** +
0.8 ++ A***A ++
| A** ** |
0.7 ++ * * * ++
| * A |
0.6 ++ * ++
| *A **A * |
0.5 ++ A*** ** * A* * * ++
| A***A * * * A |
0.4 ++ * * * A***A ** ++
| * * * * * * |
0.3 ++ ** * * * A ++
| A * * * * |
0.2 ++ A * * ++
| * * |
0.1 ++ * * ++
| ** |
0 ++ A ++
+ + + + + + + + + +
-0.1 ++------+-------+------+-------+-------+-------+------+-------+------++
1988 1990 1992 1994 1996 1998 2000 2002 2004 2006
Illustration 7: Jeffords-Democrat Correlations
Just as a sanity check, figure 8 shows the correlation between the average Republican vote and
the average Democrat vote. Not surprisingly, this correlation has been hovering below zero for
most of the past 17 years, which indicates a general disagreement between the Republicans and
the Democrats.
Correlation of Average Republican Vote
With Average Democrat Vote
0.4 ++------+-------+------+-------+-------+-------+------+-------+------++
+ + + + + +"~/data.csv" using 1:2+**A*** +
| A |
0.2 ++ * ++
| * |
| * |
| * **A* |
0 ++ A* ** ++
| A*** A** |
| A * A* * * A***A |
-0.2 ++ * * ** * A** ** * ++
| * * * * * * A***A |
| A A A * ** |
-0.4 ++ * * ++
| * A |
| * * |
| * * |
-0.6 ++ * * ++
| ** |
+ + + + + + + A + +
-0.8 ++------+-------+------+-------+-------+-------+------+-------+------++
1988 1990 1992 1994 1996 1998 2000 2002 2004 2006
Illustration 8: Republican-Democrat Correlations
Conclusions
These results really surprised me. The obvious statistic that I expected to find was the deviation
with the Republican party after 2001 when he became independent. This is clearly evident in
figure 6 and it has been summarized in figure 9. The Jeffords-Republican average correlation
from 1989 to 2001 was 0.33, which is greater than 0 and indicates general agreement. The
Jeffords-Republican average correlation from 2002 to 2005 was -0.25, which indicates general
disagreement.
Data Sets Average Correlation
1989-2001 Jeffords-Republican Correlations 0.33
2002-2005 Jeffords-Republican Correlations -0.25
1989-2005 Jeffords-Republican Correlations 0.2
1989-2001 Jeffords-Democrat Correlations 0.37
2002-2005 Jeffords-Democrat Correlations 0.75
1989-2005 Jeffords-Democrat Correlations 0.46
1989-2005 Republican-Democrat Correlations -0.16
Illustration 9: Summary Average Correlations
The sanity check statistic that I also expected to find relates to the Republican-Democrat
correlations. The Republican-Democrat average correlation from 1989 to 2005 was -0.16, which
indicates a general disagreement as expected.
The most surprising statistics that came out of this analysis relate to the Democratic party. The
Jeffords-Democrat average correlation from 1989 to 2001 was 0.37, which indicates general
agreement. The Jeffords-Democrat average correlation from 2002 to 2005 climbed to 0.75,
which indicates a higher level of general agreement.
Over the course of his congressional career, Jeffords has voted more on the side of the
Democratic party than the Republican party. This fact is clearly evident in figure 9, which shows
that from 1989 to 2005 the Jeffords-Democrat average correlation of 0.46 is higher than the
Jeffords-Republican average correlation of 0.2. This fact was also true before he became an
independent in 2001. The statistics show that from 1989 to 2001 the Jeffords-Democrat average
correlation of 0.37 is higher than the Jeffords-Republican average correlation of 0.33. This shows
that when he was a Republican and while he has been an independent, Jeffords has voted more in
line with the Democratic party than the Republican party. These are not my personal opinions,
these are the conclusions that must be drawn from the statistics that have been generated from
actual Senate voting records.
Sources and References
Senate Voting Records. (2005). Retrieved December 17, 2005, from the Senate web site:
http://www.senate.gov/pagelayout/legislative/a_three_sections_with_teasers/votes.htm
Senator Jim Jeffords. (2005). Retrieved December 17, 2005, from the Senate web site:
http://jeffords.senate.gov/
Jim Jeffords. (2005). Retrieved December 17, 2005, from the Wikipedia web site:
http://en.wikipedia.org/wiki/Jim_Jeffords
Snow, Kate. (2001). Jeffords leaves GOP, throwing Senate control to Democrats. Retrieved
December 17, 2005, from the CNN web site:
http://archives.cnn.com/2001/ALLPOLITICS/05/24/jeffords.senate/
GNU Wget. (2005). Retrieved December 17, 2005, from GNU web site:
http://www.gnu.org/software/wget/wget.html
WGET for Windows. (2005). Retrieved December 17, 2005, from tcharron web site:
http://www.interlog.com/~tcharron/wgetwin.html
Patterson, Thomas E. (2005). The American Democracy, Seventh Edition. New York: McGraw-
Hill.
Appendix A: Batch File For Retrieving All URLs
wg101-1.bat; cp output3.html votes101-1.html
wg101-2.bat; cp output3.html votes101-2.html
wg102-1.bat; cp output3.html votes102-1.html
wg102-2.bat; cp output3.html votes102-2.html
wg103-1.bat; cp output3.html votes103-1.html
wg103-2.bat; cp output3.html votes103-2.html
wg104-1.bat; cp output3.html votes104-1.html
wg104-2.bat; cp output3.html votes104-2.html
wg105-1.bat; cp output3.html votes105-1.html
wg105-2.bat; cp output3.html votes105-2.html
wg106-1.bat; cp output3.html votes106-1.html
wg106-2.bat; cp output3.html votes106-2.html
wg107-1.bat; cp output3.html votes107-1.html
wg107-2.bat; cp output3.html votes107-2.html
wg108-1.bat; cp output3.html votes108-1.html
wg108-2.bat; cp output3.html votes108-2.html
wg109-1.bat; cp output3.html votes109-1.html
Each wg###-#.bat file contains hundreds of lines of properly formulated wget commands to
retrieve each senate vote topic URL.
Appendix B: Example Text File Containing Votes
Adams (D-WA), Yea
Akaka (D-HI), Yea
Baucus (D-MT), Yea
Bentsen (D-TX), Not Voting
Biden (D-DE), Yea
Bingaman (D-NM), Yea
Bond (R-MO), Not Voting
Boren (D-OK), Yea
Bradley (D-NJ), Yea
Breaux (D-LA), Yea
Brown (R-CO), Yea
Bryan (D-NV), Yea
Bumpers (D-AR), Yea
...
...
...
20000+ individual votes per year in each text file
Appendix C: C++ Source Code for stats program
#include
#include
#include
//#include
