Gregory Hildstrom Projects Publications Resume Links Contact About Google+ Facebook Youtube Donate




          James Merrill
               “Jim“
         Jeffords (I-VT)
        An Unbiased Attempt
      at Political Profiling

      Gregory Alan Hildstrom
            12/17/2005

 Project Option 1: Write a Senator's Profile
            Fall 2005
        GVPT 170 Sect 6984
        American Government
         Professor Hrutka
             
             
Table of Contents
Introduction...................................................................................................................................... 3
Senate Voting Records.....................................................................................................................4
Voting Record Analysis................................................................................................................... 6
Conclusions.................................................................................................................................... 12
Sources and References................................................................................................................. 14
Appendix A: Batch File For Retrieving All URLs........................................................................ 15
Appendix B: Example Text File Containing Votes....................................................................... 16
Appendix C: C++ Source Code for stats program......................................................................... 17
Appendix D: Batch File for Calculating Statistics......................................................................... 22
Appendix E: Example Statistics File............................................................................................. 23
Appendix G: Snapshot of Project Folder....................................................................................... 24


Illustration Index
Illustration 1: Number of Senate Vote Topics................................................................................. 6
Illustration 2: Number of Votes and Issues Analyzed..................................................................... 7
Illustration 3: Correlation Agreement Example............................................................................... 8
Illustration 4: Correlation Disagreement Example.......................................................................... 8
Illustration 5: Correlation Split Example......................................................................................... 8
Illustration 6: Jeffords-Republican Correlations.............................................................................. 9
Illustration 7: Jeffords-Democrat Correlations.............................................................................. 10
Illustration 8: Republican-Democrat Correlations......................................................................... 11
Illustration 9: Summary Average Correlations.............................................................................. 12


Introduction
I was very intimidated by this assignment when I first read about it. A political profile was so far
outside of my realm of experiences that I hardly knew where to begin. I knew very little about
American government when I enrolled in this class, so terms like Republican, Democrat,
conservative, and liberal all put a bad taste in my mouth because they were not part of my
working vocabulary. I went to the Senate website and slowly looked through the list of senators.
Only one really stuck out to me: Jeffords, James- (I-VT). I have not been very politically active
or knowledgeable, so the only person in the Senate that is not a Republican or Democrat seemed
far more intriguing than anyone from my home state.

I then thought about ways to profile a senator without interjecting too much of my own opinions
and without being influenced by the opinions of others. Most political articles are biased in one
way or another, so I decided to go straight to the best source of information possible: actual
senate voting records. I decided that I should try to let the votes speak for themselves.

Jeffords was first elected to the senate in 1988 as a Republican. “In 2001, Senator Jeffords left
the Republican Party and became an independent.“ (“Senator Jim Jeffords“, 2005) He is
currently serving his third term in the Senate and “he has announced that he will retire at the end
of his current term, in January 2007.“ (“Jim Jeffords“, 2005)

The results of his voting records are extremely interesting. The controversy surrounding his
transition to an independent also raises many questions. What were his motives? Has he
accomplished his goals?


Senate Voting Records
What better way to profile a person than from his actions? After all, actions speak louder than
words, right? I usually take what people have to say about themselves with a grain of salt.
Listening to the self-opinion of others is always inherently biased positively, so I chose to start
my political analysis and data collection with Senate voting records. This is Jeffords' 17th year in
the Senate, so there are plenty of data somewhere. I wanted to come up with a way to compare
Jeffords voting records to Republicans and Democrats over an extended period of time. That
analysis should show his performance with cold, hard, unbiased numbers. Then I can try to
compare the facts and draw some conclusions about his political affiliations.

I performed many, many, many Google and library database searches on Senate voting records
and Jim Jeffords. I hoped that someone had already taken the time to tabulate years of votes in
spreadsheet form. I had no such luck, but I was already committed to this plan of attack and
analysis. The best source of information that I found was the Senate website, but each vote topic
for each year needed to be clicked on individually. There were thousands of vote topics over the
17 years I was interested in, so pointing, clicking, and copying all of the requisite results was not
an option. The data presented by the Senate website was also not in a very usable form; it just
listed the senators' votes with text in a standard HTML web page. The web pages are probably
generated by a database and some web script code, but I did not have access to the original
database.

I decided to write a series of scripts and small programs to mine as much data from the Senate
website as possible. I started by analyzing the URL that was generated when I clicked on a
particular year and vote topic. Each vote topic web page ended with the string
“?congress=101&session=1&vote=00001“, with the requisite numbers changed. Those three
variables, congress, session, and vote, tell the web script on the Senate web site which records to
extract from the database. All I needed to do was make thousands of web page requests and save
the results.

I decided to use the command-line tool wget, which can easily download the HTML for a given
URL. Because this tool is command-line driven it can be easily called from a script, batch file, or
program. I then created a spreadsheet that I used to create thousands of web page wget
commands using only a few clicks and drags of the mouse. I then pasted these automatically
generated commands into a batch file to execute automatically in bulk. The script created one
HTML file for each year that contained all votes from all senators. The 5773 web page requests I
made to the Senate web site can be seen in the figure 1 below along with the total number of
individual senator votes.

                            Total Senate Roll Call Votes                577300
                            Total Senate Issues 1989-2005                 5773
                            Total Senate Issues 1989                       312
                            Total Senate Issues 1990                       326
                            Total Senate Issues 1991                       280
                            Total Senate Issues 1992                       270
                            Total Senate Issues 1993                       395
                            Total Senate Issues 1994                       329
                            Total Senate Issues 1995                       613
                            Total Senate Issues 1996                       306
                            Total Senate Issues 1997                       298
                            Total Senate Issues 1998                       314
                            Total Senate Issues 1999                       374
                            Total Senate Issues 2000                       298
                            Total Senate Issues 2001                       380
                            Total Senate Issues 2002                       253
                            Total Senate Issues 2003                       459
                            Total Senate Issues 2004                       216
                            Total Senate Issues 2005                       350
                           Illustration 1: Number of Senate Vote Topics
                           
Unfortunately, the Senate.gov web server is not the best server on the planet. Not all of my
requests were answered; even with the three-second delay I inserted between web page requests.
The total process required several hours of web page requests and most were successful; at least
90%. The scripts generated one HTML file with all of the votes for each year. I then copied all of
the text for each year out of a web browser and pasted it into one ASCII text file for each year.
This process removed any errant HTML tags left in the results after the scripts had completed.


Voting Record Analysis
This process left me with one ASCII text file for each of the last 17 years that contained every
vote from every senator, corresponding political affiliation, and state. I decided to compile the
average Republican vote and average Democrat vote for comparison with Jeffords' vote on each
vote issue or topic. To accomplish this, I assigned a value of 0 for “Nay“ and 1 for “Yea“. If all
of the Republicans voted exactly the same on a given issue, the average would be exactly 0 for
all opposed, or 1 for all in favor. If the Republicans were exactly split on the issue, the average
would be 0.5. If the average vote is greater than 0.5, the party was generally in favor. If the
average vote is less than 0.5, the party was generally opposed.

I wrote a small C++ program, called stats, that read a vote ASCII text file for a single year and
calculated the following statistics: number of non-votes, number of Republican votes, average
Republican vote, number of Democrat votes, average Democrat vote, number of other party
votes, average other party vote, and Jeffords' vote. I fed the outputs from the stats program back
into spreadsheets. The program only output the results if Jeffords' vote was a “Nay“ or “Yea“. If
Jeffords did not vote or if the issue was about guilt, it was not tabulated. The number of votes
and issues analyzed is summarized in figure 2.

                       Percentage of Total Senate Issues Analyzed          83.21
                       Total Senate Issues Retrieved and Analyzed           4804
                       Total Non-Votes Analyzed                            12829
                       Total Republican Votes Analyzed                    230820
                       Total Democrat Votes Analyzed                      235117
                       Total Other Party Votes Analyzed                     1361
                       Total Jeffords Votes Analyzed                        3295
                      Illustration 2: Number of Votes and Issues Analyzed
                      
This analysis resulted in several lists of numbers for each year: average Republican vote, average
Democrat vote, and Jeffords' vote. Any two data sets can be compared for linear correlation by
calculating the correlation coefficient. A correlation of 1 reflects perfect correlation or
agreement. A correlation of -1 reflects complete disagreement. A correlation of 0 reflects
perfectly split agreement and disagreement. Three example votes and corresponding correlation
coefficients are presented in the figures 3, 4, and 5. A correlation greater than 0 indicates general
agreement and a correlation less than 0 indicates general disagreement.

    Vote Issue /Topic                     Example Republican Vote      Example Jeffords Vote      Correlation
                                  1                             1                          1                1
                                  2                             1                          1
                                  3                             0                          0
                                  4                             1                          1
                                  5                             0                          0
                                  6                             0                          0
                                  7                             0                          0
                                  8                             1                          1
   Illustration 3: Correlation Agreement Example
   
    Vote Issue /Topic                     Example Republican Vote      Example Jeffords Vote      Correlation
                                  1                             1                          0               -1
                                  2                             1                          0
                                  3                             0                          1
                                  4                             1                          0
                                  5                             0                          1
                                  6                             0                          1
                                  7                             0                          1
                                  8                             1                          0
   Illustration 4: Correlation Disagreement Example
   
    Vote Issue /Topic                     Example Republican Vote      Example Jeffords Vote      Correlation
                                  1                             1                          0                0
                                  2                             1                          1
                                  3                             0                          1
                                  4                             1                          1
                                  5                             0                          1
                                  6                             0                          0
                                  7                             0                          1
                                  8                             1                          1
   Illustration 5: Correlation Split Example
   
I then calculated correlation coefficients between Jeffords vote and the average vote of the
Republicans and Democrats for each year. I also calculated correlation between the average
Republican vote and the average Democrat vote for each year. The results are intriguing to say
the least.

Figure 6 shows the correlation between Jeffords' vote and the average Republican vote by year
from 1989 to 2005. You can clearly see the migration from general agreement to general
disagreement during 2001 when he left the Republican party and became independent. Jeffords
was true to his word and voted according to his stated disagreement with the Republican party.

Correlation of Jeffeords Vote
With Average Republican Vote
  0.7 ++------+-------+------+-------+-------+-------+------+-------+------++
      +       +       +      +       +       +"~/data.csv" using 1:2+**A*** +
  0.6 ++                                         A*                        ++
      |                                         *  **                       |
  0.5 ++                                        *                          ++
      |   A                                    *     A**                    |
  0.4 ++   *                           **A**  *         *                  ++
      |     *                        A*     * *          A                  |
  0.3 ++    *                      **        A           *                 ++
      |      *  **A***  *A**      *                       *                 |
  0.2 ++      A*      A*    *  **A                        *                ++
      |                      A*                           *                 |
  0.1 ++                                                  *                ++
      |                                                    *                |
    0 ++                                                   *               ++
      |                                                    *                |
 -0.1 ++                                                   *               ++
      |                                                     *               |
 -0.2 ++                                                    A**       **A  ++
      +       +       +      +       +       +       +      +  *  **A*      +
 -0.3 ++------+-------+------+-------+-------+-------+------+---A*--+------++
     1988    1990    1992   1994    1996    1998    2000   2002    2004    2006

Illustration 6: Jeffords-Republican Correlations

Figure 7 shows the correlation between Jeffords' vote and the average Democrat vote by year
from 1989 to 2005. You can clearly see the increase in general agreement during 2001 when he
left the Republican party and became independent. During this time period, Jeffords' general
agreement with the average Democrat vote soared higher than ever.

Correlation of Jeffeords Vote
With Average Democrat Vote
  0.9 ++------+-------+------+-------+-------+-------+------+-------+------++
      +       +       +      +       +       +"~/data.csv" using 1:2+**A*** +
  0.8 ++                                                            A***A  ++
      |                                                     A**   **        |
  0.7 ++                                                    *  * *         ++
      |                                                    *    A           |
  0.6 ++                                                   *               ++
      |              *A        **A                        *                 |
  0.5 ++  A***     **  *     A*   *                       *                ++
      |       A***A    *    *     *                      A                  |
  0.4 ++                *  *       *     A***A         **                  ++
      |                 *  *       *    *     *       *                     |
  0.3 ++                 **         *  *      *      A                     ++
      |                  A          * *        *    *                       |
  0.2 ++                             A         *    *                      ++
      |                                         *  *                        |
  0.1 ++                                        * *                        ++
      |                                          **                         |
    0 ++                                         A                         ++
      +       +       +      +       +       +       +      +       +       +
 -0.1 ++------+-------+------+-------+-------+-------+------+-------+------++
     1988    1990    1992   1994    1996    1998    2000   2002    2004    2006
Illustration 7: Jeffords-Democrat Correlations

Just as a sanity check, figure 8 shows the correlation between the average Republican vote and
the average Democrat vote. Not surprisingly, this correlation has been hovering below zero for
most of the past 17 years, which indicates a general disagreement between the Republicans and
the Democrats.

Correlation of Average Republican Vote
With Average Democrat Vote
  0.4 ++------+-------+------+-------+-------+-------+------+-------+------++
      +       +       +      +       +       +"~/data.csv" using 1:2+**A*** +
      |   A                                                                 |
  0.2 ++   *                                                               ++
      |     *                                                               |
      |     *                                                               |
      |      *  **A*                                                        |
    0 ++      A*    **                                                     ++
      |                      A***        A**                                |
      |               A     *    A*     *   *        A***A                  |
 -0.2 ++               *   *       **  *     A**   **    *                 ++
      |                 * *           *         * *       *         A***A   |
      |                  A           A           A        *       **        |
 -0.4 ++                                                  *      *         ++
      |                                                    *    A           |
      |                                                    *   *            |
      |                                                    *  *             |
 -0.6 ++                                                   *  *            ++
      |                                                     **              |
      +       +       +      +       +       +       +      A       +       +
 -0.8 ++------+-------+------+-------+-------+-------+------+-------+------++
     1988    1990    1992   1994    1996    1998    2000   2002    2004    2006
Illustration 8: Republican-Democrat Correlations


Conclusions
These results really surprised me. The obvious statistic that I expected to find was the deviation
with the Republican party after 2001 when he became independent. This is clearly evident in
figure 6 and it has been summarized in figure 9. The Jeffords-Republican average correlation
from 1989 to 2001 was 0.33, which is greater than 0 and indicates general agreement. The
Jeffords-Republican average correlation from 2002 to 2005 was -0.25, which indicates general
disagreement.

                 Data Sets                                            Average Correlation
                 1989-2001 Jeffords-Republican Correlations                          0.33
                 2002-2005 Jeffords-Republican Correlations                         -0.25
                 1989-2005 Jeffords-Republican Correlations                           0.2
                 1989-2001 Jeffords-Democrat Correlations                            0.37
                 2002-2005 Jeffords-Democrat Correlations                            0.75
                 1989-2005 Jeffords-Democrat Correlations                            0.46
                 1989-2005 Republican-Democrat Correlations                         -0.16
                Illustration 9: Summary Average Correlations
                
The sanity check statistic that I also expected to find relates to the Republican-Democrat
correlations. The Republican-Democrat average correlation from 1989 to 2005 was -0.16, which
indicates a general disagreement as expected.

The most surprising statistics that came out of this analysis relate to the Democratic party. The
Jeffords-Democrat average correlation from 1989 to 2001 was 0.37, which indicates general
agreement. The Jeffords-Democrat average correlation from 2002 to 2005 climbed to 0.75,
which indicates a higher level of general agreement.

Over the course of his congressional career, Jeffords has voted more on the side of the
Democratic party than the Republican party. This fact is clearly evident in figure 9, which shows
that from 1989 to 2005 the Jeffords-Democrat average correlation of 0.46 is higher than the
Jeffords-Republican average correlation of 0.2. This fact was also true before he became an
independent in 2001. The statistics show that from 1989 to 2001 the Jeffords-Democrat average
correlation of 0.37 is higher than the Jeffords-Republican average correlation of 0.33. This shows
that when he was a Republican and while he has been an independent, Jeffords has voted more in
line with the Democratic party than the Republican party. These are not my personal opinions,
these are the conclusions that must be drawn from the statistics that have been generated from
actual Senate voting records.


Sources and References
Senate Voting Records. (2005). Retrieved December 17, 2005, from the Senate web site:
       http://www.senate.gov/pagelayout/legislative/a_three_sections_with_teasers/votes.htm
Senator Jim Jeffords. (2005). Retrieved December 17, 2005, from the Senate web site:
       http://jeffords.senate.gov/
Jim Jeffords. (2005). Retrieved December 17, 2005, from the Wikipedia web site:
       http://en.wikipedia.org/wiki/Jim_Jeffords
Snow, Kate. (2001). Jeffords leaves GOP, throwing Senate control to Democrats. Retrieved
       December 17, 2005, from the CNN web site:
       http://archives.cnn.com/2001/ALLPOLITICS/05/24/jeffords.senate/
GNU Wget. (2005). Retrieved December 17, 2005, from GNU web site:
       http://www.gnu.org/software/wget/wget.html
WGET for Windows. (2005). Retrieved December 17, 2005, from tcharron web site:
       http://www.interlog.com/~tcharron/wgetwin.html
Patterson, Thomas E. (2005). The American Democracy, Seventh Edition. New York: McGraw-
       Hill.


Appendix A: Batch File For Retrieving All URLs
wg101-1.bat; cp output3.html votes101-1.html
wg101-2.bat; cp output3.html votes101-2.html
wg102-1.bat; cp output3.html votes102-1.html
wg102-2.bat; cp output3.html votes102-2.html
wg103-1.bat; cp output3.html votes103-1.html
wg103-2.bat; cp output3.html votes103-2.html
wg104-1.bat; cp output3.html votes104-1.html
wg104-2.bat; cp output3.html votes104-2.html
wg105-1.bat; cp output3.html votes105-1.html
wg105-2.bat; cp output3.html votes105-2.html
wg106-1.bat; cp output3.html votes106-1.html
wg106-2.bat; cp output3.html votes106-2.html
wg107-1.bat; cp output3.html votes107-1.html
wg107-2.bat; cp output3.html votes107-2.html
wg108-1.bat; cp output3.html votes108-1.html
wg108-2.bat; cp output3.html votes108-2.html
wg109-1.bat; cp output3.html votes109-1.html
Each wg###-#.bat file contains hundreds of lines of properly formulated wget commands to
retrieve each senate vote topic URL.


Appendix B: Example Text File Containing Votes
Adams (D-WA), Yea
Akaka (D-HI), Yea
Baucus (D-MT), Yea
Bentsen (D-TX), Not Voting
Biden (D-DE), Yea
Bingaman (D-NM), Yea
Bond (R-MO), Not Voting
Boren (D-OK), Yea
Bradley (D-NJ), Yea
Breaux (D-LA), Yea
Brown (R-CO), Yea
Bryan (D-NV), Yea
Bumpers (D-AR), Yea
...
...
...
20000+ individual votes per year in each text file


Appendix C: C++ Source Code for stats program
#include
#include
#include
//#include
//#include
using namespace std;
int main(unsigned int argc, char* argv[]){
        if(argc <= 1) return 0;
        ifstream infile;
        infile.open(argv[1], ios::in);
        if(!infile.good()) return 0;
        string temp;
        string senator;
        string previoussenator;
        string party;
        string state;
        string vote;
        float sumdemocratvotes;
        unsigned int numdemocrats;
        float sumrepublicanvotes;
        unsigned int numrepublicans;
        float sumothervotes;
        unsigned int numothers;
        unsigned int totalsenators;
unsigned int numnonvotes;
float voteinteger;
float jeffordsvote;
string firstsenator = “z“;
unsigned int filepos = 0;
bool endofissue = false;
cout
<< “first senator“ << “,“
<< “last senator“ << “,“
<< “num Non Votes“ << “,“
<< “num R votes“ << “,“
<< “average R vote“ << “,“
<< “num D votes“ << “,“
<< “average D vote“ << “,“
<< “num O votes“ << “,“
<< “average O vote“ << “,“
<< “jeffords vote“ << endl;
while(!infile.eof()){
        //reset stats
        sumdemocratvotes = 0;
        numdemocrats = 0;
        sumrepublicanvotes = 0;
        numrepublicans = 0;
        sumothervotes = 0;
        numothers = 0;
        jeffordsvote = 0.5;
        totalsenators = 0;
        numnonvotes = 0;
        endofissue = false;
while(!infile.eof() && totalsenators < 100 && !endofissue){
       senator.erase();
       party.erase();
       state.erase();
       vote.erase();
       voteinteger = 0.5;
       while(infile.peek() == ' ' || infile.peek() == '\n') getline(infile,temp,'\n');
       getline(infile,senator,'(');
       if(firstsenator.size() <= 1)firstsenator = senator;
       if(totalsenators > 1 && senator == firstsenator){
                filepos = infile.tellg();
                filepos -= senator.size();
                infile.seekg(filepos);
                endofissue = true;
                senator = previoussenator;
                break;
       }//end if
       previoussenator = senator;
       getline(infile,party,'-');
       getline(infile,state,')');
       getline(infile,temp,' ');
       getline(infile,vote,'\n');
       if(vote == “Yea“)
                voteinteger = 1;
       if(vote == “Nay“)
                voteinteger = 0;
       totalsenators++;
                        if(voteinteger != 0.5){
                                if(party == “D“){
                                        numdemocrats++;
                                        sumdemocratvotes += voteinteger;
                                }//end if
                                else if(party == “R“){
                                        numrepublicans++;
                                        sumrepublicanvotes += voteinteger;
                                }//end if
                                else {
                                        numothers++;
                                        sumothervotes += voteinteger;
                                }//end if
                                if(senator == “Jeffords “){
                                        jeffordsvote = voteinteger;
                                }//end if
                        }//end if person actually voted
                        else{
                                numnonvotes++;
                        }//end else
               }//end while read a single issue
               if(totalsenators != 100 && totalsenators != 1) cerr << “error: total senators “ <<
totalsenators << “ != 100\n“;
                if(totalsenators > 1 && jeffordsvote != 0.5){
                         cout
                         << firstsenator << “,“
                         << senator << “,“
                         << numnonvotes << “,“
                         << numrepublicans << “,“
                         << sumrepublicanvotes/numrepublicans << “,“
                         << numdemocrats << “,“
                         << sumdemocratvotes/numdemocrats << “,“
                         << numothers << “,“
                         << sumothervotes/numothers << “,“
                         << jeffordsvote
                         << endl;
                }//end if
       }//end while reading file
       infile.close();
       return 0;
}//end main


Appendix D: Batch File for Calculating Statistics
stats votes101-1.txt > stats101-1.csv
stats votes101-2.txt > stats101-2.csv
stats votes102-1.txt > stats102-1.csv
stats votes102-2.txt > stats102-2.csv
stats votes103-1.txt > stats103-1.csv
stats votes103-2.txt > stats103-2.csv
stats votes104-1.txt > stats104-1.csv
stats votes104-2.txt > stats104-2.csv
stats votes105-1.txt > stats105-1.csv
stats votes105-2.txt > stats105-2.csv
stats votes106-1.txt > stats106-1.csv
stats votes106-2.txt > stats106-2.csv
stats votes107-1.txt > stats107-1.csv
stats votes107-2.txt > stats107-2.csv
stats votes108-1.txt > stats108-1.csv
stats votes108-2.txt > stats108-2.csv
stats votes109-1.txt > stats109-1.csv


Appendix E: Example Statistics File
first senator,last senator,num Non Votes,num R votes,average R vote,num D votes,average D
vote,num O votes,average O vote,jeffords vote
Adams ,Wirth ,1,44,1,55,1,0,-1.#IND,1
Adams ,Wirth ,1,44,1,55,1,0,-1.#IND,1
Adams ,Wirth ,0,45,1,55,1,0,-1.#IND,1
Adams ,Wirth ,0,45,1,55,1,0,-1.#IND,1
Adams ,Wirth ,0,45,1,55,1,0,-1.#IND,1
Adams ,Wirth ,0,45,1,55,1,0,-1.#IND,1
Adams ,Wirth ,0,45,1,55,1,0,-1.#IND,1
Adams ,Wirth ,0,45,1,55,1,0,-1.#IND,1
Adams ,Wirth ,0,45,1,55,1,0,-1.#IND,1
Adams ,Wirth ,0,45,0.955556,55,0.945455,0,-1.#IND,1
Adams ,Wirth ,0,45,0.933333,55,0.945455,0,-1.#IND,0
Adams ,Wirth ,0,45,1,55,1,0,-1.#IND,1
Adams ,Wirth ,1,45,0.977778,54,1,0,-1.#IND,1
Adams ,Wirth ,1,45,1,54,1,0,-1.#IND,1
Adams ,Wirth ,6,43,1,51,1,0,-1.#IND,1


Appendix G: Snapshot of Project Folder