Phillips did not play his first first-class match until 1885/86, and altogether played 124 matches for Canterbury, Middlesex and Victoria between 1885/86 and 1898/99 as a right-hand batsman and right-arm medium bowler. He scored 1827 runs at an average of 12.59 with a highest score of 110 not out, and took 355 wickets at an average of 20.00 with best figures of 8 for 69. Seven times he took 10 or more wickets in a match. He also took 50 catches. As a player and umpire he travelled between Australia and England, following the cricket seasons. Middlesex valued his contribution so highly that he was given a benefit match, Australia v. Middlesex, in 1899. He was also able to help Australian players, such as Albert Trott find jobs and play cricket in England. For a while he coached in Christchurch, New Zealand and played first-class cricket for Canterbury.
Phillips umpired a total of 29 Test matches. His first match as an umpire was between Australia and England in Melbourne on 21 March 1885 to 25 March 1885, comfortably won by England by an innings and 98 runs. His colleague was George Hodges, standing in his only Test match.
He umpired 13 Test matches in Australia, including all 5 in the 1894/95 and the 1897/98 series against England. He also umpired 11 matches in England against Australian touring sides, including all 3 in 1896, and 4 out of 5 in 1905. His last matches were the 5 played in South Africa against England in 1905/06. Thus he can claim to be the first genuinely "international" Test umpire.
According to Jack Pollard, 'Dimboola Jim' Phillips was "a fearless umpire who was largely responsible for stamping out throwing around the turn of the [20th] century … establishing an international reputation for acumen and honesty." Amongst the bowlers called for throwing by Phillips were Australian Test players Ernie Jones and Tom McKibbin. Jones was first no-balled in a match between South Australia and the visiting English side in 1897/98. Phillips again no-balled him once in the 2nd Test of that series, Jones thus becoming the first bowler to be called for throwing in a Test match. The selectors omitted Jack Saunders from the 1905 team to tour England because it was feared that Phillips would no-ball him for throwing. Wisden blamed English umpires for the problem, claiming that "Australian bowlers never threw in England until we had shown them over and over again that [the Law] could be broken with impunity." Phillips' actions, including the no-balling of the English amateur champion C. B. Fry, led to a meeting of County captains in 1900, which recommended that nine regular bowlers not be used in the following season.
In the first Test match of 1897/98 Phillips gave Charlie McLeod run out. McLeod was bowled by a no-ball but, because of his deafness, did not hear the call. He left the wicket, believing he was out, and a fieldsman pulled out a stump and appealed. Under present-day Laws McLeod would be given 'not out' as he had "left his wicket under a misapprehension that he [was] out". (Dean Jones suffered a similar fate in the West Indies).
When smoke from bushfires made visibility difficult at Melbourne in the 4th Test of the 1897/98 series, the English captain claimed that play should cease because of bad light. Umpire Phillips was unimpressed, saying that "if that light was bad, then cricket had better be given up entirely at Bramall Lane, Bradford, and Old Trafford." An English player remarked that Australia was the only place where the country was set alight just to win a cricket match.
On retirement, Phillips became a mining engineer and is reputed to have made a fortune in North America.
