The Berlin Philharmonic was founded in Berlin in 1882 by 54 musicians under the name Frühere Bilsesche Kapelle (literally, "Former Bilse's Band"); the group broke away from their previous conductor Benjamin Bilse after he announced his intention of taking the band on a fourth-class train to Warsaw for a concert. The orchestra was renamed and reorganized under the financial management of Hermann Wolff in 1882. Their new conductor was Ludwig von Brenner; in 1887 Hans von Bülow, one of the most esteemed conductors in the world, took over the post. This helped to establish the orchestra's international reputation, and guests Hans Richter, Felix von Weingartner, Richard Strauss, Gustav Mahler, Johannes Brahms and Edvard Grieg conducted the orchestra over the next few years. Programmes of this period show that the orchestra possessed only 46 strings, much less than the Wagnerian ideal of 64.

In 1895, Arthur Nikisch became chief conductor, and was succeeded in 1923 by Wilhelm Furtwängler. Despite several changes in leadership, the orchestra continued to perform throughout World War II. After Furtwängler fled to Switzerland in 1945, Leo Borchard became chief conductor. This arrangement lasted only a few months, as Borchard was accidentally shot and killed by the American forces occupying Berlin. Sergiu Celibidache then took over as chief conductor for seven years, from 1945 to 1952. Furtwängler returned in 1952 and conducted the orchestra until his death in 1954.

His successor was Herbert von Karajan, who led the orchestra from 1955 until his resignation in April 1989, only months before his death. Under him, the orchestra made a vast number of recordings and toured widely, growing and gaining fame. The orchestra hired its first female musician, violinist Madeleine Carruzzo, in 1982. However, Karajan's hiring in September 1982 of Sabine Meyer, the first female wind player to the orchestra, led to controversy when the orchestra voted 73 to 4 not to admit her to the orchestra. Meyer subsequently left the orchestra. After Karajan stood down from the orchestra in 1989, the orchestra offered the chief conductorship to Carlos Kleiber, but he declined.

In 1989, the orchestra elected Claudio Abbado as its next principal conductor. It was the first time the Philharmonic resorted to democratic voting after the fall of the Berlin Wall in 1989. He was known to be humorous in his first months at the Philharmonic. He expanded the orchestra's repertoire beyond the core classical and romantic works into more modern 20th-century works. Abbado stepped down from the chief conductorship of the orchestra in 2002. During the post-unification period, the orchestra encountered financial problems resulting from budgetary stress in the city of Berlin. In 2006, the Orchestra Academy of the Berlin Philharmonic established the Claudio Abbado Composition Prize in Abbado's honour.

In June 1999, the musicians elected Sir Simon Rattle as their next chief conductor. Rattle made it a condition of his signing with the Berlin Philharmonic that it be turned into a self-governing public foundation, with the power to make its own artistic and financial decisions. This required a change to state law, which was approved in 2001, allowing him to join the organization in 2002. Rattle's contract with the orchestra was initially through 2012. In April 2008, the BPO musicians voted in favour of retaining Rattle as their chief conductor through 2018.