The poll, published on Thursday, found that 34 percent of respondents believe Islam is a part of Germany, which has a population of between 2.1 million and 4 million Muslims.
The figures show a significant shift in public opinion since 2010, when the then German President Christian Wulff said in a speech that Islam is part of Germany. At the time, 49 percent of respondents agreed with the president while 47 percent rejected his statement.
Reservations over Islam were also more apparent among older Germans, with 71 percent of those aged over 64 stating that Islam does not belong to Germany. In contrast, almost half of Germans aged between 18 and 34 said that Islam does belong there.
Concerns over influence of Islam
The survey, carried out by Infratest dimap for German public broadcaster WDR, comes amid growing skepticism towards Islam in Germany, particularly among supporters of the right-wing Alternative for Germany (AfD) and free-market Free Democratic Party (FDP).
According to the poll, just over three quarters of surveyed FDP supporters believed that Islam does not belong in Germany. Among AfD supporters, figures were even higher, coming in at 94 percent.
A majority of 58 percent of Germans were also found to believe that the country's established parties - the Christian Democrats (CDU), the CDU's Bavarian sister party, the Christian social Union (CSU), the Social Democrats, (SPD), the Greens, the Left, and the FDP - are failing to take concerns about radical Islam seriously enough.
These are concerns the AfD has drawn upon in recent months, much to its advantage. The right-wing populist party is now represented in eight of Germany's 16 state parliaments. At the national level, the AfD has made strong gains, with opinion polls suggesting that they now hold between 12 percent and 14 percent of the vote.
Half of Germans also said they were concerned about the influence of Islam in Germany because of the number of refugees entering the country, the poll found. Germany took in some 1.1 million migrants in 2015 - the largest influx since the Second World War.
The number of arrivals has significantly decreased in recent months, however, largely due to the closure of the Balkan migrant route in early March, which left thousands of refugees stranded in Greece.
Fear of a terrorist attack by Islamist extremists in Germany also remains relatively high, with 72 percent of respondents fearing such an attack. A quarter of Germans didn't have this fear, the poll found.
The Infratest dimap representative survey was carried out on May 2 and 3, with around 1,000 participants.