In the show airing next Thursday, ten prisoners from Wormwood Scrubs and ten prison officers will join 100 members of the public to discuss a range of issues.
The inmates will be able to quiz justice secretary Ken Clarke, along with former home secretary Jack Straw and other panellists. The show will be recorded at around 8.30pm at the West London prison and aired at 10.35pm on BBC One.
The move follows a European Court of Human Rights verdict in April giving the UK six months to comply with rules extending the right to vote to prisoners.
Discussing the show, a BBC spokesman said: "There will be 10 prisoners, none of whom will have been serving for any violent crimes. They will be vetted by the prison staff and the BBC.
"The involvement in the debate of prisoners and prison staff will offer Question Time viewers a unique insight into their views on the issue of the right to vote as well as more general questions."
Question Time is no stranger to controversy, having given a seat on the panel to British National Party leader Nick Griffin in 2009, despite widespread protests.
Speaking to The Sun, a Question Time source reiterated that the Wormwood Scrubs broadcast will not feature any high risk offenders, such as murderers and rapists.
The source added: "Many people will see it as a stunt, but this is a very high-risk move from the BBC. The aim is to try to get some debate around the rights of prisoners. You won't see people there who are murderers, rapists or paedophiles."
> Read our latest Media news