They shot Constable Jim Doherty just after 8.30am on 8 November 2007, as he sat in his car in slow-moving traffic in Londonderry, after dropping off his son at his Catholic school in the city.

He represented a new type of policeman, a working-class Catholic who joined the reformed police force which has come into being as part of the new political dispensation. The Northern Ireland Police Service, formerly the Royal Ulster Constabulary, has the support of almost everyone, including Sinn Fein. But the shadow of the gunman has not totally vanished, and one of them fired at the constable with a shotgun.

A close friend said of Jim Doherty that in joining the police he did what he believed was best for his family and for society. "He obviously thought that times have changed and things like this would never happen," the friend said. "Unfortunately it happened to a really good person."

Jim Doherty was struck in the arm and face by shotgun pellets. It was his ill-fortune to be targeted by one of the small bands of republican gunmen who are still active. But then came a stroke of luck: a second man appeared with a handgun to finish him off, but the weapon jammed. The policeman survived and is now recovering after surgery, much to the disappointment of those who attacked him. They said in a statement, part menacing, part truculent: "He might not be so lucky next time."

The attack made the local news but rated barely a mention in mainland Britain. The incident did not, after all, threaten the overall peace process and produced no huge political or security furore. Nobody was killed. Yet it was not an isolated incident but one of many, and there are few indications that the security forces are close to eliminating these republican renegades. The police assessment is, in fact, that the dissident threat is at an all-time high.

The mainstream IRA has decommissioned its weapons and become inactive. But pockets of diehards are still out there – some of them openly violent, some ostensibly political, some a little of both. Compared to the old IRA, the strength of what are known as the dissidents is tiny: they are sometimes referred to as micro-groups. But as the Jim Doherty attack illustrates, they are intent on taking life.

All the advances of the peace process have not persuaded Jim Doherty's attackers to put away their weapons. They still believe that shooting people like him will one day lead to victory and a united Ireland. They try to kill other police officers by attacking them with guns, rockets, landmines and under-car boobytraps. Over the past year alone, a dozen officers have had to move home due to direct terrorist threats.

These republicans have also attempted to bomb a variety of targets. They managed to inflict well over £10m of damage with two attacks in late 2006, one on a major shopping centre in Newry and the other on a Homebase store which was destroyed by incendiaries in south Belfast. The rest of Ireland, north and south, has moved on: Northern Ireland is pretty much at peace, most of its problems having been transmuted by the peace process from the violent to the constitutional. Belfast resounds not to gunfire but to debates on health and education and job creation. The struggle is no longer an armed one, since Sinn Fein has led almost all republicans into the political arena. But not quite all: there are still several thousand who disapprove of the new political settlement. Among them there are a couple of hundred prepared still to use guns and bombs.

The military have practically gone from Northern Ireland, but ironically the dissidents want them back on the streets again. At some psychological level they're more comfortable with what they can claim is an army of occupation rather than a civilianised police service. According to commentator Dr Brian Feeney: "There's no doubt they are trying to provoke a reaction. Ideally they want to kill a policeman in order to get the army back on the streets. They want the nationalist population to get oppressed again."

The leader of Sinn Fein, Martin McGuinness, scornfully echoed this point: "Do they really want to see twenty or thirty thousand soldiers back on the streets? Those people think that more car-bombing, more military activity, is going to bring about the freedom of Ireland. They're living in cloud cuckoo land."

McGuinness was voicing the politics of the new republicanism, but he and Sinn Fein have failed to convince the small, almost hermetically sealed world of dissident republicans. These groups, a modern version of what in the 1920s was referred to as the legion of the rearguard, disapprove of the peace because it is bringing the wrong sort of peace. They insist that Northern Ireland is occupied territory, that the British presence is illegal, and that getting rid of the British will only be achieved through force.

This was spelt out bluntly by a leading Derry dissident, Gary "Donzo" Donnelly. He declared defiantly: "I believe whilst there is a British presence in Ireland there will be people who will resent it." Quoting Padraig Pearse, one of the leaders of the 1916 rising in Dublin, who was subsequently executed, he said that "Ireland unfree shall never be at peace," and added: "I think there's never been a truer statement."

The groups who share such sentiments are not numerically strong but they remain dangerous. Sometimes they make efforts to achieve closer cooperation with each other, but generally they have remained separate and often rival entities. In fact, Sir Hugh Orde, the Chief Constable, warns that they vie with each other. "What worries us," he said, "is that there is probably an appalling competition between the dissident republican groups to be successful. In their eyes this would mean to kill a police officer. They are determined to kill an officer – they are just determined to cause mayhem."

None of the dissident groups has succeeded in killing a police officer or soldier yet: a small number of officers have been injured but there have been no police fatalities. The renegades mainly in fact kill their own members. The half-dozen deaths they have been responsible for in recent times include a couple of uninvolved civilians, but mostly they include members of their own groups, killed for internal reasons.

There are at least four active groups. The oldest, the Irish National Liberation Army (INLA), has been in existence since the 1970s and has carried out three killings in the last few years. With more or less independent gangs north and south of the border, it has of late become relatively inactive, though members remain involved in criminal activity, including "fund-raising activities". That means robberies and extortion, ostensibly for "the cause" but individuals actually keep a large percentage for themselves. According to an official intelligence assessment of the INLA: "We believe that it retains a desire to be able to mount attacks. We continue to believe it is a threat and has the capacity for extreme violence."

Another newer grouping calling itself ONH emerged last year, quickly becoming involved in targeting, recruiting and training new members as well as attempting to raise funds and obtain weapons. Its name is an abbreviation of Oglaigh na hEireann, the name the old IRA used to use in formal circumstances, for example when issuing their annual Easter statements. ONH's internal discipline might be described as stern: it has already exiled one disobedient member – they regarded him as something of a dissident dissident – and killed another in an internal dispute. The security forces regard it as "a continuing and serious threat".

The Continuity IRA has been in existence for many years and in recent times has been responsible for two internal killings as well as attacks on police officers, vehicles and premises. It is described as "active, dangerous and committed, and capable of a greater level of violent and other crime". Its political wing, Republican Sinn Fein, is blunt about its attitude towards "the British colonial police". When a politician accused it of attempted murders, a spokesman coolly explained that there were no murder attempts on police, "There were simply a number of military engagements designed to kill them."

But of the violent groups, the Real IRA is the best-known, or rather the most notorious. It was they who shot Jim Doherty, they who caused the damage to the Newry shopping centre. Above all, they killed 29 people in the 1998 Omagh bombing. After that atrocity, they seemed to be on the floor, hastily and shamefacedly declaring a ceasefire. For a moment, it seemed that the wholesale civilian slaughter, by illustrating the tragedy and futility of violence, would bring dissident activity to a close.

Yet back through the centuries of Irish history the ending of republican insurgencies has rarely been a tidy business, and so it has proved this time. Dissident violence did not go away. The Omagh carnage brought a determined political and security effort to put Real IRA members behind bars, but it was only partly successful. The organisation's founder Michael McKevitt, who had broken away from the mainstream IRA in opposition to the peace process, is serving 20 years in the Republic for directing terrorism. He remains in jail, having this summer lost his appeal against his sentence. But some of his colleagues were never imprisoned, or served shorter terms for other offences. A number of alleged Real IRA leaders currently face a civil action in the courts.

The incarceration of McKevitt did not halt Real IRA violence, however, and nor did a serious split among its members. Instead it regrouped and became active again in both Northern Ireland and Britain. Two years after Omagh it mounted a "spectacular", with a rocket attack on the London headquarters of MI6. Although this caused little actual damage it was an undeniable coup for the dissidents, announcing they were back in business.

But although it remains active it has lowered its sights and has been unable to mount more attacks in Britain. McKevitt once thought big: the FBI agent who gave evidence at his trial related conversations with McKevitt in which he ambitiously dreamt of assassinating Tony Blair and of kidnapping British peers and their sons. He is of limited use in jail, but he has been of continuing value to the Real IRA. He was the IRA's quarter-master general, and when he walked out he took some of its material with him, including the deadly Semtex plastic explosive. Some of this was used as a booster to set off the 150-200 kilos of homemade explosives in the Omagh bomb. It still turns up in terrorist incidents: in August it was used in a rocket attack on police in Fermanagh.

MI5 and the Special Branches in both parts of Ireland pour huge effort and huge resources into combating the dissident republicans, with some success. Efforts to smuggle in weaponry through Lithuania and the Balkans have been thwarted. On the loyalist side, there are no active campaigns of violence, although "fundraising activities" such as robberies and other criminal activities still occur. The last notable killing from loyalist paramilitaries – known with typical black humour in Northern Ireland as "internal housekeeping" – was in 2005.

An unusually high percentage of dissident operations go wrong, with weapons and explosives regularly seized and arrests made. Their members often appear in court, and a total of more than 90 dissidents are currently in prison in the two parts of Ireland.

The universal belief is that many of the little groups have been penetrated by the security agencies, often using sophisticated surveillance. The BBC reporter John Ware recently revealed, for example, that intelligence eavesdroppers were able to listen in to the Real IRA's mobile phones. There have also been unexplained oddities during legal proceedings, with certain defendants being spirited away from their former colleagues. One Belfast man convicted of involvement in dissident activity was taken into protective custody, appearing in court protected by eight prison officers. He has now vanished from the prison system and is assumed to be somewhere in Britain in hiding from colleagues who regard him as a security force agent and would dearly love to kill him. Everyone assumes he is not the only dissident recruited by the intelligence agencies.

Internal security was never the Real IRA's strong suit. fMichael McKevitt was put away largely on the evidence of an unusual character named David Rupert, who testified against him after being recruited by the FBI. Rupert, an American with Mohawk blood who was 6ft 4 inches tall and weighed more than 20 stone, described in court how he and the naive McKevitt "clicked right off". McKevitt confided many Real IRA secrets to him.

There is undoubtedly a permanent secret battle going on between the dissidents and the intelligence services. The latter can claim credit for preventing any attacks in Britain since the MI6 rocket incident, and for putting scores of dissidents behind bars. Intelligence people can allow some terrorist operations to go ahead, on the proviso that they pose little or no risk to life. But it is hardly conceivable, in the conditions of today, that they would permit operations such as the attack on Constable Jim Doherty to go ahead.

This indicates that, despite all the intelligence effort, there are still parts of the dissident underworld which remain unpenetrated. This is probably not surprising, in that many of them operate in small, largely independent units. The lack of central co-ordination between and......... within the dissident groups means that a whole network of agents is needed to penetrate them all. This much was frankly admitted by the Chief Constable when he explained: "They are well infiltrated both north and south of the border, and are paranoid about their activity because we keep disrupting it. We keep arresting people, we have good coverage and these groups are badly disrupted. But the harsh reality is we have never had a full intelligence picture. Any successful crime in a way is an intelligence failure."

So more than a decade of effort has not eradicated dissident activity; in fact it is clear that enough new recruits are joining up to keep the groups going. In the early days, seasoned ex-IRA operators formed the backbone of these organisations. But today, according to Sir Hugh: "The people we are arresting are not 50- or 60-year-olds from the old world. These are young people who are being targeted by dissidents – disenfranchised, marginalised young people who they are now using to do their dirty work."

Martin McGuinness echoed this, declaring: "I appeal to the small number of young people who may have been influenced by these groups – do not get involved in these pointless activities." This ability of the mavericks to keep going has not been matched by any political progress on their part. Sinn Fein has built itself up to become by far the largest nationalist party in Northern Ireland with an important place in government.

It has faltered lately in the south, but in a recent council by-election it took almost 2,000 votes while an anti-Sinn Fein republican got less than two hundred; the latter organisation has developed no political programme beyond the traditional "Brits Out".

The political failure of the dissidents is easy to understand. The devolved administration at Stormont is based on the theory that the various factions and forces are too weak to win but too strong to be defeated. The essential compromise is this intricate political settlement which gives all involved political input. No one, save for the dissidents, believes that all this can be overturned by the violence of minuscule groups.

The public at large take a curiously relaxed attitude towards their campaigns. The general view is that they are wrong-headed, irrelevant and peripheral and that their activities are, in McGuinness's word, pointless. But their unexpected survival is partly explained by the fact that it is grounded in old pre-Troubles republicanism. This was scornful about politics and held that the republican aim should be victory and not the present inclusive settlement.

Today there are few takers for such simplistic propositions, especially when faced with Sinn Fein's electoral gains. These represent a standing rebuttal of the dissident contention that progress can only be achieved through violence. But the new political system is taking time to bed in, with a continuing trial of strength between Sinn Fein and loyalist politicians. The Assembly is meeting, but the ruling executive is not.

Sinn Fein has to thrash out a deal with the loyalists, but it also has to confound the dissidents by demonstrating that politics can deliver. The hard core of dissidents is never going to be convinced that the gun should be taken out of Irish politics, but the hope is that an effectively functioning system of powersharing in Belfast will eventually undermine them. This intertwining of high politics and low terrorism is something that concerns the authorities, who want to get the political show on the road. They already detect a new attitude towards the police, a senior officer reporting: "There is a huge wind of change out there. I have seen that in a number of investigations – there is a completely different response to police on the ground." But in the meantime relaxations in security policy are being held up. The police have already been civilianised to a high degree, but it is impossible to dispense with all the old armoured Land Rovers while the threat of rockets and landmines remains. In other words, even a small number of violent dissidents can have a disproportionate impact on policing.

A high degree of impatience was evident in comments from the Chief Constable when he said: "The politicians need to nail the political vacuum in which these people are now operating. Until they do that, quite frankly I think the opportunities for recruiting more marginalised young people into this crazy activity will continue."

In the meantime, the Real IRA and the other groups will go on their lethal way. The rest of the world has moved on, but a few young men are still out there with guns, ready to put at risk both their own lives and those of people like Constable Jim Doherty.