By Ben Hirschler

LONDON (Reuters) - GlaxoSmithKline Plc (GSK) (LSE:GSK) has stopped a high-profile clinical trial using a novel vaccine to fight lung cancer after deciding it will not be possible to find a sub-group of patients who might benefit.

The decision comes less than two weeks after GSK said the MAGE-A3 therapeutic vaccine did not help patients with non-small cell lung cancer in the Phase III study overall, but it was still looking for improvements among patients with a particular genetic profile.

In the event, scientists analysing the trial decided there was insufficient treatment effect to pinpoint benefit in such gene-signature patients.

Vincent Brichard, head of immunotherapeutics at GSK Vaccines, who announced the decision on Wednesday, said he was "extremely disappointed" by the result.

Identifying a sub-group for whom MAGE-A3 would work had been considered a long shot by investors. Nonetheless, Citi analyst Andrew Baum said the latest setback removed "important optionality" for future sales and profits.

A second trial testing the vaccine in melanoma, which also failed to help patients overall, will continue to investigate benefits in sub-populations. However, the commercial opportunity in melanoma is only 20 to 30 percent that of lung cancer, according to Citi.

The outcome of the melanoma research is expected in 2015.

Unlike traditional preventative vaccines, the MAGE-A3 treatment was designed for people with established disease, helping their immune systems to prevent the return of disease after surgery.

Other companies, including Bristol-Myers Squibb Co (NYS:BMY), Roche Holding AG (VTX:ROG) and Merck & Co Inc (NYS:MRK), have had some recent notable successes in clinical trials of innovative drugs to boost the immune system, but GSK's vaccine-based approach is different and has met with less success.

Shares in Britain's biggest drugmaker were 0.7 percent lower at 8:50 a.m. British time, underperforming a 0.3 percent decline in the European drugs sector. (.SXDP).

MAGE-A3 is one of two high-risk, high-reward research projects that GSK has invested heavily in over recent years - so far, to little avail.

Its experimental heart drug darapladib, which fights clogged arteries in a novel way, has also failed to hit its goals in a late-stage clinical trial, although GSK still hopes a second study may succeed.

U.S.-based Agenus Inc (NAQ:AGEN) has contributed technology to GSK's MAGE-A3 vaccine and its shares are sensitive to news on the project. The vaccine contains Agenus' QS-21 Stimulon adjuvant, or booster.

(Editing by Paul Sandle and David Holmes)