How to substitute a specific string pattern in a multi-line FASTA file using bash?

Question

I have a large multi-line FASTA file that looks like the following:

>NWQ47741.1 CLTR1 protein, partial [Melospiza_melodia] Vertebrate
CLSQGTMTALSPNLSCHNPSIDDFRNSVYSTLYSMISIMGFVGNGVVLYVLIRTYRQKTA
FQIYMLNLALSDFLCVLTLPLRVIYYVHKGHWFFSDFLCRLSSYALYVNLYCSIFFMTAM
SFFRCIAIVFPVRNISLVSEKKAKFLCVGIWVFVTLTSAPFLRNGTYQHGNKTKCFEPPE
NSQKTNMVVILDFIALFVGFIFPFVIITICYTMIIRTLLRNSLRKNEANRRKAVWMIVIV
TATFLVSFTPYHVLRTVHLHALRLRGPGCADTVFLQKAVIVTLPLAAANCCFDPLLYFFS
GGNFRQRLTTLRKASSSSLSQAFRKKISVKEKEEEPFGE
>XP_002763076.2 cysteinyl leukotriene receptor 1 [Callithrix_jacchus] Vertebrate
MDGTGNLTVSSATCHDTIDEFRNQVYSTLYSMISVVGFFGNGFVLYVLIKTYHEKSAFQI
YMINLAIADLLCVCTLPLRVVYYVHKGIWFFGDFLCRLSTYALYVNLYCSIFFMTAMSFF
RCIAIVFPVQNINLVTQKKARFVCVGIWIFVILASSPFLITKSYKDEKNNTKCFEPPQDN
QTKNHVLILHYVSLFLGFIIPFVIIIVCYTMIILTLLKKSMKKNLSSHKKAIRMIMVVTA
AFLVSFMPYHIQRTIHLHFLHNETKPCDSVLRMQKSVVITLSLAASNCCFDPLLYFFSGG
NFRRRLSTFRKHSLSSMTYVPRKKASLPEKGEEICKV
>XP_036988076.1 cysteinyl leukotriene receptor 1 [Artibeus_jamaicensis] Vertebrate
MDGTGNLTASSASNNMCNSSIDDFRNQVYSTMYSMISIVGFFGNGFVLYVLIRTYHEKSA
FQIYMINLAVSDLLCVCTLPLRVVYYVHKGMWFFGDILCRLSTYALYVNLYCSIFFMTAM
SFFRCIAIVFPVKNINLVTEKKARFVCASIWVFVILTSSPFLMSKSYKDEKNNTKCFEPP
QDNETKNHIFILHYVSLLVGFLIPFIIIIVCYTMIIFTLLKNSMQKNVPSRKKAVGMIII
VTAAFLISFMPYHIQRTIHLHFLYNETKPCDSVLRMQKSVVITLSLAASNCCFDPLLYFF
SGGNFRRRLSTFRKHSLSSMTYVPKKKVSLPEKEDEVCK

I need to replace a specific substring within each sequence in the file. The substring to be replaced is the part after the accession ID and before the first '[' character. For example, in the first sequence, I want to replace "CLTR1 protein, partial" with "new_string", resulting in:

>NWQ47741.1 new_string [Melospiza_melodia] Vertebrate
CLSQGTMTALSPNLSCHNPSIDDFRNSVYSTLYSMISIMGFVGNGVVLYVLIRTYRQKTA
FQIYMLNLALSDFLCVLTLPLRVIYYVHKGHWFFSDFLCRLSSYALYVNLYCSIFFMTAM
SFFRCIAIVFPVRNISLVSEKKAKFLCVGIWVFVTLTSAPFLRNGTYQHGNKTKCFEPPE
NSQKTNMVVILDFIALFVGFIFPFVIITICYTMIIRTLLRNSLRKNEANRRKAVWMIVIV
TATFLVSFTPYHVLRTVHLHALRLRGPGCADTVFLQKAVIVTLPLAAANCCFDPLLYFFS
GGNFRQRLTTLRKASSSSLSQAFRKKISVKEKEEEPFGE

I'm looking for a way to achieve this using AWK or Sed, as the file is large, and I want an efficient solution. Can someone provide a sample script or command for this task?

I tried the following script:

awk '/^>/ {gsub(/\[[^[]*/, "new_string"); print; next} 1' your_file.fasta > modified_file.fasta

But got the following output:

>NWQ47741.1 CLTR1 protein, partial new_string
CLSQGTMTALSPNLSCHNPSIDDFRNSVYSTLYSMISIMGFVGNGVVLYVLIRTYRQKTA
FQIYMLNLALSDFLCVLTLPLRVIYYVHKGHWFFSDFLCRLSSYALYVNLYCSIFFMTAM
SFFRCIAIVFPVRNISLVSEKKAKFLCVGIWVFVTLTSAPFLRNGTYQHGNKTKCFEPPE
NSQKTNMVVILDFIALFVGFIFPFVIITICYTMIIRTLLRNSLRKNEANRRKAVWMIVIV
TATFLVSFTPYHVLRTVHLHALRLRGPGCADTVFLQKAVIVTLPLAAANCCFDPLLYFFS
GGNFRQRLTTLRKASSSSLSQAFRKKISVKEKEEEPFGE
>XP_002763076.2 cysteinyl leukotriene receptor 1 new_string
MDGTGNLTVSSATCHDTIDEFRNQVYSTLYSMISVVGFFGNGFVLYVLIKTYHEKSAFQI
YMINLAIADLLCVCTLPLRVVYYVHKGIWFFGDFLCRLSTYALYVNLYCSIFFMTAMSFF
RCIAIVFPVQNINLVTQKKARFVCVGIWIFVILASSPFLITKSYKDEKNNTKCFEPPQDN
QTKNHVLILHYVSLFLGFIIPFVIIIVCYTMIILTLLKKSMKKNLSSHKKAIRMIMVVTA
AFLVSFMPYHIQRTIHLHFLHNETKPCDSVLRMQKSVVITLSLAASNCCFDPLLYFFSGG
NFRRRLSTFRKHSLSSMTYVPRKKASLPEKGEEICKV
>XP_036988076.1 cysteinyl leukotriene receptor 1 new_string
MDGTGNLTASSASNNMCNSSIDDFRNQVYSTMYSMISIVGFFGNGFVLYVLIRTYHEKSA
FQIYMINLAVSDLLCVCTLPLRVVYYVHKGMWFFGDILCRLSTYALYVNLYCSIFFMTAM
SFFRCIAIVFPVKNINLVTEKKARFVCASIWVFVILTSSPFLMSKSYKDEKNNTKCFEPP
QDNETKNHIFILHYVSLLVGFLIPFIIIIVCYTMIIFTLLKNSMQKNVPSRKKAVGMIII
VTAAFLISFMPYHIQRTIHLHFLYNETKPCDSVLRMQKSVVITLSLAASNCCFDPLLYFF
SGGNFRRRLSTFRKHSLSSMTYVPKKKVSLPEKEDEVCK

Answer 1

Using sed

$ sed -E '/^>/s/ [^[]*/ new_string /' input_file
>NWQ47741.1 new_string [Melospiza_melodia] Vertebrate
CLSQGTMTALSPNLSCHNPSIDDFRNSVYSTLYSMISIMGFVGNGVVLYVLIRTYRQKTA
FQIYMLNLALSDFLCVLTLPLRVIYYVHKGHWFFSDFLCRLSSYALYVNLYCSIFFMTAM
SFFRCIAIVFPVRNISLVSEKKAKFLCVGIWVFVTLTSAPFLRNGTYQHGNKTKCFEPPE
NSQKTNMVVILDFIALFVGFIFPFVIITICYTMIIRTLLRNSLRKNEANRRKAVWMIVIV
TATFLVSFTPYHVLRTVHLHALRLRGPGCADTVFLQKAVIVTLPLAAANCCFDPLLYFFS
GGNFRQRLTTLRKASSSSLSQAFRKKISVKEKEEEPFGE
>XP_002763076.2 new_string [Callithrix_jacchus] Vertebrate
MDGTGNLTVSSATCHDTIDEFRNQVYSTLYSMISVVGFFGNGFVLYVLIKTYHEKSAFQI
YMINLAIADLLCVCTLPLRVVYYVHKGIWFFGDFLCRLSTYALYVNLYCSIFFMTAMSFF
RCIAIVFPVQNINLVTQKKARFVCVGIWIFVILASSPFLITKSYKDEKNNTKCFEPPQDN
QTKNHVLILHYVSLFLGFIIPFVIIIVCYTMIILTLLKKSMKKNLSSHKKAIRMIMVVTA
AFLVSFMPYHIQRTIHLHFLHNETKPCDSVLRMQKSVVITLSLAASNCCFDPLLYFFSGG
NFRRRLSTFRKHSLSSMTYVPRKKASLPEKGEEICKV
>XP_036988076.1 new_string [Artibeus_jamaicensis] Vertebrate
MDGTGNLTASSASNNMCNSSIDDFRNQVYSTMYSMISIVGFFGNGFVLYVLIRTYHEKSA
FQIYMINLAVSDLLCVCTLPLRVVYYVHKGMWFFGDILCRLSTYALYVNLYCSIFFMTAM
SFFRCIAIVFPVKNINLVTEKKARFVCASIWVFVILTSSPFLMSKSYKDEKNNTKCFEPP
QDNETKNHIFILHYVSLLVGFLIPFIIIIVCYTMIIFTLLKNSMQKNVPSRKKAVGMIII
VTAAFLISFMPYHIQRTIHLHFLYNETKPCDSVLRMQKSVVITLSLAASNCCFDPLLYFF
SGGNFRRRLSTFRKHSLSSMTYVPKKKVSLPEKEDEVCK

Answer 2

The problem with your attempt is that the gsub regex you passed in matches the square brackets, not the text before them.

Try

awk '/^>/ {
  sub(/ [^][]+ \[/, " new string ["} 
  1' your_file.fasta > modified_file.fasta

There can only be one match per line, so sub is more appropriate than gsub here, especially if performance is a concern (it doesn't have to look for additional matches, and fail to find any). Notice how the match expression includes some context around the tokens to replace, which we take care to put back in the replacement.

Answer 3

I want an efficient solution

Observe that you do not need gsub (global sub) there as you will have at most 1 replacement in each line sub will suffice.

I would GNU AWK for this task following way, let file.txt content be

>NWQ47741.1 CLTR1 protein, partial [Melospiza_melodia] Vertebrate
CLSQGTMTALSPNLSCHNPSIDDFRNSVYSTLYSMISIMGFVGNGVVLYVLIRTYRQKTA
FQIYMLNLALSDFLCVLTLPLRVIYYVHKGHWFFSDFLCRLSSYALYVNLYCSIFFMTAM
SFFRCIAIVFPVRNISLVSEKKAKFLCVGIWVFVTLTSAPFLRNGTYQHGNKTKCFEPPE
NSQKTNMVVILDFIALFVGFIFPFVIITICYTMIIRTLLRNSLRKNEANRRKAVWMIVIV
TATFLVSFTPYHVLRTVHLHALRLRGPGCADTVFLQKAVIVTLPLAAANCCFDPLLYFFS
GGNFRQRLTTLRKASSSSLSQAFRKKISVKEKEEEPFGE
>XP_002763076.2 cysteinyl leukotriene receptor 1 [Callithrix_jacchus] Vertebrate
MDGTGNLTVSSATCHDTIDEFRNQVYSTLYSMISVVGFFGNGFVLYVLIKTYHEKSAFQI
YMINLAIADLLCVCTLPLRVVYYVHKGIWFFGDFLCRLSTYALYVNLYCSIFFMTAMSFF
RCIAIVFPVQNINLVTQKKARFVCVGIWIFVILASSPFLITKSYKDEKNNTKCFEPPQDN
QTKNHVLILHYVSLFLGFIIPFVIIIVCYTMIILTLLKKSMKKNLSSHKKAIRMIMVVTA
AFLVSFMPYHIQRTIHLHFLHNETKPCDSVLRMQKSVVITLSLAASNCCFDPLLYFFSGG
NFRRRLSTFRKHSLSSMTYVPRKKASLPEKGEEICKV
>XP_036988076.1 cysteinyl leukotriene receptor 1 [Artibeus_jamaicensis] Vertebrate
MDGTGNLTASSASNNMCNSSIDDFRNQVYSTMYSMISIVGFFGNGFVLYVLIRTYHEKSA
FQIYMINLAVSDLLCVCTLPLRVVYYVHKGMWFFGDILCRLSTYALYVNLYCSIFFMTAM
SFFRCIAIVFPVKNINLVTEKKARFVCASIWVFVILTSSPFLMSKSYKDEKNNTKCFEPP
QDNETKNHIFILHYVSLLVGFLIPFIIIIVCYTMIIFTLLKNSMQKNVPSRKKAVGMIII
VTAAFLISFMPYHIQRTIHLHFLYNETKPCDSVLRMQKSVVITLSLAASNCCFDPLLYFF
SGGNFRRRLSTFRKHSLSSMTYVPKKKVSLPEKEDEVCK

then

awk 'BEGIN{FS=OFS="["}/^>/{sub(/ .*/,"",$1);$1=$1 " new_string "}{print}' file.txt

gives output

>NWQ47741.1 new_string [Melospiza_melodia] Vertebrate
CLSQGTMTALSPNLSCHNPSIDDFRNSVYSTLYSMISIMGFVGNGVVLYVLIRTYRQKTA
FQIYMLNLALSDFLCVLTLPLRVIYYVHKGHWFFSDFLCRLSSYALYVNLYCSIFFMTAM
SFFRCIAIVFPVRNISLVSEKKAKFLCVGIWVFVTLTSAPFLRNGTYQHGNKTKCFEPPE
NSQKTNMVVILDFIALFVGFIFPFVIITICYTMIIRTLLRNSLRKNEANRRKAVWMIVIV
TATFLVSFTPYHVLRTVHLHALRLRGPGCADTVFLQKAVIVTLPLAAANCCFDPLLYFFS
GGNFRQRLTTLRKASSSSLSQAFRKKISVKEKEEEPFGE
>XP_002763076.2 new_string [Callithrix_jacchus] Vertebrate
MDGTGNLTVSSATCHDTIDEFRNQVYSTLYSMISVVGFFGNGFVLYVLIKTYHEKSAFQI
YMINLAIADLLCVCTLPLRVVYYVHKGIWFFGDFLCRLSTYALYVNLYCSIFFMTAMSFF
RCIAIVFPVQNINLVTQKKARFVCVGIWIFVILASSPFLITKSYKDEKNNTKCFEPPQDN
QTKNHVLILHYVSLFLGFIIPFVIIIVCYTMIILTLLKKSMKKNLSSHKKAIRMIMVVTA
AFLVSFMPYHIQRTIHLHFLHNETKPCDSVLRMQKSVVITLSLAASNCCFDPLLYFFSGG
NFRRRLSTFRKHSLSSMTYVPRKKASLPEKGEEICKV
>XP_036988076.1 new_string [Artibeus_jamaicensis] Vertebrate
MDGTGNLTASSASNNMCNSSIDDFRNQVYSTMYSMISIVGFFGNGFVLYVLIRTYHEKSA
FQIYMINLAVSDLLCVCTLPLRVVYYVHKGMWFFGDILCRLSTYALYVNLYCSIFFMTAM
SFFRCIAIVFPVKNINLVTEKKARFVCASIWVFVILTSSPFLMSKSYKDEKNNTKCFEPP
QDNETKNHIFILHYVSLLVGFLIPFIIIIVCYTMIIFTLLKNSMQKNVPSRKKAVGMIII
VTAAFLISFMPYHIQRTIHLHFLYNETKPCDSVLRMQKSVVITLSLAASNCCFDPLLYFF
SGGNFRRRLSTFRKHSLSSMTYVPKKKVSLPEKEDEVCK

Explanation: I inform GNU AWK that [ is both field separator (FS) and output field separator (OFS), for lines starting with > I use sub function to remove 1st space and everything after it up to end of 1st line (i.e. [) then I concatenate it with new_string (observe leading and trailing space). Every line is printed.

(tested in GNU Awk 5.1.0)

Answer 4

Biopython library contains excellent tools to handle fasta and genebank files.

from Bio import SeqIO
import re

with open('input.fasta' , 'r') as in_fh, open('output.fasta', 'w') as out_fh:
    for seq_record in SeqIO.parse(in_fh, "fasta"):
        newheader = re.sub("(^\S+).+(\[.*)", r"\1 new_string \2", seq_record.description)
        seq_record.description = newheader
        SeqIO.write(seq_record, out_fh, "fasta")