I'm trying to get rid of <script> tags and the content inside the tag utilizing beatifulsoup. I went to the documentation and seems to be a really simple function to call. More information about the function is here. Here is the content of the html page that I have parsed so far...
<body> <div>
</div>
<div>
</div> <div> <script> (function(a){ TWP=window.TWP||{}; TWP.Features=TWP.Features||{}; TWP.Features.Page=TWP.Features.Page||{}; TWP.Features.Page.PostRecommends={}; TWP.Features.Page.PostRecommends.url=""; TWP.Features.Page.PostRecommends.trackUrl=""; TWP.Features.Page.PostRecommends.profileUrl=""; TWP.Features.Page.PostRecommends.canonicalUrl="" })(jQuery); </script> </div>
</body>Imagine you have some web content like that and you have that in a BeautifulSoup object called soup_html. If I run soup_html.script.decompose() and them call the object soup_html the script tags still there. How I can get rid of the <script> and the content inside those tags?
markup = 'The html above'
soup = BeautifulSoup(markup)
html_body = soup.body
soup.script.decompose()
html_body4 Answers
soup.script.decompose()
This would remove a single script element from the "Soup" only. Instead, I think you meant to decompose all of them:
for script in soup("script"): script.decompose()To elaborate on the answer provided by alecxe, here is a full script for anyone's reference:
selects = soup.findAll('select')
for match in selects: match.decompose()I was able to fix the issue with the following code...
scripts = soup.findAll(['script', 'style']) for match in scripts: match.decompose() file_content = soup.get_text() # Striping 'ascii' code content = re.sub(r'[^\x00-\x7f]', r' ', file_content) # Creating 'txt' files with open(my_params['q'] + '_' + str(count) + '.txt', 'w+') as webpage_out: webpage_out.write(content) print('The file ' + my_params['q'] + '_' + str(count) + '.txt ' + 'has been created successfully.') count += 1The error was that the with open(... was part or the for match...
Code that did not work...
scripts = soup.findAll(['script', 'style']) for match in scripts: match.decompose() file_content = soup.get_text() # Striping 'ascii' code content = re.sub(r'[^\x00-\x7f]', r' ', file_content) # Creating 'txt' files with open(my_params['q'] + '_' + str(count) + '.txt', 'w+') as webpage_out: webpage_out.write(content) print('The file ' + my_params['q'] + '_' + str(count) + '.txt ' + 'has been created successfully.') count += 1The soup.script.decompose() would only remove it from the soup variable... not the html_body variable. you would have to remove it from the html_body variable as well. (I think.)
