## input text article
article_text="By the time Alan Turing enrolled at Cambridge, the Mathematical and Philosophical worlds had already collapsed in on one another by way of a bitter civil war between logicians. Set off by Georg Cantor’s set theory in 1874, then divided by his diagonal argument in 1891, two camps had emerged. On the one hand were the intuitionists, led by Henri Poincare, who thought set theory was literally a disease that needed to be cured. On the other were the formalists, led by David Hilbert, a disciple of Cantor’s whose battle cry would literally become his epitaph; “We Must Know, We Will Know.” If Poincare’s reliance on natural intuition for certain axioms without proof frustrated you, you were probably a formalist. And if Hilbert’s faith in formality dispelling any mystery of veracity made you uncomfortable, you were probably an intuitionist. Either way, the gauntlet that was laid down would come to be the defining questions of not just mathematics, but every logical system that has ever or will ever be conceived of: Is Mathematics Complete? Is it Consistent? Is it Decideable? If Mathematics is Complete, then there is a way to prove every true statement. If Mathematics is Consistent, then it means it is free of contradictions. And if Mathematics is Decideable, then there is an algorithm that can always determine whether a statement follows from the axioms of the system.Answering the call almost immediately was Kurt Gödel. In 1930, he was awarded his doctorate, and revealed his incompleteness theorems to the world. he proved for any computable axiomatic system that is powerful enough to describe the arithmetic of the natural numbers that:1. If a logical or axiomatic formal system is consistent, it cannot be syntactically complete.2. The consistency of axioms cannot be proved within their own system.So not only was Mathematics not complete, it could only be called consistent until the day that it wasn’t. All that remained to be answered was Hilbert’s Entscheidungsproblem; the “decision problem” that asks for an algorithm programmed with the axioms of a logical system that considers an input statement, and then is able to determine whether that input is true or false (It’s important to note that up until this point, Hilbert believed there was no such thing as an unsolvable problem). Whether it had become all too ironic or paradoxical by this point, the problem was one of self-reference; the problem was with the question itself. Before the question could be answered the notion of an algorithm needed to be formally defined. Up until this point mathematicians only described functions in terms of their effective calculability, but no one had been able to determine whether they were algorithmically computable."
import re
import nltk
article_text = article_text.lower()
article_text
'by the time alan turing enrolled at cambridge, the mathematical and philosophical worlds had already collapsed in on one another by way of a bitter civil war between logicians. set off by georg cantor’s set theory in 1874, then divided by his diagonal argument in 1891, two camps had emerged. on the one hand were the intuitionists, led by henri poincare, who thought set theory was literally a disease that needed to be cured. on the other were the formalists, led by david hilbert, a disciple of cantor’s whose battle cry would literally become his epitaph; “we must know, we will know.” if poincare’s reliance on natural intuition for certain axioms without proof frustrated you, you were probably a formalist. and if hilbert’s faith in formality dispelling any mystery of veracity made you uncomfortable, you were probably an intuitionist. either way, the gauntlet that was laid down would come to be the defining questions of not just mathematics, but every logical system that has ever or will ever be conceived of: is mathematics complete? is it consistent? is it decideable? if mathematics is complete, then there is a way to prove every true statement. if mathematics is consistent, then it means it is free of contradictions. and if mathematics is decideable, then there is an algorithm that can always determine whether a statement follows from the axioms of the system.answering the call almost immediately was kurt gödel. in 1930, he was awarded his doctorate, and revealed his incompleteness theorems to the world. he proved for any computable axiomatic system that is powerful enough to describe the arithmetic of the natural numbers that:1.\tif a logical or axiomatic formal system is consistent, it cannot be syntactically complete.2.\tthe consistency of axioms cannot be proved within their own system.so not only was mathematics not complete, it could only be called consistent until the day that it wasn’t. all that remained to be answered was hilbert’s entscheidungsproblem; the “decision problem” that asks for an algorithm programmed with the axioms of a logical system that considers an input statement, and then is able to determine whether that input is true or false (it’s important to note that up until this point, hilbert believed there was no such thing as an unsolvable problem). whether it had become all too ironic or paradoxical by this point, the problem was one of self-reference; the problem was with the question itself. before the question could be answered the notion of an algorithm needed to be formally defined. up until this point mathematicians only described functions in terms of their effective calculability, but no one had been able to determine whether they were algorithmically computable.'
# remove spaces, punctuations and numbers
clean_text = re.sub('[^a-zA-Z]', ' ', article_text)
clean_text = re.sub('\s+', ' ', clean_text)
clean_text
'by the time alan turing enrolled at cambridge the mathematical and philosophical worlds had already collapsed in on one another by way of a bitter civil war between logicians set off by georg cantor s set theory in then divided by his diagonal argument in two camps had emerged on the one hand were the intuitionists led by henri poincare who thought set theory was literally a disease that needed to be cured on the other were the formalists led by david hilbert a disciple of cantor s whose battle cry would literally become his epitaph we must know we will know if poincare s reliance on natural intuition for certain axioms without proof frustrated you you were probably a formalist and if hilbert s faith in formality dispelling any mystery of veracity made you uncomfortable you were probably an intuitionist either way the gauntlet that was laid down would come to be the defining questions of not just mathematics but every logical system that has ever or will ever be conceived of is mathematics complete is it consistent is it decideable if mathematics is complete then there is a way to prove every true statement if mathematics is consistent then it means it is free of contradictions and if mathematics is decideable then there is an algorithm that can always determine whether a statement follows from the axioms of the system answering the call almost immediately was kurt g del in he was awarded his doctorate and revealed his incompleteness theorems to the world he proved for any computable axiomatic system that is powerful enough to describe the arithmetic of the natural numbers that if a logical or axiomatic formal system is consistent it cannot be syntactically complete the consistency of axioms cannot be proved within their own system so not only was mathematics not complete it could only be called consistent until the day that it wasn t all that remained to be answered was hilbert s entscheidungsproblem the decision problem that asks for an algorithm programmed with the axioms of a logical system that considers an input statement and then is able to determine whether that input is true or false it s important to note that up until this point hilbert believed there was no such thing as an unsolvable problem whether it had become all too ironic or paradoxical by this point the problem was one of self reference the problem was with the question itself before the question could be answered the notion of an algorithm needed to be formally defined up until this point mathematicians only described functions in terms of their effective calculability but no one had been able to determine whether they were algorithmically computable '
# split into sentence list
sentence_list = nltk.sent_tokenize(article_text)
sentence_list
['by the time alan turing enrolled at cambridge, the mathematical and philosophical worlds had already collapsed in on one another by way of a bitter civil war between logicians.', 'set off by georg cantor’s set theory in 1874, then divided by his diagonal argument in 1891, two camps had emerged.', 'on the one hand were the intuitionists, led by henri poincare, who thought set theory was literally a disease that needed to be cured.', 'on the other were the formalists, led by david hilbert, a disciple of cantor’s whose battle cry would literally become his epitaph; “we must know, we will know.” if poincare’s reliance on natural intuition for certain axioms without proof frustrated you, you were probably a formalist.', 'and if hilbert’s faith in formality dispelling any mystery of veracity made you uncomfortable, you were probably an intuitionist.', 'either way, the gauntlet that was laid down would come to be the defining questions of not just mathematics, but every logical system that has ever or will ever be conceived of: is mathematics complete?', 'is it consistent?', 'is it decideable?', 'if mathematics is complete, then there is a way to prove every true statement.', 'if mathematics is consistent, then it means it is free of contradictions.', 'and if mathematics is decideable, then there is an algorithm that can always determine whether a statement follows from the axioms of the system.answering the call almost immediately was kurt gödel.', 'in 1930, he was awarded his doctorate, and revealed his incompleteness theorems to the world.', 'he proved for any computable axiomatic system that is powerful enough to describe the arithmetic of the natural numbers that:1.\tif a logical or axiomatic formal system is consistent, it cannot be syntactically complete.2.', 'the consistency of axioms cannot be proved within their own system.so not only was mathematics not complete, it could only be called consistent until the day that it wasn’t.', 'all that remained to be answered was hilbert’s entscheidungsproblem; the “decision problem” that asks for an algorithm programmed with the axioms of a logical system that considers an input statement, and then is able to determine whether that input is true or false (it’s important to note that up until this point, hilbert believed there was no such thing as an unsolvable problem).', 'whether it had become all too ironic or paradoxical by this point, the problem was one of self-reference; the problem was with the question itself.', 'before the question could be answered the notion of an algorithm needed to be formally defined.', 'up until this point mathematicians only described functions in terms of their effective calculability, but no one had been able to determine whether they were algorithmically computable.']
## run this cell once to download stopwords
# import nltk
# nltk.download('stopwords')
stopwords = nltk.corpus.stopwords.words('english')
word_frequencies = {}
for word in nltk.word_tokenize(clean_text):
if word not in stopwords:
if word not in word_frequencies:
word_frequencies[word] = 1
else:
word_frequencies[word] += 1
maximum_frequency = max(word_frequencies.values())
for word in word_frequencies:
word_frequencies[word] = word_frequencies[word] / maximum_frequency
sentence_scores = {}
for sentence in sentence_list:
for word in nltk.word_tokenize(sentence):
if word in word_frequencies and len(sentence.split(' ')) < 30:
if sentence not in sentence_scores:
sentence_scores[sentence] = word_frequencies[word]
else:
sentence_scores[sentence] += word_frequencies[word]
word_frequencies
{'time': 0.16666666666666666,
'alan': 0.16666666666666666,
'turing': 0.16666666666666666,
'enrolled': 0.16666666666666666,
'cambridge': 0.16666666666666666,
'mathematical': 0.16666666666666666,
'philosophical': 0.16666666666666666,
'worlds': 0.16666666666666666,
'already': 0.16666666666666666,
'collapsed': 0.16666666666666666,
'one': 0.6666666666666666,
'another': 0.16666666666666666,
'way': 0.5,
'bitter': 0.16666666666666666,
'civil': 0.16666666666666666,
'war': 0.16666666666666666,
'logicians': 0.16666666666666666,
'set': 0.5,
'georg': 0.16666666666666666,
'cantor': 0.3333333333333333,
'theory': 0.3333333333333333,
'divided': 0.16666666666666666,
'diagonal': 0.16666666666666666,
'argument': 0.16666666666666666,
'two': 0.16666666666666666,
'camps': 0.16666666666666666,
'emerged': 0.16666666666666666,
'hand': 0.16666666666666666,
'intuitionists': 0.16666666666666666,
'led': 0.3333333333333333,
'henri': 0.16666666666666666,
'poincare': 0.3333333333333333,
'thought': 0.16666666666666666,
'literally': 0.3333333333333333,
'disease': 0.16666666666666666,
'needed': 0.3333333333333333,
'cured': 0.16666666666666666,
'formalists': 0.16666666666666666,
'david': 0.16666666666666666,
'hilbert': 0.6666666666666666,
'disciple': 0.16666666666666666,
'whose': 0.16666666666666666,
'battle': 0.16666666666666666,
'cry': 0.16666666666666666,
'would': 0.3333333333333333,
'become': 0.3333333333333333,
'epitaph': 0.16666666666666666,
'must': 0.16666666666666666,
'know': 0.3333333333333333,
'reliance': 0.16666666666666666,
'natural': 0.3333333333333333,
'intuition': 0.16666666666666666,
'certain': 0.16666666666666666,
'axioms': 0.6666666666666666,
'without': 0.16666666666666666,
'proof': 0.16666666666666666,
'frustrated': 0.16666666666666666,
'probably': 0.3333333333333333,
'formalist': 0.16666666666666666,
'faith': 0.16666666666666666,
'formality': 0.16666666666666666,
'dispelling': 0.16666666666666666,
'mystery': 0.16666666666666666,
'veracity': 0.16666666666666666,
'made': 0.16666666666666666,
'uncomfortable': 0.16666666666666666,
'intuitionist': 0.16666666666666666,
'either': 0.16666666666666666,
'gauntlet': 0.16666666666666666,
'laid': 0.16666666666666666,
'come': 0.16666666666666666,
'defining': 0.16666666666666666,
'questions': 0.16666666666666666,
'mathematics': 1.0,
'every': 0.3333333333333333,
'logical': 0.5,
'system': 1.0,
'ever': 0.3333333333333333,
'conceived': 0.16666666666666666,
'complete': 0.6666666666666666,
'consistent': 0.6666666666666666,
'decideable': 0.3333333333333333,
'prove': 0.16666666666666666,
'true': 0.3333333333333333,
'statement': 0.5,
'means': 0.16666666666666666,
'free': 0.16666666666666666,
'contradictions': 0.16666666666666666,
'algorithm': 0.5,
'always': 0.16666666666666666,
'determine': 0.5,
'whether': 0.6666666666666666,
'follows': 0.16666666666666666,
'answering': 0.16666666666666666,
'call': 0.16666666666666666,
'almost': 0.16666666666666666,
'immediately': 0.16666666666666666,
'kurt': 0.16666666666666666,
'g': 0.16666666666666666,
'del': 0.16666666666666666,
'awarded': 0.16666666666666666,
'doctorate': 0.16666666666666666,
'revealed': 0.16666666666666666,
'incompleteness': 0.16666666666666666,
'theorems': 0.16666666666666666,
'world': 0.16666666666666666,
'proved': 0.3333333333333333,
'computable': 0.3333333333333333,
'axiomatic': 0.3333333333333333,
'powerful': 0.16666666666666666,
'enough': 0.16666666666666666,
'describe': 0.16666666666666666,
'arithmetic': 0.16666666666666666,
'numbers': 0.16666666666666666,
'formal': 0.16666666666666666,
'syntactically': 0.16666666666666666,
'consistency': 0.16666666666666666,
'within': 0.16666666666666666,
'could': 0.3333333333333333,
'called': 0.16666666666666666,
'day': 0.16666666666666666,
'remained': 0.16666666666666666,
'answered': 0.3333333333333333,
'entscheidungsproblem': 0.16666666666666666,
'decision': 0.16666666666666666,
'problem': 0.6666666666666666,
'asks': 0.16666666666666666,
'programmed': 0.16666666666666666,
'considers': 0.16666666666666666,
'input': 0.3333333333333333,
'able': 0.3333333333333333,
'false': 0.16666666666666666,
'important': 0.16666666666666666,
'note': 0.16666666666666666,
'point': 0.5,
'believed': 0.16666666666666666,
'thing': 0.16666666666666666,
'unsolvable': 0.16666666666666666,
'ironic': 0.16666666666666666,
'paradoxical': 0.16666666666666666,
'self': 0.16666666666666666,
'reference': 0.16666666666666666,
'question': 0.3333333333333333,
'notion': 0.16666666666666666,
'formally': 0.16666666666666666,
'defined': 0.16666666666666666,
'mathematicians': 0.16666666666666666,
'described': 0.16666666666666666,
'functions': 0.16666666666666666,
'terms': 0.16666666666666666,
'effective': 0.16666666666666666,
'calculability': 0.16666666666666666,
'algorithmically': 0.16666666666666666}
sentence_scores
{'by the time alan turing enrolled at cambridge, the mathematical and philosophical worlds had already collapsed in on one another by way of a bitter civil war between logicians.': 3.666666666666666,
'set off by georg cantor’s set theory in 1874, then divided by his diagonal argument in 1891, two camps had emerged.': 2.8333333333333326,
'on the one hand were the intuitionists, led by henri poincare, who thought set theory was literally a disease that needed to be cured.': 3.8333333333333335,
'and if hilbert’s faith in formality dispelling any mystery of veracity made you uncomfortable, you were probably an intuitionist.': 2.3333333333333335,
'is it consistent?': 0.6666666666666666,
'is it decideable?': 0.3333333333333333,
'if mathematics is complete, then there is a way to prove every true statement.': 3.5,
'if mathematics is consistent, then it means it is free of contradictions.': 2.1666666666666665,
'in 1930, he was awarded his doctorate, and revealed his incompleteness theorems to the world.': 0.9999999999999999,
'the consistency of axioms cannot be proved within their own system.so not only was mathematics not complete, it could only be called consistent until the day that it wasn’t.': 4.333333333333333,
'whether it had become all too ironic or paradoxical by this point, the problem was one of self-reference; the problem was with the question itself.': 4.166666666666666,
'before the question could be answered the notion of an algorithm needed to be formally defined.': 2.333333333333333,
'up until this point mathematicians only described functions in terms of their effective calculability, but no one had been able to determine whether they were algorithmically computable.': 4.166666666666666}
# get top 5 sentences
import heapq
summary = heapq.nlargest(5, sentence_scores, key=sentence_scores.get)
print(" ".join(summary))
the consistency of axioms cannot be proved within their own system.so not only was mathematics not complete, it could only be called consistent until the day that it wasn’t. whether it had become all too ironic or paradoxical by this point, the problem was one of self-reference; the problem was with the question itself. up until this point mathematicians only described functions in terms of their effective calculability, but no one had been able to determine whether they were algorithmically computable. on the one hand were the intuitionists, led by henri poincare, who thought set theory was literally a disease that needed to be cured. by the time alan turing enrolled at cambridge, the mathematical and philosophical worlds had already collapsed in on one another by way of a bitter civil war between logicians.