Added 1.5.16 (Updated 6.10.17)
Is al-Qur'an 4:101 really calling non-Muslims "open enemies"? And doesn’t 8:12 say to kill the infidels?
Neither revelation was in general. Rather, both were in reference to circumstances at the time of the Prophet Muhammad, peace on him. Let me explain the background and then I’ll get to the specifics. I'll also add important evidences at the end about how Muslims are to treat non-Muslims in general.
The people of Makkah, the Quraysh, had turned the Ka’bah into a place of idol worship and since people made pilgrimage there, this became part of their economy. When the Prophet Muhammad, peace on him, came with a revelation saying to worship none but Allah, and especially when people started following him, the Quraysh objected leaving their way of life and the religion of their forefathers. More than that, they persecuted those early Muslims. They tortured, killed, lied about, and boycotted them. The boycott, in particular, went on for three years wherein the Muslims were cast out of their homes and were trapped in a valley eating, at a point, leaves and camel skin. Eventually, the Muslims migrated north to al-Madinah (Yathrib) to escape that persecution.
Ayah 8:12, the second one mentioned in the question, was revealed when 1000 Quryash fighters had come to al-Madinah and were descending on virtually the only 300 or so able-bodied Muslim men on the face of the earth. Allah ordered those Muslims in that situation to fight.
As for the first ayah (4:101), the statement relates to disbelievers who fit the condition laid out in the second part of the ayah, which says, "if you fear that those who disbelieve may disrupt (or attack) you." More specifically, it is in relation to the speakers in this hadith who say they should have attacked the Muslims when they were praying: Narrated AbuAyyash az-Zuraqi: “We accompanied the Apostle of Allah at Usfan, and Khalid ibn al-Walid was the chief of unbelievers. We offered the noon prayer. Thereupon, the unbelievers said: 'We suffered from negligence; we became careless. We should have attacked them while they were praying.' Thereupon the verse was revealed, relating to the shortening of the prayer (in time of danger) between the noon and afternoon prayers (i.e., 4:101)" (Abu Dawood 4:1232). Thus, the open enemies are specifically those who hate the Muslims and have the desire to harm or kill them.
Allah explains the default position of Muslims toward non-Muslims in the first of the following ayatayn (two ayah), the second one describing the position toward those who mean dire harm: "Allah does not forbid you from those who do not fight you because of religion and do not expel you from your homes—from being righteous toward them and acting justly toward them. Indeed, Allah loves those who act justly. Allah only forbids you from those who fight you because of religion and expel you from your homes and aid in your expulsion—[forbids] that you make allies of them. And whoever makes allies of them, then it is those who are the wrongdoers" (al-Qur'an 60:8-9).
Further, even when the Muslims were in governance over a diverse population, the Prophet Muhammad, peace on him, said, “Whoever kills a mu‘aahid (a non-Muslim living under Muslim rule) will not smell the fragrance of Paradise, although its fragrance may be detected from a distance of forty years” (Bukhari 2995). In another narration he said that whoever harms such a non-Muslim “harms me and whoever harms me annoys Allah.” In yet another he said, “He who hurts a non-Muslim living under Muslim rule, I am his adversary, and I shall be his adversary on the Day of a Judgment.” The Prophet, peace on him, put himself between the other Muslims and the non-Muslims. Subhan Allah, Glory to Allah.
Someone will rightly ask, though, what happens where there is battle. When it does come to that the rules revealed through the Prophet Muhammad, peace on him, are much more restrictive than the Geneva Conventions. Killing unarmed non-Muslim targets graphically departs from the actions of Ahlus-Sunnah wal-Jam’ah and whoever does this stands in violation of Shari'ah, the Law of Allah. Based on the facts reported regarding, for example, the Paris attacks of November 2015, the early December shooting in San Bernardino, California, or all similar events, they are of the deeds hated by Allah and they tragically complicate life for all, including Muslims. (And in fact 90%+ of ISIS's victims are Muslims!) When military action is required to defend against oppression, whether that oppression be on Muslims or non-Muslims in Muslim protection, the Shari’ah demands firm restrictions, restrictions ISIS wantonly disregards. Foremost, killing of non-combatants is forbidden with much emphasis on safeguarding women, children, and the infirm. Even when the Prophet, peace on him, had fallen ill and died, the leader after him, Abu Bakr, Allah be pleased with him, emphasized these points and more to the military leader Usama. The great companion said, “Do not kill an old man, a woman or a child. Do not injure date palms and do not cut down fruit trees. Do not slaughter any sheep or cows or camels except for food. You will encounter persons who spend their lives in monasteries. Leave them alone and do not molest them” (qtd. in “The First Caliph”). Even within these limitations personal anger is an invalid intention for lethal force as evidenced in the famous narration in which Ali was about to prevail in a fight when he tossed his sword aside due to his anger when the man spat in his face.There is absolutely no getting around the clear matter that anyone who behaves rashly and kills citizenry has violated the Shari’ah and is going against the way of Prophet Muhammad, peace on him.